Trilio seamlessly integrates with OpenStack, functioning exclusively through APIs utilizing the OpenStack Endpoints. Furthermore, Trilio establishes its own set of OpenStack endpoints. Additionally, both the Trilio appliance and compute nodes interact with the backup target, impacting the network strategy for a Trilio installation.

Existing endpoints in OpenStack

OpenStack comprises three endpoint groupings:

• Public Endpoints

Public endpoints are meant to be used by the OpenStack end-users to work with OpenStack.

• Internal Endpoints

Internal endpoints are intended to be used by the OpenStack services to communicate with each

• Admin Endpoints

Admin endpoints are meant to be used by OpenStack administrators.

Among these three endpoint categories, it's important to note that the admin endpoint occasionally hosts APIs not accessible through any other type of endpoint.

To learn more about OpenStack endpoints please visit the official OpenStack documentation.

OpenStack endpoints required by Trilio

Trilio communicates with all OpenStack services through a designated endpoint type, determined and configured during the deployment of Trilio's services.

The following network requirements can be identified this way:

• Trilio services need access to the Keystone admin endpoint on the admin endpoint network if it is available.

• Trilio services need access to all endpoints of the set endpoint type during deployment.

Recommendation: Provide access to all OpenStack Endpoint types

Trilio recommends granting comprehensive access to all OpenStack endpoints for all Trilio services, aligning with OpenStack's established standards and best practices.

Additionally, Trilio generates its own endpoints, which are integrated within the same network as other OpenStack API services.

To adhere to OpenStack's prescribed standards and best practices, it's advisable that Trilio containers operate on the same network as other OpenStack containers.

• The public endpoint to be used by OpenStack users when using Trilio CLI or API

• The internal endpoint to communicate with the OpenStack services

• The admin endpoint to use the required admin-only APIs of Keystone

Backup target access required by Trilio

The Trilio solution uses backup target storage to place the backup data securely. Trilio divides its backup data into two parts:

1. Metadata

2. Volume Disk Data

The first type of data is generated by the Trilio Workloadmgr services through communication with the OpenStack Endpoints. All metadata that is stored together with a backup is written by the Trilio Workloadmgr services to the backup target in the JSON format.

The second type of data is generated by the Trilio Datamover service running on the compute nodes. The Datamover service reads the Volume Data from the Cinder or Nova storage and transfers this data as a qcow2 image to the backup target. Each Datamover service is hereby responsible for the VMs running on its compute node.

The network requirements are therefor:

• Every Trilio Workloadmgr service containers need access to the backup target

• Every Trilio Datamover service containers need access to the backup target

Before embarking on the installation process for Trilio in your OpenStack environment, it is highly advisable to carefully consider several key elements. These considerations will not only streamline the installation procedure but also ensure the optimal setup and functionality of Trilio's solutions within your OpenStack infrastructure.

Tenant Quotas

Trilio leverages Cinder snapshots to facilitate the computation of both full and incremental backups.

When executing full backups, Trilio orchestrates the generation of Cinder snapshots for all volumes included in the backup job. These Cinder snapshots remain intact for subsequent incremental backup image calculations.

During incremental backup operations, Trilio generates fresh Cinder snapshots and computes the altered blocks between these new snapshots and the earlier retained snapshots from full or previous backups. The old snapshots are subsequently deleted, while the newly generated snapshots are preserved.

Consequently, it becomes imperative for each tenant benefiting from Trilio's backup functionality to possess adequate Cinder snapshot quotas capable of accommodating these supplementary snapshots. As a guiding principle, it is recommended to append 2 snapshots for each volume incorporated into the backup quotas for the respective tenant. Additionally, a commensurate increase in volume quotas for the tenant is advisable, as Trilio briefly materializes a volume from the snapshot to access data for backup purposes.

During the restoration process, Trilio generates supplementary instances and Cinder volumes. To facilitate seamless restore operations, tenants should maintain adequate quota levels for Nova instances and Cinder volumes. Failure to meet these quota requirements may lead to disruptions in restoration procedures.

AWS S3 Eventual Consistency

The AWS S3 object consistency model includes:

1. Read-after-write

2. Read-after-update

3. Read-after-delete

Each of these models explains how an object becomes consistent after being created, updated, or deleted. None of these methods ensures strong consistency, leading to a delay before an object becomes fully consistent.

Although Trilio has introduced measures to address AWS S3's eventual consistency limitations, the exact time an object achieves consistency cannot be predicted through deterministic means.

There is no official statement from AWS on how long it takes for an object to reach a consistent state. However, read-after-write has a shorter time to reach consistency compared to other IO patterns. Therefore, our solution is designed to maximize the read-after-write IO pattern.

AWS Region Considerations

The time in which an object reaches eventual consistency also depends on the AWS region.

For instance, the AWS-standard region doesn't offer the same level of strong consistency as regions like us-east or us-west. Opting for these regions when setting up S3 buckets for Trilio is advisable. While fully avoiding the read-after-update IO pattern is complex, we've introduced significant access delays for objects to achieve consistency over longer periods. On rare occasions when this does happen, it will cause a backup failure and require a retry.

Trilio Cluster

Trilio can be deployed as a single node or a three-node cluster. It is highly recommended that Trilio is deployed as a three-node cluster for fault tolerance and load balancing. Starting with the 3.0 release, Trilio requires additional IP or FQDN for the cluster and is required for both single-node and three-node deployments. Cluster IP a.k.a virtual IP is used for managing clusters and is used to register the Trilio service endpoint in the keystone service catalog.

Trilio Data, Inc

Trilio Data, Inc. is the leader in providing backup and recovery solutions for cloud-native applications. Established in 2013, its flagship products, Trilio for OpenStack(T4O) and Trilio for Kubernetes(T4K) are used by a wide number of large corporations around the world.

About Trilio for OpenStack

T4O, by Trilio Data, is a native OpenStack service-based solution that provides policy-based comprehensive backup and recovery for OpenStack workloads. It captures point-in-time workloads (Application, OS, Compute, Network, Configurations, Data, and Metadata of an environment) as full or incremental snapshots. These snapshots can be held in a variety of storage environments including NFS, AWS S3, and other S3-compatible storages. With Trilio and its one-click recovery, organizations can improve Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO). With Trilio, IT departments are enabled to fully deploy OpenStack solutions and provide business assurance through enhanced data retention, protection, and integrity.

With the use of Trilio’s VAST (Virtual Snapshot Technology), Enterprise IT and Cloud Service Providers can now deploy backup and disaster recovery as a service to prevent data loss or data corruption through point-in-time snapshots and seamless one-click recovery. Trilio takes point-in-time backup of the entire workload consisting of compute resources, network configurations, and storage data as one unit. It also takes incremental backups that capture only the changes that were made since the last backup. Incremental snapshots save a considerable amount of storage space as the backup only includes changes since the last backup. The benefits of using VAST for backup and restore could be summarized below:

About this Documentation

This documentation serves as the end-user technical resource to accompany Trilio for OpenStack. You will learn about the architecture, installation, and vast number of operations of this product. The intended audience is anyone who wants to understand the value, operations, and nuances of protecting their cloud-native applications with Trilio for OpenStack.

Backup-as-a-Service

Trilio is an add-on service to OpenStack cloud infrastructure and provides backup and disaster recovery solutions for tenant workloads. Trilio is very similar to other OpenStack services including Nova, Cinder, Glance, etc., and adheres to all tenets of OpenStack. It is a stateless service that scales with your cloud.

Main Components

Trilio has three main software components which are again segregated into multiple services.

WorkloadManager

This component is registered as a keystone service of type workloads which manages all the workloads being created for utilizing the snapshot and restore functionalities. It has 4 services responsible for managing these workloads, their snapshots, and restores.

1. workloadmgr-api

2. workloadmgr-scheduler

3. workloadmgr-workloads

4. workloadmgr-cron

DataMover

Similar to WorkloadManager, this component registers a keystone service of the type datamover which manages the transfer of extensive data to/from the backup targets. It has 2 services that are responsible for taking care of the data transfer and communication with WorkloadManager

1. datamover-api

2. datamover

Horizon Dashboard Plugin

For ease of access and better user experience, T4O provides an integrated UI with the OpenStack dashboard service Horizon,

1. Trilio API is a Python module that is installed on all OpenStack controller nodes where the nova-api service is running.

2. Trilio Datamover is a Python module that is installed on every OpenStack compute nodes

3. Trilio Horizon plugin is installed as an add-on to Horizon servers. This module is installed on every server that runs Horizon service.

Service Endpoints

Trilio is both a provider and consumer in the OpenStack ecosystem. It uses other OpenStack services such as Nova, Cinder, Glance, Neutron, and Keystone and provides its own services to OpenStack tenants. To accommodate all possible OpenStack deployments, Trilio can be configured to use either public or internal URLs of services. Likewise, Trilio provides its own public, internal, and admin URLs for two of its services WorkloadManager API and Datamover API.

Network Topology

Unlike the previous versions of Trilio for OpenStack, now it utilizes the existing network of the OpenStack deployed environment. The networks for the Trilio services can be configured as per the user's desire in the same way the user configures any other OpenStack service. Additionally, a dedicated network can be provided to Trilio services on both control and compute planes for storing and retrieving backup data from the backup target store.

Each T4O release includes a set of artifacts such as version-tagged containers, package repositories, and distribution packages.

To help users quickly identify the resources associated with each release, we have added dedicated sub-pages corresponding to a specific release version.

Cloud-Native and Containerized

T4O represents a comprehensively containerized deployment model, eliminating the necessity for any KVM-based appliances for service deployment. This marks a departure from earlier T4O releases, where such appliances were required.

Summary of Requirements

Trilio requires its containers to be deployed on the same plane as OpenStack, utilizing existing cluster resources.

As described in the , Trilio requires sufficient cluster resources to deploy its components on both the Controller Plane and Compute Planes.

1. Valid Trilio License & Acceptance of the

2. When deploying the Control Plane to OpenShift, ensure sufficient resources are available on Worker Nodes.

3. Sufficient storage capacity and connectivity on Cinder for snapshotting operations

4. Sufficient network capabilities for efficient data transfer of workloads

5. User and Role permissions for access to required cluster objects

6. Optional features may have specific requirements such as encryption, file search, snapshot mount, FRM instance, etc

7. Set hw_qemu_guest_agent=True property on the image and install qemu-guest-agent on the VM, in order to avoid any file system inconsistencies post restore.

8. For the VMware to OpenStack migration feature, please refer to the and pages.

Trilio for OpenStack Compatibility Matrix

NFS & S3 Support:

All versions of T4O-6.x releases support NFSv3 and S3 as backup targets on all the compatible distributions.

Encryption Support:

All versions of T4O-6.x releases support encryption using Barbican service on all the compatible distributions.

QEMU Guest Agent

Installing the QEMU guest agent is an optional but recommended service to run inside the VM to ensure backup data consistency with Trilio. QEMU Guest Agent is the standard OpenSource QEMU/KVM hypervisor agent that runs inside a virtual machine (VM) and communicates with the host system (the hypervisor) to provide enhanced management and control of the VM. It is an essential component of any hypervisor platform that OpenStack can use to provide enhanced functionality when required.

Enabling QEMU Guest Agent

To enable this feature, you must set hw_qemu_guest_agent=yes as a metadata parameter on the image you wish to use to create the guest-agent-capable instances from. Reference: https://docs.openstack.org/nova/pike/admin/configuration/hypervisor-kvm.html

If the Virtual Machine is already created from the glance image which doesn't have the hw_qemu_guest_agent=yes property set, guest agent can be enabled using following process:

The process is stopping the VM, manually adding the lines below to the virsh definition using "virsh edit ", and starting it again:

<channel type='unix'>
<target type='virtio' name='org.qemu.guest_agent.0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>

The VM the needs to be running the qemu-guest agent software:

Ubuntu/Debian: sudo apt install -y qemu-guest-agent && sudo systemctl enable qemu-guest-agent && sudo systemctl restart qemu-guest-agent && sudo systemctl status qemu-guest-agent --no-pager

RHEL/CentOS VMs: sudo yum install -y qemu-guest-agent && sudo systemctl enable qemu-guest-agent && sudo systemctl restart qemu-guest-agent && sudo systemctl status qemu-guest-agent --no-pager

Windows VMs: Inside the guest OS, install the guest agent tools from: https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win-guest-tools.exe. Finally, make sure the service is running and set to start on boot automatically, in PowerShell, as Administrator: Set-Service QEMU-GA -StartupType Automatic; Restart-Service QEMU-GA; Get-Service QEMU-GA

You can confirm the qemu-guest-agent is running properly after installed, from the compute host, like this:

virsh qemu-agent-command $INSTANCE '{"execute":"guest-ping"}'

The output must be: {"return":{}}

Software Driven Migration: VMware to OpenStack

Software Driven Migration is supported in T4O 5.X release. Please refer VM-Migration-T4O-5.X

Ceph is the most common OpenSource solution to provide block storage through OpenStack Cinder.

Ceph is a very flexible solution. The possibilities of Ceph require additional steps to the Trilio solution.

The Trilio solution is using the qcow2 backing file to provide full synthetic backups.

Especially when the NFS backup target is used, there are scenarios at which this backing file needs to be updated to a new mount path.

To make this process easier and streamlined Trilio is providing the following rebase tool.

6.1.2

Release Date : 3rd July'2025

What's New?

1. Support for T4O on MOSK.

2. Addition/Deletion of Backup Target (BT) is enabled without the need of T4O redeployment.

Known Issues

1. If any of the configured BT is slow/offline, then workload/snapshot creation slows down OR fails

   • Workaround:

     • If the impacted BT is default, change the default to other BT which doesn't seem impacted.

     • Delete the impacted BT from CLI.

     • Manually unmount from wlm, datamover and respective BT units/containers.

2. To update any BTT so as to make it default, modify command should NOT be used. Instead, user should use backup-target-set-default CLI command.

3. During T4O deployment, if S3 Immutable bucket is set as the default target, the UI goes into hang state.

   • Workaround:

     • Deploy T4O on OpenStack helm by keeping s3 immutable target as default, then add s3 mutable/NFS as dynamic target. Later, change the s3 mutable(dynamic target) target to default.

4. workload-reassign-workloads with --source-btt-all cli option doesn't work as expected

   • Workaround:

     • User can use --source-btt option to reassign the workloads from provided backup target type. Sample command : workloadmgr workload-reassign-workloads --old_tenant_ids 71637e3b98434ceeb5158087074af4aa --new_tenant_id 8e325e1056c94625854b3dffb4d1e64e --user_id 3a446f94fb57448a8c99c08238df0f13 --migrate_cloud True --source-btt 979a459d-26f9-4aa9-8f0d-21ca7a1f6b4f

6.1.1

Release Date : 21st May'2025

What's New?

1. Support for T4O on OpenStack Helm.

Known Issues

1. [OpenStack Helm] During T4O deployment, if S3 Immutable bucket is set as the default target, the UI goes into hang state.

   • Workaround:

     • Deploy T4O on OpenStack helm by keeping s3 immutable target as default, then add s3 mutable/NFS as dynamic target. Later, change the s3 mutable(dynamic target) target to default.

2. workload-reassign-workloads with --source-btt-all cli option doesn't work as expected

   • Workaround:

     • User can use --source-btt option to reassign the workloads from provided backup target type. Sample command : workloadmgr workload-reassign-workloads --old_tenant_ids 71637e3b98434ceeb5158087074af4aa --new_tenant_id 8e325e1056c94625854b3dffb4d1e64e --user_id 3a446f94fb57448a8c99c08238df0f13 --migrate_cloud True --source-btt 979a459d-26f9-4aa9-8f0d-21ca7a1f6b4f

6.1.0

Release Date : 10th April'2025

What's New?

1. Support for T4O on RHOSO18.0.

2. Support for Dynamic Backup Target addition without the need of redeployment of T4O.

3. Support for T4O on OpenStack Helm.

Known Issues

1. trilio-openstack-operator-rhoso images are not available on Operator Hub

   • Workaround:

     • To install from cli using install script as mentioned in the deployment doc.

2. workload-reassign-workloads with --source-btt-all cli option doesn't work as expected

   • Workaround:

     • User can use --source-btt option to reassign the workloads from provided backup target type. Sample command : workloadmgr workload-reassign-workloads --old_tenant_ids 71637e3b98434ceeb5158087074af4aa --new_tenant_id 8e325e1056c94625854b3dffb4d1e64e --user_id 3a446f94fb57448a8c99c08238df0f13 --migrate_cloud True --source-btt 979a459d-26f9-4aa9-8f0d-21ca7a1f6b4f

6.0.0

Release Date : 18th Jan'2025

What's New?

1. Deployment of Trilio on RHOCP (with RHOSP17.1).

2. Support for multiple target backends.

3. Support for advanced scheduling of workloads' snapshots.

4. Support for Immutable Backups on S3 target backend.

5. Support for Canonical Yoga, 2023.1(Antelope) & 2023.2(Bobcat) on Jammy.

Known Issues

1. T4O upgrade on Canonical OpenStack is not supported

2. Restore operations fail for image-booted instance after changing the image from public to private

   • Workaround:

     • Make the image public or accessible to the project/user.

3. Select All option does not work on Backup Target Types page

4. Sorting the columns on the backup's admin page is not working

5. Workload Policy field is missing in the workload creation wizard until the scheduler is enabled

6. In the workload-reassign-workloads help text --new_tenant_ids is provided instead of --new_tenant_id

7. [Canonical multiple Backup Target] If NFS Backup Target getting deployed before S3 Backup Targets, then object store services against S3 Backup Target go into error state

   • If the overlay bundle is having multiple Backup Target Types defined such that at least one is NFS and one is S3 and if deployment of NFS happens first, then it is noticed that NFS mount happens successfully whereas object store services against S3 Backup Target Types start failing. Error An error occured (403) when calling HeadBucket operation: Forbidden is observed.

   • Workaround:

     • manually unmount the NFS mountpoint. All required S3 object store services will start successfully along with mounting of S3 sharepath. Unmounted NFS also gets remounted within 2-3 min (via mtab).

8. workload modify CLI command with workload_id at end is throwing an error

   • Workaround:

     • Place the workload_id before the optional arguments.

     • Sample command workloadmgr workload-modify 858da6e0-5b55-4c63-9a8e-d9f116130686 --jobschedule enabled=True --jobschedule start_time='07:00 AM' --jobschedule start_date='01/08/2025' --hourly interval='1' retention='2' snapshot_type='incremental'

Trilio employs a base64 hash to establish the mount point for NFS Backup targets, ensuring compatibility across multiple NFS Shares within a single Trilio installation. This hash is an integral component of Trilio's incremental backups, functioning as an absolute path for backing files.

Consequently, during a disaster recovery or rapid migration situation, the utilization of a mount bind becomes necessary.

In scenarios that allow for a comprehensive migration period, an alternative approach comes into play. This involves modifying the backing file, thereby enabling the accessibility of Trilio backups from a different NFS Share. The backing file is updated to correspond with the mount point of the new NFS Share.

Backing file change script

Trilio provides a shell script for the purpose of changing the backing file. This script is used after the Trilio appliance has been reconfigured to use the new NFS share.

Downloading the shell script

Prerequisites

The following requirements need to be met before the change of the backing file can be attempted.

• The Trilio Appliance has been reconfigured with the new NFS Share

• The OpenStack environment has been reconfigured with the new NFS Share

  • Please check here for Red Hat OpenStack Platform

• The workloads are available on the new NFS Share

• The workloads are owned by nova:nova user

Usage

The shell script is changing one workload at a time.

Run the following command:

./backing_file_update.sh /var/triliovault-mounts/<base64>/workload_<workload_id>

with

• /var/triliovault-mounts/<base64>/ being the new NFS mount path

• workload_<workload_id> being the workload to rebase

Logging of the procedure

The shell script is generating the following log file at the following location:

/tmp/backing_file_update.log

The log file will not get overwritten when the script is run multiple times. Each run of the script will append the available log file.

By default, during a snapshot, Trilio will consider all the instances that are members of a workload and starts uploading the resources of each instance simulateously. In some cases, it may be required that the snapshots must be done in serial per VM such that backup of all the disks of one VM/instance must be finished before taking backup of the other VMs that are part of that workload.

Trilio has a provision to update the default behaviour of considering all the instances of a workload during backup process, by introducing a configrable parameter in the on of its services.

Follow the below mentioned steps to set the serial upload as the default behavior.

Finding the required configuration file

As per the OpenStack distribution, look for the configuration file of workloadmgr-api(wlm-api) and workloadmgr-workloads(wlm-workloads) services or containers on the controller plane.

The default location of this configration file in the container or VM where it is running is /etc/triliovault/triliovault-wlm.conf

Once the file is located, take a backup of the file in-case of future reference.

Updating the configuration file

Once the files are located, add the below mentioned parameter with its value to the [DEFAULT] section of the configuration file. serial_vm_backup=true

After successfully updating the file, restart the services or containers and these steps need to be followed on all the controller plane.

Trilio is using the OpenStack Keystone Trust system which enables the Trilio service user to act in the name of another OpenStack user.

This system is used during all backup and restore features.

Trusts can only be worked with via CLI

List all trusts

workloadmgr trust-list

Show a trust

workloadmgr trust-show <trust_id>

• <trust_id> ➡️ ID of the trust to show

Create a trust

workloadmgr trust-create [--is_cloud_trust {True,False}] <role_name>

• <role_name> ➡️Name of the role that trust is created for

• --is_cloud_trust {True,False} ➡️ Set to true if creating cloud admin trust. While creating cloud trust use same user and tenant which used to configure Trilio and keep the role admin.

Delete a trust

workloadmgr trust-delete <trust_id>

• <trust_id> ➡️ ID of the trust to be deleted

After all Trilio components are installed, the license can be applied.

The license can be applied either through the Admin tab in Horizon or the CLI

Apply license through Horizon

To apply for the license through Horizon follow these steps:

1. Login to Horizon using admin user.

2. Click on the Admin Tab.

3. Navigate to Backups-Admin

4. Navigate to Trilio

5. Navigate to License

6. Click "Update License"

7. Read the license agreement

8. Click on "I accept the terms in the License Agreement"

9. Click on "Next"

10. Click "Choose File"

11. Choose license file on the client system

12. Click "Apply"

Apply license through CLI

workloadmgr license-create <license_file>

• <license_file> ➡️ path to the license file

Read and accept the End User License Agreement to complete the license application.

Users can preview the latest EULA at our main site: https://trilio.io/eula/

The following versions can be upgraded to each other:

1] Uninstall trilio-openstack helm chart

Run following commands to uninstall trilio services from MOSK cloud.

git clone -b <branch> https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/
cd openstack-helm/trilio-openstack/utils
./uninstall.sh

2] Uninstall Dynamic Backup Targets (Optional)

If you have installed multiple backup targets other than the default one, you can uninstall them using:

helm uninstall <dynamic-backup-target-name> -n trilio-openstack

3] Verify uninstallation

Verify if the trilio-openstack chart and other components got uninstalled. No objects should be listed in following command output:

kubectl get pods -n trilio-openstack
kubectl get jobs -n trilio-openstack
kubectl get pv -n trilio-openstack | grep nfs (Only if backup target was NFS)
kubectl get pvc -n trilio-openstack | grep nfs (Only if backup target was NFS)

4] Clean TrilioVault’s OpenStack resources

openstack service list | grep -E 'TrilioVaultWLM|dmapi'
openstack endpoint list | grep -E 'TrilioVaultWLM|dmapi'
# Delete TrilioVault services from keystone catalog
openstack service delete TrilioVaultWLM
openstack service delete dmapi
# Verify TrilioVault services and endpoints got cleaned. Following command output should be empty.
openstack service list | grep -E 'TrilioVaultWLM|dmapi'
openstack endpoint list | grep -E 'TrilioVaultWLM|dmapi'
# Login to database and clean TrilioVault's databases and db users.
MYSQL_DBADMIN_PASSWORD=`kubectl -n openstack get secrets/mariadb-dbadmin-password --template={{.data.MYSQL_DBADMIN_PASSWORD}} | base64 -d`
kubectl -n openstack exec -it mariadb-server-0 -- bash
mysql -u root -p${MYSQL_DBADMIN_PASSWORD} -e "REVOKE ALL PRIVILEGES, GRANT OPTION FROM 'dmapi'@'%%';"
mysql -u root -p${MYSQL_DBADMIN_PASSWORD} -e "DROP USER 'dmapi'@'%%';"
mysql -u root -p${MYSQL_DBADMIN_PASSWORD} -e "DROP DATABASE dmapi;"
mysql -u root -p${MYSQL_DBADMIN_PASSWORD} -e "REVOKE ALL PRIVILEGES, GRANT OPTION FROM 'workloadmgr'@'%%';"
mysql -u root -p${MYSQL_DBADMIN_PASSWORD} -e "DROP USER 'workloadmgr'@'%%';"
mysql -u root -p${MYSQL_DBADMIN_PASSWORD} -e "DROP DATABASE workloadmgr;"

5] Unmount the targets from All Worker and Master Nodes

To unmount the backup target mount points from each node, run:

umount /var/lib/trilio/triliovault-mounts/<base64>

Repeat this on each worker and master node where the target was mounted.

Trilio services logs on RHOSP

On the Controller Node

Below Trilio containers gets deployed on the Contoller node.

• triliovault_datamover_api

• triliovault_wlm_api

• triliovault_wlm_scheduler

• triliovault_wlm_workloads

• triliovault-wlm-cron

The log files for above Trilio services can be found here:

/var/log/containers/triliovault-datamover-api/triliovault-datamover-api.log

/var/log/containers/triliovault-wlm-api/triliovault-wlm-api.log

/var/log/containers/triliovault-wlm-cron/triliovault-wlm-cron.log

/var/log/containers/triliovault-wlm-scheduler/triliovault-wlm-scheduler.log

/var/log/containers/triliovault-wlm-workloads/triliovault-wlm-workloads.log

In the case of using S3 as a backup target is there also a log file that keeps track of the S3-Fuse plugin used to connect with the S3 storage.

/var/log/containers/triliovault-wlm-workloads/triliovault-object-store.log

For file serach operation, logs can be found on Controller node at below location:

/var/log/containers/triliovault-wlm-workloads/workloadmgr-filesearch.log

On the Compute Node

Trilio Datamover container gets deployed on the Compute node.

The log file for the Trilio Datamover service can be found at:

/var/log/containers/triliovault-datamover/triliovault-datamover.log

Troubleshooting inside a complex environment like OpenStack can be very time-consuming.

The following tips will help to speed up the troubleshooting process to identify root causes.

What is happening where

OpenStack and Trilio are divided into multiple services. Each service has a very specific purpose that is called during a backup or recovery procedure. Knowing which service is doing what helps to understand where the error is happening, allowing more focused troubleshooting.

Trilio Workloadmgr

The Trilio Workloadmgr is the Controller of Trilio. It receives all Workload related requests from the users.

Every task of a backup or restore process is triggered and managed from here. This includes the creation of the directory structure and initial metadata files on the Backup Target.

During a backup process

During a backup process is the Trilio Workloadmgr also responsible for gathering the metadata about the backed-up VMs and networks from the OpenStack environment. It sends API calls to the OpenStack endpoints on the configured endpoint type to fetch this information. Once the metadata has been received, the Trilio Workloadmgr writes it as JSON files on the Backup Target.

The Trilio Workloadmgr is also sending the Cinder Snapshot command.

During a restore process

During the restore process, the Trilio Workloadmgr reads the VM metadata from its Database and uses the metadata to create the Shell for the restore. It sends API calls to the OpenStack environment to create the necessary resources.

Datamover API

The dmapi service is the connector between the Trilio cluster and the Datamover running on the compute nodes.

The purpose of the dmapi service is to identify which compute node is responsible for the current backup or restore task. To do so, the dmapi service connects to the nova API requesting the compute hose of a provided VM.

Once the compute host has been identified, the dmapi forwards the command from the Trilio Workloadmgr to the datamover running on the identified compute host.

Datamover

The datamover is the Trilio service running on the compute nodes.

Each datamover is responsible for the VMs running on top of its compute node. A datamover can not work with VMs running on a different compute node.

The datamover controls the freeze and thaw of VMs as well as the actual movement of the data.

Everything on the Backup Target is happening as user nova

Trilio is reading and writing on the Backup Target as nova:nova.

The POSIX user-id and group-id of nova:nova need to be aligned between the Trilio Cluster and all compute nodes. Otherwise, backup or restores may fail with permission or file not found issues.

Alternativ ways to achieve the goal are possible, as long as all required nodes can fully write and read as nova:nova on the Backup Target.

It is recommended to verify the required permissions on the Backup Target in case of any errors during the data transfer phase or in case of any file permission errors.

Input/Output Error with Cohesity NFS

On Cohesity NFS if an Input/Output error is observed, then increase the timeo and retrans parameter values in your NFS options.

Timeout receiving packet error in multipath ISCSi environment

Logging inside all datamover containers and add uxsock_timeout with value as 60000 which is equal to 60 sec inside /etc/multipath.conf.Restart datamover container

Trilio Trustee Role

Trilio is using RBAC to allow the usage of Trilio features to users.

This trustee role is absolutely required and can not be overwritten using the admin role.

It is recommended to verify the assignment of the Trilio Trustee Role in case of any permission errors from Trilio during the creation of Workloads, backups, or restores.

OpenStack Quotas

Trilio is creating Cinder Snapshots and temporary Cinder Volumes. The OpenStack Quotas need to allow that.

Every disk that is getting backed up requires one temporary Cinder Volumes.

Every Cinder Volume that is getting backup requires two Cinder Snapshots. The second Cinder Snapshot is temporary to calculate the incremental.

The high-level process is the same for all Distributions.

1. Uninstall the Horizon Plugin or the Trilio Horizon container

2. Uninstall the datamover-api container

3. Uninstall the datamover container

4. Uninstall the workloadmgr container

1. Can Trilio for OpenStack restore instance UUIDs?

Answer: NO

Trilio for OpenStack does not restore Instance UUIDs (also known as Instance IDs). The only scenario where we do not modify the Instance UUID is during an Inplace Restore, where we only recover the data without creating new instances.

When Trilio for OpenStack restores virtual machines (VMs), it effectively creates new instances. This means that new Virtual Machine Instance UUIDs are generated for the restored VMs. We achieve this by orchestrating a call to Nova, which creates new VMs with new UUIDs.

By following this approach, we maintain the principles of OpenStack and auditing. We do not update or modify existing database entries when objects are deleted and subsequently recovered. Instead, all deletions are marked as such, and new instances, including the recovered ones, are created as new objects in the Nova tables. This ensures compliance and preserves the integrity of the OpenStack environment.

2. Can Trilio for OpenStack restore MAC addresses?

Answer: YES

Trilio can restore the VMs MAC address, however, there is a caveat when restoring a virtual machine (VM) to a different IP address: a new MAC address will be assigned to the VM.

In the case of a One-Click Restore, the original MAC addresses and IP addresses will be recovered, but the VM will be created with a new UUID, as mentioned in question #1.

When performing a Selective Restore, you have the option to recover the original MAC address. To do so, you need to select the original IP address from the available dropdown menu during the recovery process.

By choosing the original IP address, Trilio for OpenStack will ensure that the VM is restored with its original MAC address, providing more flexibility and customization in the restoration process.

Example of Selective Restore with original MAC (and IP address):

1. In this example, we have taken a Trilio backup of a VM called prod-1.

1. The VM is deleted and we perform a Selective Restore of a VM called prod-1, selecting the IP address it was originally assigned from the drop-down menu:

1. Trilio then restores the VM with the original MAC address:

1. If you left the option as "Choose next available IP address", it will assign a new MAC to the VM instead as Neutron maps all MAC addresses to IP addresses on the Subnet - so logically a new IP will result in a new MAC address.

Definition

Trilio can notify users via E-Mail upon the completion of backup and restore jobs.

The E-Mail will be sent to the owner of the Workload.

Requirements to activate E-Mail Notifications

To use the E-mail notifications, two requirements need to be met.

User E-Mail assigned

As the E-Mail will be sent to the owner of the Workload does the OpenStack User, who created the workload, require to have an E-Mail address associated.

Trilio E-Mail Server configured

Trilio needs to know which E-Mail server to use, to send the E-mail notifications. Backup Administrators can do this in the "Backup Admin" area.

Activate/Deactivate the E-Mail Notifications

E-Mail notifications are activated tenant wide. To activate the E-Mail notification feature for a tenant follow these steps:

1. Login to Horizon

2. Navigate to the Backups

3. Navigate to Settings

4. Check/Uncheck the box for "Enable Email Alerts"

Example E-Mails

The following screenshots show example E-mails send by Trilio.

Trilio Branch

Branch of the repository to be used for DevOps scripts.

Deployment Scripts

Container/Revision Information

Package Repositories & Versions

All packages are Python 3 (py3.6 - py3.10) compatible only.

Trilio Branch

Branch of the repository to be used for DevOps scripts.

Deployment Scripts

Container/Revision Information

Package Repositories & Versions

All packages are Python 3 (py3.6 - py3.10) compatible only.

Trilio Workloads are designed to allow a Disaster Recovery without the need to backup the Trilio database.

As long as the Trilio Workloads are existing on the Backup Target Storage and a Trilio installation has access to them, it is possible to restore the Workloads.

Disaster Recovery Process

1. and Trilio for the target cloud

4. Notify users to of Workloads being available

Mount-paths

Trilio incremental Snapshots involve a backing file to the prior backup taken, which makes every Trilio incremental backup a synthetic full backup.

Trilio is using qcow2 backing files for this feature:

As can be seen in the example is the backing file an absolute path, which makes it necessary, that this path exists so the backing files can be accessed.

Trilio is using the base64 hashing algorithm for the NFS mount-paths, to allow the configuration of multiple NFS Volumes at the same time. The hash value is calculated using the provided NFS path.

When the path of the backing file is not available on the Trilio appliance and Compute nodes, will the restores of incremental backups fail.

The tested and recommended method to make the backing files available is creating the required directory path and using mount --bind to make the path available for the backups.

Running the mount --bind command will make the necessary path available until the next reboot. If it is required to have access to the path beyond a reboot is it necessary to edit the fstab.

Same cloud - different owner

Migration within the same cloud to a different owner Cloud A — Domain A — Project A — User A => Cloud A — Domain A — Project A — User B Cloud A — Domain A — Project A — User A => Cloud A — Domain A — Project B — User B Cloud A — Domain A — Project A — User A =>Cloud A — Domain B — Project B — User B

Steps used:

1. Create a secret for Project A in Domain A via User A.

2. Create encrypted workload in Project A in Domain A via User A. Take snapshot.

3. Reassign workload to new owner

4. Load rc file of User A & provide read only rights through acl to the new owner

   openstack acl user add --user <userB_id> <secret_href> --insecure

Different cloud

Migration between clouds Cloud A — Domain A — Project A — User A => Cloud B — Domain B — Project B — User B

Steps used:

1. Create a secret for Project A in Domain A via User A.

2. Create an encrypted workload in Project A in Domain A via User A. Trigger snapshot.

3. Reassign workload to Cloud B - Domain B — Project B — User B

4. Load RC file of User B.

5. Create a secret for Project B in Domain B via User B with the same payload used in Cloud A.

6. Create token via “openstack token issue --insecure”

7. Add migrated workload's metadata to the new secret (provide issued token to Auth-Token & workload id to matadata as below)

1] Prerequisites

Please ensure the following requirements are met before starting the upgrade process:

• No Snapshot or Restore is running

• Global job scheduler is disabled

• Disable the wlm services on all controller nodes

• Disable the datamover service on all compute nodes

2] Install latest Trilio release

Take a backup of existing triliovault-cfg-scripts directory

Clone triliovault-cfg-scripts github repository of latest release.

Copy the passwords and triliovault wlm ids files from old cfg-scripts directory.

Please follow from section 1.4 to section 10. Please note that sections 5 and 6 needs to be skipped during this process.

While providing the backup target details in section 4.1, please ensure to provide the existing backup target (of 5.x release) as default.

3] Restart wlm-cron resource

From any one of the controller nodes, run the below command.

4] Fetch backup target details

From any one of the controller nodes, login to wlm api container and run the below commands to fetch the backup target details.

This command will return the backup_target_id and backup_target_type_id.

5] Import workloads

From any one of the controller ndoes, login to wlm api container and run the below commands.

6] Update backing file path

Please follow to update the backing file path of existing snapshots.

1] Configuration change

If any config parameter changed in tvo-operator-inputs.yaml like db user password or service endpoints, you can apply the changes using following command.

Above command will output ‘configured' or 'unchanged’ depending upon changes happened in tvo-operator-inputs.yaml.

2] Upgrade to new build

Please follow below steps to upgrade to new build on RHOSO18 setup.

Take a backup of existing triliovault-cfg-scripts and clone latest triliovault-cfg-scripts github repository.

Manually copy the input values from triliovault-cfg-scripts-old to latest directory.

2.1] Upgrade Trilio for OpenStack Operator

Run operator deployment with new image tag as mentioned in

2.2] Upgrade Trilio OpenStack Control Plane Services

Update the image tags in tvo-operator-inputs.yaml as mentioned in and apply the changes using below command

Verify the deployment status and successful deployment.

2.3] Upgrade Trilio Data Plane Services

Update the image tags in cm-trilio-datamover.yaml and apply the changes as mentioned in

Update the ansible runner tag in trilio-datamover-service.yaml and apply the changes as mentioned in

Update the deployment name and trigger deployment as mentioned in

Verify the deployment as mentioned in

2.4] Upgrade Trilio Horizon Plugin

Follow and update the trilio horizon plugin image tag.

It is possible to configure Cinder and Ceph to use different Ceph users for different Ceph pools and Cinder volume types. Or to have the nova boot volumes and cinder block volumes controlled by different users.

In the case of multiple Ceph users, it is required to delete the keyring extension from the triliovault-datamover.conf inside the Ceph block by following below mentioned steps:

1. Deploy Trilio as per the documented steps.

2. Post successful deployment, please modify the triliovault-datamover.conf file present at following locations on all compute nodes.

   For RHOSP : /var/lib/config-data/puppet-generated/triliovaultdm/etc/triliovault-datamover/

   For Kolla : /etc/kolla/triliovault-datamover/

3. Modify keyring_ext value with valid keyring extension (eg. .keyring). This extension is expected to be same for all the keyring files. It will be present under [ceph] block in triliovault-datamover.conf file.

Sample conf entry below. This will try all files with the extension keyring that are located inside /etc/ceph to access the Ceph cluster for a Trilio related task.

1. Restart triliovault_datamover container on all compute nodes.

About the workloadmgr CLI client

The workloadmgr CLI client is provided as rpm and deb packages.

The following operating systems have been verified with the installation:

• Rocky9, RHEL9

• CentOS8

• Ubuntu 18.04, Ubuntu 20.04

Installing the workloadmgr client will automatically install all required OpenStack clients as well.

Further, the installation of the workloadmgr client will integrate the client into the global OpenStack Python client, if available.

Install workloadmgr client rpm package on CentOS8

The Trilio workload manager CLI client has several requirements that need to be met before the client can be installed without dependency issues.

Preparing the workloadmgr client installation

The following steps need to be done to prepare the installation of the workloadmgr client:

1. Add required repositories

   1. epel-release yum -y install epel-release

   2. centos-release-openstack-train yum -y install centos-release-openstack-train

Installing the workloadmgr client

Installing from the Trilio online repository

To install the latest available workloadmgr package for a Trilio release from the Trilio repository the following steps need to be done:

Create the Trilio yum repository file /etc/yum.repos.d/trilio.repo Enter the following details into the repository file:

[trilio]
name=Trilio Repository
baseurl=https://yum.fury.io/trilio-6-0/
enabled=1
gpgcheck=0

Install the workloadmgr client issuing the following command:

yum install python-3-workloadmgrclient-el8

Install workloadmgr client rpm package on Rocky9 and RHEL9

The Trilio workload manager CLI client has several requirements that need to be met before the client can be installed without dependency issues.

Preparing the workloadmgr client installation

The following steps need to be done to prepare the installation of the workloadmgr client:

1. Add required repositories

   1. epel-release yum -y install epel-release

Installing the workloadmgr client

Installing from the Trilio online repository

To install the latest available workloadmgr package for a Trilio release from the Trilio repository the following steps need to be done:

Create the Trilio yum repository file /etc/yum.repos.d/trilio.repo Enter the following details into the repository file:

[trilio]
name=Trilio Repository
baseurl=https://yum.fury.io/trilio-6-0/
enabled=1
gpgcheck=0

Install the workloadmgr client issuing the following command:

yum install python-3-workloadmgrclient-el9

Install workloadmgr client deb packages on Ubuntu

The Trilio workloadmgr client packages for Ubuntu are only available from the online repository.

Preparing the workloadmgr client installation

There is no preparation required. All dependencies are automatically resolved by the standard repositories provided by Ubuntu.

Installing the Workloadmgr client

Installing from the Trilio online repository

To install the latest available workloadmgr package for a Trilio release from the Trilio repository the following steps need to be done:

Create the Trilio yum repository file /etc/apt/sources.list.d/fury.list Enter the following details into the repository file:

deb [trusted=yes] https://apt.fury.io/trilio-6-0/ /

run apt update to make the new repository available.

The apt package manager is used to install the workloadmgr client package:

apt-get install python3-workloadmgrclient

After the installation and configuration of Trilio for OpenStack did succeed the following steps can be done to verify that the Trilio installation is healthy.

On the Controller node

Make sure below containers are in a running state, triliovault-wlm-cron would be running on only one of the controllers in case of multi-controller setup.

• triliovault_datamover_api

• triliovault_wlm_api

• triliovault_wlm_scheduler

• triliovault_wlm_workloads

• triliovault-wlm-cron

If the containers are in a restarting state or not listed by the following command then your deployment is not done correctly. Please recheck if you followed the complete documentation.

[root@overcloudtrain5-controller-0 /]# podman ps  | grep trilio-
76511a257278  undercloudqa.ctlplane.trilio.local:8787/trilio/trilio-horizon-plugin:<CONTAINER-TAG-VERSION>-rhosp17.1          kolla_start           12 days ago   Up 12 days ago           horizon
5c5acec33392  cluster.common.tag/trilio-wlm:pcmklatest                                                          /bin/bash /usr/lo...  7 days ago    Up 7 days ago            triliovault-wlm-cron-podman-0
8dc61a674a7f  undercloudqa.ctlplane.trilio.local:8787/trilio/trilio-datamover-api:<CONTAINER-TAG-VERSION>-rhosp17.1           kolla_start           7 days ago    Up 7 days ago            triliovault_datamover_api
a945fbf80554  undercloudqa.ctlplane.trilio.local:8787/trilio/trilio-wlm:<CONTAINER-TAG-VERSION>-rhosp17.1                     kolla_start           7 days ago    Up 7 days ago            triliovault_wlm_scheduler
402c9fdb3647  undercloudqa.ctlplane.trilio.local:8787/trilio/trilio-wlm:<CONTAINER-TAG-VERSION>-rhosp17.1                     kolla_start           7 days ago    Up 6 days ago            triliovault_wlm_workloads
f9452e4b3d14  undercloudqa.ctlplane.trilio.local:8787/trilio/trilio-wlm:<CONTAINER-TAG-VERSION>-rhosp17.1                     kolla_start           7 days ago    Up 6 days ago            triliovault_wlm_api

After successful deployment, triliovault-wlm-cron service would get added in pcs cluster as a cluster resource, you can verify through pcs status command

[root@overcloudtrain5-controller-0 /]# pcs status
Cluster name: tripleo_cluster
Cluster Summary:
  * Stack: corosync
  * Current DC: overcloudtrain5-controller-0 (version 2.0.5-9.el8_4.3-ba59be7122) - partition with quorum
  * Last updated: Mon Jul 24 11:19:05 2023
  * Last change:  Mon Jul 17 10:38:45 2023 by root via cibadmin on overcloudtrain5-controller-0
  * 4 nodes configured
  * 14 resource instances configured

Node List:
  * Online: [ overcloudtrain5-controller-0 ]
  * GuestOnline: [ galera-bundle-0@overcloudtrain5-controller-0 rabbitmq-bundle-0@overcloudtrain5-controller-0 redis-bundle-0@overcloudtrain5-controller-0 ]

Full List of Resources:
  * ip-172.30.6.27      (ocf::heartbeat:IPaddr2):        Started overcloudtrain5-controller-0
  * ip-172.30.6.16      (ocf::heartbeat:IPaddr2):        Started overcloudtrain5-controller-0
  * Container bundle: haproxy-bundle [cluster.common.tag/openstack-haproxy:pcmklatest]:
    * haproxy-bundle-podman-0   (ocf::heartbeat:podman):         Started overcloudtrain5-controller-0
  * Container bundle: galera-bundle [cluster.common.tag/openstack-mariadb:pcmklatest]:
    * galera-bundle-0   (ocf::heartbeat:galera):         Master overcloudtrain5-controller-0
  * Container bundle: rabbitmq-bundle [cluster.common.tag/openstack-rabbitmq:pcmklatest]:
    * rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster):       Started overcloudtrain5-controller-0
  * Container bundle: redis-bundle [cluster.common.tag/openstack-redis:pcmklatest]:
    * redis-bundle-0    (ocf::heartbeat:redis):  Master overcloudtrain5-controller-0
  * Container bundle: openstack-cinder-volume [cluster.common.tag/openstack-cinder-volume:pcmklatest]:
    * openstack-cinder-volume-podman-0  (ocf::heartbeat:podman):         Started overcloudtrain5-controller-0
  * Container bundle: triliovault-wlm-cron [cluster.common.tag/trilio-wlm:pcmklatest]:
    * triliovault-wlm-cron-podman-0     (ocf::heartbeat:podman):         Started overcloudtrain5-controller-0

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

Verify the HAproxy configuration under:

/var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg

On Compute node

Make sure the Trilio Datamover container is in running state and no other Trilio container is deployed on compute nodes.

[root@overcloudtrain5-novacompute-0 heat-admin]# podman  ps | grep -i datamover
c750a8d0471f  undercloudqa.ctlplane.trilio.local:8787/trilio/trilio-datamover:<CONTAINER-TAG-VERSION>-rhosp17.1              kolla_start  7 days ago   Up 7 days ago           triliovault_datamover

Check if provided backup target is mounted well on Compute host.

[root@overcloudtrain5-novacompute-0 heat-admin]# df -h  | grep triliovault-mounts
172.30.1.9:/mnt/rhosptargetnfs  7.0T  5.1T  2.0T  72% /var/lib/nova/triliovault-mounts/L21udC9yaG9zcHRhcmdldG5mcw==

On the node with Horizon service

Make sure the horizon container is in a running state. Please note that the Horizon container is replaced with Trilio's Horizon container. This container will have the latest OpenStack horizon + Trilio horizon plugin.

[root@overcloudtrain5-controller-0 heat-admin]# podman ps  | grep horizon
76511a257278  undercloudqa.ctlplane.trilio.local:8787/trilio/trilio-horizon-plugin:<CONTAINER-TAG-VERSION>-rhosp17.1          kolla_start           12 days ago   Up 12 days ago           horizon

Introduction

Filename and location:

triliovault-cfg-scripts/common/triliovault_nfs_map_input.yml

This file exclusively comes into play when users aim to configure Trilio with a NFS backup target that employs multiple network endpoints. For all other scenarios, such as single IP NFS or S3, this file remains inactive, and in such instances, please consult the standard installation documentation.

When using an NFS backup target with multiple network endpoints, T4O will mount a single IP/endpoint on a designated compute node for a specific NFS share. This approach enables users to distribute NFS share IPs/endpoints across various compute nodes.

The 'triliovault_nfs_map_input.yml' file allows users to distribute/load balance NFS share endpoints across compute nodes in a given cloud.

Examples

Using hostnames

Here, the User has ‘one’ NFS share exposed with three IP addresses. 192.168.1.34, 192.168.1.35, 192.168.1.33 Share directory path is: /var/share1

So, this NFS share supports the following full paths that clients can mount:

192.168.1.33:/var/share1 
192.168.1.34:/var/share1 
192.168.1.35:/var/share1

There are 32 compute nodes in the OpenStack cloud. 30 node hostnames have the following naming pattern

prod-compute-1.trilio.demo 
prod-compute-2.trilio.demo 
prod-compute-3.trilio.demo 
. 
. 
. 
prod-compute-30.trilio.demo

The remaining 2 node hostnames do not follow any format/pattern.

compute_bare.trilio.demo 
compute_virtual

Now the mapping file will look like this

multi_ip_nfs_shares: 
 - "192.168.1.34:/var/share1": ['prod-compute-[1:10].trilio.demo', 'compute_bare.trilio.demo'] 
   "192.168.1.35:/var/share1": ['prod-compute-[11:20].trilio.demo', 'compute_virtual'] 
   "192.168.1.33:/var/share1": ['prod-compute-[21:30].trilio.demo'] 

single_ip_nfs_shares: []

Using IPs

Compute node IP range used here: 172.30.3.11-40 and 172.30.4.40, 172.30.4.50 Total of 32 compute nodes

multi_ip_nfs_shares: 
 - "192.168.1.34:/var/share1": ['172.30.3.[11:20]', '172.30.4.40'] 
   "192.168.1.35:/var/share1": ['172.30.3.[21:30]', '172.30.4.50'] 
   "192.168.1.33:/var/share1": ['172.30.3.[31:40]'] 

single_ip_nfs_shares: []

Getting the correct compute hostnames/IPs

RHOSP

Use the following command to get compute hostnames. Check the ‘Name' column. Use these exact hostnames in 'triliovault_nfs_map_input.yml' file.

In the following command output, ‘overcloudtrain1-novacompute-0' and ‘overcloudtrain1-novacompute-1' are correct hostnames.

(undercloud) [stack@ucqa161 ~]$ openstack server list
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+
| ID                                   | Name                          | Status | Networks             | Image          | Flavor  |
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+
| 8c3d04ae-fcdd-431c-afa6-9a50f3cb2c0d | overcloudtrain1-controller-2  | ACTIVE | ctlplane=172.30.5.18 | overcloud-full | control |
| 103dfd3e-d073-4123-9223-b8cf8c7398fe | overcloudtrain1-controller-0  | ACTIVE | ctlplane=172.30.5.11 | overcloud-full | control |
| a3541849-2e9b-4aa0-9fa9-91e7d24f0149 | overcloudtrain1-controller-1  | ACTIVE | ctlplane=172.30.5.25 | overcloud-full | control |
| 74a9f530-0c7b-49c4-9a1f-87e7eeda91c0 | overcloudtrain1-novacompute-0 | ACTIVE | ctlplane=172.30.5.30 | overcloud-full | compute |
| c1664ac3-7d9c-4a36-b375-0e4ee19e93e4 | overcloudtrain1-novacompute-1 | ACTIVE | ctlplane=172.30.5.15 | overcloud-full | compute |
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+

Definition

Every Workload has its own schedule. Those schedules can be activated, deactivated and modified.

A schedule is defined by:

• Status (Enabled/Disabled)

• Start Day/Time

• End Day

• Hrs between 2 snapshots

Disable a schedule

Using Horizon

To disable the scheduler of a single Workload in Horizon do the following steps:

1. Login to the Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload to be modified

5. Click the small arrow next to "Create Snapshot" to open the sub-menu

6. Click "Edit Workload"

7. Navigate to the tab "Schedule"

8. Uncheck "Enabled"

9. Click "Update"

Using CLI

workloadmgr disable-scheduler --workloadids <workloadid>

• --workloadid <workloadid> ➡️ Requires at least one workloadid, Specify an ID of the workload whose scheduler disables. Specify option multiple times to include multiple workloads. --workloadids <workloadid> --workloadids <workloadid>

Enable a schedule

Using Horizon

To disable the scheduler of a single Workload in Horizon do the following steps:

1. Login to the Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload to be modified

5. Click the small arrow next to "Create Snapshot" to open the sub-menu

6. Click "Edit Workload"

7. Navigate to the tab "Schedule"

8. check "Enabled"

9. Click "Update"

Using CLI

workloadmgr enable-scheduler --workloadids <workloadid>

• --workloadid <workloadid> ➡️ Requires at least one workloadid, Specify an ID of the workload whose scheduler disables. Specify option multiple times to include multiple workloads. --workloadids <workloadid> --workloadids <workloadid>

Modify a schedule

To modify a schedule the workload itself needs to be modified.

Please follow this procedure to modify the workload.

Verify the scheduler trust is working

Trilio is using the OpenStack Keystone Trust system which enables the Trilio service user to act in the name of another OpenStack user.

This system is used during all backup and restore features.

Using Horizon

As a trust is bound to a specific user for each Workload does the Trilio Horizon plugin show the status of the Scheduler on the Workload list page.

Using CLI

workloadmgr scheduler-trust-validate <workload_id>

• <workload_id> ➡️ ID of the workload to validate

Trilio Branch

Branch of the repository to be used for DevOps scripts.

Deployment Scripts

Container/Revision Information

Package Repositories & Versions

All packages are Python 3 (py3.6 - py3.10) compatible only.

Please follow the below steps to add new backup target on RHOSO18.

1] Mount Backup target on Trilio Control Plane

Navigate to trilio ctlplane-scripts directory

Plan which type of backup target you want to add. T4O supports two types of backup targets.

1. nfs

2. s3

If you want to add backup target of type 'nfs' then edit the following yaml file and create TVOBackupTarget resource.

If you want to add backup target of type 's3' then edit the following yaml file. In following secret user needs to mention base64 encoded s3 bucket access key and secret key.

To get base64 encoded string use linux command

If your S3 bucket is an Amazon S3 bucket.

If your S3 is of any other type edit following file and create TVOBackupTarget resource

Check deployment progress

2] Mount Backup target on Trilio Data Plane

Navigate to data plane scripts directory

Create templates needed for adding backup targets. Please note that use unique backup target name for parameter <BACKUP_TARGET_NAME>. You should not have used this backup target name earlier for any other trilio backup target. For parameter <BACKUP_TARGET_TYPE>, valid choices are ‘s3' and 'nfs’

A new directory gets created with backup target name having necessary templates. You can list these templates.

Backup target name gets set in all the yaml files created in this directory. BACKUP_TARGET_NAME gets converted to all small case and if any underscore ‘_' character is there in it, it gets replaced by hyphen character '-'. Add backup target details to template

Change to <BACKUP_TARGET_NAME> if not done already

Edit config map file

Create config map

If you are adding 's3' type backup target, then only you need to create following secret. We have already filled in all details in yaml file. We just need to apply it.

Set Trilio Ansible Runner container image url and tag in parameter 'openStackAnsibleEERunnerImage:' of trilio-add-backup-target-service.yaml You don’t need to change any other parameter.

Create custom data plane service for this backup target

Edit trilio-add-backup-target-deployment.yaml file and set parameter ‘nodeSets' with correct value from your environment. You don’t need to change any other parameter.

Get OpenStackDataPlaneNodeSet name

Set 'nodeSets' parameter

Trigger deployment of this backup target

Check logs Edit <DEPLOYMENT_NAME>, take it from above deployment yaml.

If this pod does not get created, it means you have not used unique backup target name or some other issue happend. Please verify that

3] Add Backup Target Records

Login to triliovault-wlm-api pod and run below CLI command to create backup target in Trilio DB.

For NFS Backup Target:

Sample command:

For Object lock enabled S3 Backup Target:

Sample command:

For non-object lock S3 Backup Target:

Sample command:

1] Uninstall TVO Control Plane Services

Delete tvocontrolplane resource

1.1] Unmount the backup targets

List openshift trilio control plane nodes

SSH Login to all these nodes and unmount the backup targets used by Trilio control plane services

2] Uninstall TVO Data Plane Services

Delete dataplane trilio resources from openshift

2.1] Log into each compute node, then stop and disable all Trilio services using the approach outlined below.

2.2] Clean Trilio Keystone resources

Trilio registers services and users in Keystone. Those need to be unregistered and deleted.

2.3] Clean Trilio database resources

Trilio creates databases for dmapi and workloadmgr services. These databases need to be cleaned.

Login into the database cluster

Run the following SQL statements to clean the database.

Definition

The file search functionality allows the user to search for files and folders located on a chosen VM in a workload in one or more Backups.

Navigating to the file search tab in Horizon

The file search tab is part of every workload overview. To reach it follow these steps:

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload a file search shall be done in

5. Click the workload name to enter the Workload overview

6. Click File Search to enter the file search tab

Configuring and starting a file search Horizon

A file search runs against a single virtual machine for a chosen subset of backups using a provided search string.

To run a file search the following elements need to be decided and configured

Choose the VM the file search shall run against

Under VM Name/ID choose the VM that the search is done upon. The drop down menu provides a list of all VMs that are part of any Snapshot in the Workload.

Set the File Path

The File Path defines the search string that is run against the chosen VM and Snapshots. This search string does support basic RegEx.

Example File Path for all files inside /etc : /etc/*

Define the Snapshots to search in

"Filter Snapshots by" is the third and last component that needs to be set. This defines which Snapshots are going to be searched.

There are 3 possibilities for a pre-filtering:

1. All Snapshots - Lists all Snapshots that contain the chosen VM from all available Snapshots

2. Last Snapshots - Choose between the last 10, 25, 50, or custom Snapshots and click Apply to get the list of the available Snapshots for the chosen VM that match the criteria.

3. Date Range - Set a start and end date and click apply to get the list of all available Snapshots for the chosen VM within the set dates.

After the pre-filtering is done all matching Snapshots are automatic prechosen. Uncheck any Snapshot that shall not be searched.

Start the File Search and retrieve the results in Horizon

To start a File Search the following elements need to be set:

• A VM to search in has to be chosen

• A valid File Path provided

• At least one Snapshot to search in selected

Once those have been set click "Search" to start the file search.

After a short time the results will be presented. The results are presented in a tabular format grouped by Snapshots and Volumes inside the Snapshot.

For each found file or folder the following information are provided:

• POSIX permissions

• Amount of links pointing to the file or folder

• User ID who owns the file or folder

• Group ID assigned to the file or folder

• Actual size in Bytes of the file or folder

• Time of creation

• Time of last modification

• Time of last access

• Full path to the found file or folder

Once the Snapshot of interest has been identified it is possible to go directly to the Snapshot using the "View Snapshot" option at the top of the table. It is also possible to directly mount the Snapshot using the "Mound Snapshot" Button at the end of the table.

Doing a CLI File Search

workloadmgr filepath-search [--snapshotids <snapshotid>]
                            [--end_filter <end_filter>]
                            [--start_filter <start_filter>]
                            [--date_from <date_from>]
                            [--date_to <date_to>]
                            <vm_id> <file_path>

• <vm_id> ➡️ ID of the VM to be searched

• <file_path> ➡️ Path of the file to search for

• --snapshotids <snapshotid> ➡️ Search only in specified snapshot ids snapshot-id: include the instance with this UUID

• --end_filter <end_filter> ➡️Displays last snapshots, example , last 10 snapshots, default 0 means displays all snapshots

• --start_filter <start_filter> ➡️Displays snapshots starting from , example , snapshot starting from 5, default 0 means starts from first snapshot

• --date_from <date_from> ➡️ From date in format 'YYYY-MM-DDTHH:MM:SS' eg 2016-10-10T00:00:00, If time isn't specified then it takes 00:00 by default

• --date_to <date_to> ➡️ To date in format 'YYYY-MM-DDTHH:MM:SS'(defult is current day),Specify HH:MM:SS to get snapshots within same day inclusive/exclusive results for date_from and date_to

Trilio Branch

Branch of the repository to be used for DevOps scripts.

trilio_branch : 6.1.2

Deployment Scripts

git clone -b 6.1.2 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/redhat-director-scripts/rhosp18/

git clone -b 6.1.2 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/redhat-director-scripts/rhosp17/

git clone -b 6.1.2 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/openstack-helm/

git clone -b 6.1.2 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/openstack-helm/

Container/Revision Information

Package Repositories & Versions

All packages are Python 3 (py3.6 - py3.10) compatible only.

https://pypi.fury.io/trilio-6-1/

deb [trusted=yes] https://apt.fury.io/trilio-6-1/ /

https://yum.fury.io/trilio-6-1/

[trilio-fury]
name=Trilio Gemfury Private Repo
baseurl=https://yum.fury.io/trilio-6-1/
enabled=1
gpgcheck=0

Trilio enables OpenStack administrators to set Project Quotas against the usage of Trilio.

The following Quotas can be set:

• Number of Workloads a Project is allowed to have

• Number of Snapshots a Project is allowed to have

• Number of VMs a Project is allowed to protect

• Amount of Storage a Project is allowed to use on the Backup Target

Work with Workload Quotas via Horizon

Workload Quotas are managed like any other Project Quotas.

1. Login into Horizon as user with admin role

2. Navigate to Identity

3. Navigate to Projects

4. Identify the Project to modify or show the quotas on

5. Use the small arrow next to "Manage Members" to open the submenu

6. Choose "Modify Quotas"

7. Navigate to "Workload Manager"

8. Edit Quotas as desired

9. Click "Save"

Work with Workload Quotas via CLI

List available Quota Types

Trilio is providing several different Quotas. The following command allows listing those.

workloadmgr project-quota-type-list

Show Quota Type Details

The following command will show the details of a provided Quota Type.

workloadmgr project-quota-type-show <quota_type_id>

• <quota_type_id> ➡️ID of the Quota Type to show

Create a Quota

The following command will create a Quota for a given project and set the provided value.

workloadmgr project-allowed-quota-create --quota-type-id quota_type_id
                                         --allowed-value allowed_value 
                                         --high-watermark high_watermark 
                                         --project-id project_id

• <quota_type_id> ➡️ID of the Quota Type to be created

• <allowed_value>➡️ Value to set for this Quota Type

• <high_watermark>➡️ Value to set for High Watermark warnings

• <project_id>➡️ Project to assign the quota to

List allowed Quotas

The following command lists all Trilio Quotas set for a given project.

workloadmgr project-allowed-quota-list <project_id>

• <project_id>➡️ Project to list the Quotas from

Show allowed Quota

The following command shows the details about a provided allowed Quota.

workloadmgr project-allowed-quota-show <allowed_quota_id>

• <allowed_quota_id> ➡️ID of the allowed Quota to show.

Update allowed Quota

The following command shows how to update the value of an already existing allowed Quota.

workloadmgr project-allowed-quota-update [--allowed-value <allowed_value>]
                                         [--high-watermark <high_watermark>]
                                         [--project-id <project_id>]
                                         <allowed_quota_id>

• <allowed_value>➡️ Value to set for this Quota Type

• <high_watermark>➡️ Value to set for High Watermark warnings

• <project_id>➡️ Project to assign the quota to

• <allowed_quota_id> ➡️ID of the allowed Quota to update

Delete allowed Quota

The following command will delete an allowed Quota and sets the value of the connected Quota Type back to unlimited for the affected project.

workloadmgr project-allowed-quota-delete <allowed_quota_id>

• <allowed_quota_id> ➡️ID of the allowed Quota to delete

Note: Run following steps on node from where you have install openstack cloud helm charts. (Master Node)

1] Clone trilio-openstack helm chart code repository [Run on Install Client Node]

• ssh to 'installation client node'

## Change name of old 'triliovault-cfg-scripts' repository directory.
mv triliovault-cfg-scripts triliovault-cfg-scripts-old
## Clone latest git repository 'triliovault-cfg-scripts'.
git clone -b {{ trilio_branch }} https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/
cd openstack-helm/trilio-openstack/
helm dep up
cd ../../../

2] Set container image tags [Run on Install Client Node]

Set updated image tags for trilio-openstack images and other images mentioned in following file.

Edit below file.

triliovault-cfg-scripts/openstack-helm/trilio-openstack/values_overrides/2023.2.yaml

Replace content of above 2023.2.yaml file with below YAML block with upgrade image tags.

---
images:
  tags:
    bootstrap: docker.io/openstackhelm/heat:2023.2-ubuntu_jammy
    db_init: docker.io/openstackhelm/heat:2023.2-ubuntu_jammy
    db_drop: docker.io/openstackhelm/heat:2023.2-ubuntu_jammy
    rabbit_init: docker.io/rabbitmq:3.7-management
    ks_user: docker.io/openstackhelm/heat:2023.2-ubuntu_jammy
    ks_service: docker.io/openstackhelm/heat:2023.2-ubuntu_jammy
    ks_endpoints: docker.io/openstackhelm/heat:2023.2-ubuntu_jammy
    dep_check: quay.io/airshipit/kubernetes-entrypoint:v1.0.0
    image_repo_sync: docker.io/docker:17.07.0
    triliovault_wlm_cloud_trust: docker.io/trilio/trilio-wlm-helm:<image-tag>
    triliovault_wlm_api: docker.io/trilio/trilio-wlm-helm:<image-tag>
    triliovault_wlm_cron: docker.io/trilio/trilio-wlm-helm:<image-tag>
    triliovault_wlm_scheduler: docker.io/trilio/trilio-wlm-helm:<image-tag>
    triliovault_wlm_workloads: docker.io/trilio/trilio-wlm-helm:<image-tag>
    triliovault_wlm_db_sync: docker.io/trilio/trilio-wlm-helm:<image-tag>
    triliovault_datamover: docker.io/trilio/trilio-datamover-helm:<image-tag>
    triliovault_datamover_api: docker.io/trilio/trilio-datamover-api-helm:<image-tag>
    triliovault_datamover_db_sync: docker.io/trilio/trilio-datamover-api-helm:<image-tag>

3] Provide trilio-openstack backup target details [Run on Install Client Node]

• If your backup target is of type ‘nfs' then you can provide it’s details in nfs.yaml. Your old nfs share details will be available in file ‘triliovault-cfg-scripts-old/openstack-helm/trilio-openstack/values_overrides/nfs.yaml' Copy those details in following file in given format. You need to provide the same share here.

cp triliovault-cfg-scripts-old/openstack-helm/trilio-openstack/values_overrides/nfs.yaml triliovault-cfg-scripts/openstack-helm/trilio-openstack/values_overrides/nfs.yaml

• If the backup target for trilio-openstack is of ‘s3' type with S3 backend has TLS enabled with self signed certificates, then user needs store it’s ca certificate in the following file. It will be placed at correct location by deployment scripts. We can copy this certificate from old repo to current one using following command.

cp triliovault-cfg-scripts-old/openstack-helm/trilio-openstack/files/s3-cert.pem triliovault-cfg-scripts/openstack-helm/trilio-openstack/files/s3-cert.pem

4] Provide cloud admin user credentials - keystone.yaml [Run on Install Client Node]

Provide cloud administrator user details in the following file. It should have 'admin' role on cloud domain.

You can copy these details from old repository file ‘triliovault-cfg-scripts-old/openstack-helm/trilio-openstack/values_overrides/keystone.yaml'

vi triliovault-cfg-scripts/openstack-helm/triliovault/values_overrides/keystone.yaml
conf:
  triliovault:
    cloud_admin_user_name: "admin"
    cloud_admin_project_name: "admin"
    cloud_admin_domain_name: "default"
    ## Keystone endpoint interface that triliovault workloadmgr services will use to communicate to other openstack services
    ## Valid values: internal, public, admin
    interface: "internal"
    ## Provide any role name as your preference.
    ## Significance: To run any backups in triliovault an openstack user needs to have this role on given project.
    trustee_role: "creator"

5] Keystone, rabbitmq and database creds - admin_creds.yaml [Run on Install Client Node]

Copy following files from old repository to current one as it is. [No changes from triliovault install side]

If you do not have these old files, then you need to perform the same step from 'TrilioVault install document on Openstack Helm'

cp triliovault-cfg-scripts-old/openstack-helm/trilio-openstack/values_overrides/admin_creds.yaml triliovault-cfg-scripts/openstack-helm/trilio-openstack/values_overrides/admin_creds.yaml
cp triliovault-cfg-scripts-old/openstack-helm/trilio-openstack/templates/bin/_triliovault-nova-compute.conf.tpl triliovault-cfg-scripts/openstack-helm/trilio-openstack/templates/bin/_triliovault-nova-compute.conf.tpl

6] ceph.yaml changes [Run on Install Client Node]

You can copy ceph.yaml from old repository to current repository as it is. [No changes from triliovault install side]

cp triliovault-cfg-scripts-old/openstack-helm/trilio-openstack/values_overrides/ceph.yaml triliovault-cfg-scripts/openstack-helm/trilio-openstack/values_overrides/ceph.yaml
cp triliovault-cfg-scripts-old/openstack-helm/trilio-openstack/templates/bin/_triliovault-ceph.conf.tpl triliovault-cfg-scripts/openstack-helm/trilio-openstack/templates/bin/_triliovault-ceph.conf.tpl

7] Docker registry credentials secret. [Run on Install Client Node]

No changes here. So old registry credentials should work fine. Just ensure that it exists.

# kubectl get secrets | grep triliovault-image-registry
triliovault-image-registry                                 kubernetes.io/dockerconfigjson        1      89d

If user credentials are changed or old secret does not exist; then we need to recreate it. In that case follow this step from installation document.

8] Review and edit upgrade.sh script [Run on Install Client Node]

• User need to edit following file and review all the yaml files and options provide in command.

• If any option is not valid for your Openstack Helm setup, you need to edit it.

• Yaml files that user has used in install.sh should be available in upgrade.sh too.

• Make sure that nfs.yaml/s3.yaml(Both not required- select as per backup target type) and keystone.yaml should be available in upgrade.sh.

• db_drop.yaml file should not be included in upgrade.sh

• If your keystone’s public endpoint is on TLS then include yaml file 'tls_public_endpoint.yaml' from upgrade.sh

• If your keystone’s all endpoints are on non TLS (http only) then you need to remove yaml file tls_public_endpoint.yaml from upgrade.sh

vi triliovault-cfg-scripts/openstack-helm/trilio-openstack/utils/upgrade.sh

#!/bin/bash -x
cd ../../
helm upgrade trilio-openstack ./trilio-openstack --namespace=trilio-openstack \
--values=./trilio-openstack/values_overrides/image_pull_secrets.yaml \
--values=./trilio-openstack/values_overrides/keystone.yaml \
--values=./trilio-openstack/values_overrides/nfs.yaml \
--values=./trilio-openstack/values_overrides/2023.2.yaml \
--values=./trilio-openstack/values_overrides/admin_creds.yaml \
--values=./trilio-openstack/values_overrides/tls_public_endpoint.yaml \
--values=./trilio-openstack/values_overrides/ceph.yaml \
--values=./trilio-openstack/values_overrides/ingress.yaml \
--values=./trilio-openstack/values_overrides/triliovault_passwords.yaml
echo -e "Waiting for triliovault pods to get into running state"
./trilio-openstack/utils/wait_for_pods.sh trilio-openstack
kubectl get pods -n trilio-openstack

9] Run upgrade script to upgrade Trilio for Openstack [Run on Install Client Node]

cd triliovault-cfg-scripts/openstack-helm/trilio-openstack/utils
./upgrade.sh

10] Verify that trilio-openstack upgrade finished well using following steps [Run on Install Client Node]

## Check trilio-openstack helm chart version. It should be latest, mentioned in file 'triliovault-cfg-scripts/openstack-helm/trilio-openstack/Chart.yaml'
helm ls | grep trilio-openstack

## Check status of trilio-openstack helm chart release
helm status trilio-openstack

## Check if all pods are in running/Completed state.
kubectl get pods -n trilio-openstack
Sample Output:
----------------------
# kubectl get pods -n trilio-openstack
NAME                                                 READY   STATUS      RESTARTS   AGE
triliovault-datamover-api-5877ff9b48-49wsq           1/1     Running     0          6d
triliovault-datamover-api-5877ff9b48-8zfmh           1/1     Running     0          6d
triliovault-datamover-api-5877ff9b48-xx4rm           1/1     Running     0          6d
triliovault-datamover-db-init-r4j9w                  0/1     Completed   0          6d
triliovault-datamover-db-sync-c254n                  0/1     Completed   0          6d
triliovault-datamover-ks-endpoints-8qnt8             0/3     Completed   0          6d
triliovault-datamover-ks-service-hps97               0/1     Completed   0          6d
triliovault-datamover-ks-user-sgrz6                  0/1     Completed   0          6d
triliovault-datamover-openstack-compute-node-2s7k7   1/1     Running     0          6d
triliovault-datamover-openstack-compute-node-64sm5   1/1     Running     0          6d
triliovault-wlm-api-5bbc74cb4b-69b2b                 1/1     Running     0          6d
triliovault-wlm-api-5bbc74cb4b-hbzfj                 1/1     Running     0          6d
triliovault-wlm-api-5bbc74cb4b-zxlcl                 1/1     Running     0          6d
triliovault-wlm-cloud-trust-hnxj8                    0/1     Completed   0          6d
triliovault-wlm-cron-764658cb59-b8jx6                1/1     Running     0          6d
triliovault-wlm-db-init-rbpw2                        0/1     Completed   0          6d
triliovault-wlm-db-sync-p97z8                        0/1     Completed   0          6d
triliovault-wlm-ks-endpoints-tx65s                   0/3     Completed   0          6d
triliovault-wlm-ks-service-9mkkt                     0/1     Completed   0          6d
triliovault-wlm-ks-user-x5xrx                        0/1     Completed   0          6d
triliovault-wlm-rabbit-init-qqzwr                    0/1     Completed   0          6d
triliovault-wlm-scheduler-6c85688899-55tz7           1/1     Running     0          6d
triliovault-wlm-workloads-f9dc944db-5w7s9            1/1     Running     0          6d
triliovault-wlm-workloads-f9dc944db-84wj7            1/1     Running     0          6d
triliovault-wlm-workloads-f9dc944db-ccdvz            1/1     Running     0          6d

## Check if jobs finished well
kubectl get jobs -n trilio-openstack
Sample Output:
--------------
NAME                                 COMPLETIONS   DURATION   AGE
triliovault-datamover-db-init        1/1           4s         6d
triliovault-datamover-db-sync        1/1           8s         6d
triliovault-datamover-ks-endpoints   1/1           19s        6d
triliovault-datamover-ks-service     1/1           8s         6d
triliovault-datamover-ks-user        1/1           21s        6d
triliovault-wlm-cloud-trust          1/1           3m10s      6d
triliovault-wlm-db-init              1/1           4s         6d
triliovault-wlm-db-sync              1/1           13s        6d
triliovault-wlm-ks-endpoints         1/1           20s        6d
triliovault-wlm-ks-service           1/1           7s         6d
triliovault-wlm-ks-user              1/1           20s        6d
triliovault-wlm-rabbit-init          1/1           9s         6d

## If you are using NFS backup target, check if nfs pvc got into Bound state


kubectl get pvc -n trilio-openstack
Sample Output:
-------------
NAME                                               STATUS   VOLUME                                            CAPACITY   ACCESS MODES   STORAGECLASS   AGE
triliovault-nfs-pvc-172-25-0-10-mnt-tvault-42424   Bound    triliovault-nfs-pv-172-25-0-10-mnt-tvault-42424   20Gi       RWX            nfs            6d

Trilio For Openstack Helm Chart Upgrade is Done!

11] Upgrade Trilio horizon plugin

• Follow this documentation #id-13-Install Trilio for Openstack Horizon Plugin from section 12.2 to section 12.3 and update the trilio horizon plugin image tag.

• Set correct image tag for horizon plugin image.

• Follow further steps from above doc to complete the upgrade of Trilio horizon plugin

Start File Search

POST https://$(tvm_address):8780/v1/$(tenant_id)/search

Starts a File Search with the given parameters

Path Parameters

Headers

HTTP/1.1 200 OK
Server: nginx/1.16.1
Date: Mon, 09 Nov 2020 13:23:25 GMT
Content-Type: application/json
Content-Length: 244
Connection: keep-alive
X-Compute-Request-Id: req-bdfd3fb8-5cbf-4108-885f-63160426b2fa

{
   "file_search":{
      "created_at":"2020-11-09T13:23:25.698534",
      "updated_at":null,
      "id":14,
      "deleted_at":null,
      "status":"executing",
      "error_msg":null,
      "filepath":"/etc/h*",
      "json_resp":null,
      "vm_id":"08dab61c-6efd-44d3-a9ed-8e789d338c1b"
   }
}

Body format

{
   "file_search":{
      "start":<Integer>,
      "end":<Integer>,
      "filepath":"<Reg-Ex String>",
      "date_from":<Date Format: YYYY-MM-DDTHH:MM:SS>,
      "date_to":<Date Format: YYYY-MM-DDTHH:MM:SS>,
      "snapshot_ids":[
         "<Snapshot-ID>"
      ],
      "vm_id":"<VM-ID>"
   }
}

Get File Search Results

POST https://$(tvm_address):8780/v1/$(tenant_id)/search/<search_id>

Starts a filesearch with the given parameters

Path Parameters

Headers

HTTP/1.1 200 OK
Server: nginx/1.16.1
Date: Mon, 09 Nov 2020 13:24:28 GMT
Content-Type: application/json
Content-Length: 819
Connection: keep-alive
X-Compute-Request-Id: req-d57bea9a-9968-4357-8743-e0b906466063

{
   "file_search":{
      "created_at":"2020-11-09T13:23:25.000000",
      "updated_at":"2020-11-09T13:23:48.000000",
      "id":14,
      "deleted_at":null,
      "status":"completed",
      "error_msg":null,
      "filepath":"/etc/h*",
      "json_resp":"[
                      {
                         "ed4f29e8-7544-4e1c-af8a-a76031211926":[
                            {
                               "/dev/vda1":[
                                  "/etc/hostname",
                                  "/etc/hosts"
                               ],
                               "/etc/hostname":{
                                  "dev":"2049",
                                  "ino":"32",
                                  "mode":"33204",
                                  "nlink":"1",
                                  "uid":"0",
                                  "gid":"0",
                                  "rdev":"0",
                                  "size":"1",
                                  "blksize":"1024",
                                  "blocks":"2",
                                  "atime":"1603455255",
                                  "mtime":"1603455255",
                                  "ctime":"1603455255"
                               },
                               "/etc/hosts":{
                                  "dev":"2049",
                                  "ino":"127",
                                  "mode":"33204",
                                  "nlink":"1",
                                  "uid":"0",
                                  "gid":"0",
                                  "rdev":"0",
                                  "size":"37",
                                  "blksize":"1024",
                                  "blocks":"2",
                                  "atime":"1603455257",
                                  "mtime":"1431011050",
                                  "ctime":"1431017172"
                               }
                            }
                         ]
                      }
                  ]",
      "vm_id":"08dab61c-6efd-44d3-a9ed-8e789d338c1b"
   }
}