5.2.4 (2024-09-18)

What's New?

1. Support for Kolla Caracal on Rocky9 added

2. Support for Canonical Antelope & Bobcat on Jammy added

3. Bug Fixes

Fixed Bugs and Issues

1. Hide the VMware/Virtual Machine Migration related menus, if that feature is not active.

2. Issue where some volume were not being attached after restore.

3. S3 logging is now fixed.

4. Slow performance while creating snapshot identified and fixed.

5. Periods of inactivity identified during full snapshot of large volume.

6. Deployment failing on kolla-anisble Antelope.

7. Modify workload by another tenant user brought workload in error.

8. Fixed issue where Trilio looked for instance in wrong region if OpenStack is multi region.

9. Restore failing with security group not found.

10. Backup failing with multipath device not found.

11. Backup failing due to virtual size of LUN zero.

12. Backup Failure in contego; error in _refresh_multipath_size call. Unexpected error while running command tee -a /sys/bus/scsi/drivers/sd/None:None:None:None/rescan

Known Issues

1. Upgrade of python3-oslo.messaging breaks Trilio in Juju

   • The Trilio charms should not allow installations of python3-oslo.messaging newer than 12.1.6-0ubuntu1, as they disrupt correct RabbitMQ connectivity from the trilio-wlm units.

   • Workaround: Run below commands
2. RHOSP16.1/16.2 : Unexpected behavior with OpenStack Horizon Dashboard

   • At times, after the overcloud deployment, the OpenStack Horizon Dashboard encounters issues, leading to unexpected behavior in certain functionalities.

   • Workaround:

     1. Login to the Horizon container and run the following commands: podman exec -it -u root horizon /bin/bash /usr/bin/python3 /usr/share/openstack-dashboard/manage.py collectstatic --clear --noinput /usr/bin/python3 /usr/share/openstack-dashboard/manage.py compress --force

     2. Restart the Horizon container : podman restart horizon

3. Horizon pod keeps restarting on RHOSP 16.2.4

   • It has been observed that on a specific RHOSP minor version, i.e. 16.2.4, the Horizon pod keeps restarting.

   • The issue has been identified with Memcached. Either of the below workarounds can be performed on all the controller nodes where the issue seems to occur.

   • Workaround:

     1. Workaround 1 : Restart the memcached service on the controller using systemctl systemctl restart tripleo_memcached.service

     2. Workaround 2 : Restart the memcached pod podman restart memcached

4. get-importworkload-list and get-orphaned-workloads-list show incorrect workloads

   • The commands get-importworkload-list and get-orphaned-workloads-list show incorrect workloads in the output.

   • Workaround:

     • Use --project option with get-importworkload-list CLI command to get the list of workloads that can be imported on any particular project of the OpenStack. workloadmgr workload-get-importworkloads-list --project_id <project_id>

5. workload-importworklaods cli command throws gateway timed out error

   • workload-importworklaods CLI command throws gateway timed out error, however, import actually succeeds.

   • In the background, the import process continues to run in spite of timeout until the operation is complete.

   • Workaround:

     • To avoid the timeout error, the respective timeout server value can be increased inside /var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg for RHOSP

5.2.3 (2024-07-05)

What's New?

1. VMware Migration Mapping Dashboard

2. Bug Fixes

Fixed Bugs and Issues

1. [VMware Migration] boot options shows BIOS when the guest VM has unsecured EFI boot.

2. wlm client does not work on mac having python 3.10.

3. Cloud admin trust creation was failing.

4. Modify workload with other user brings workload in error.

5. Trilio looks for instance in wrong region if OpenStack is multi region

Known Issues

1. Encryption checkbox is disabled when creating 1st workload from UI

   • While creating 1st workload from the UI, the Enable Encryption checkbox on the Create workload page is disabled. Creating a non-encrypted workload first will resolve this issue and the user can start creating the encrypted workloads.

2. RHOSP16.1/16.2 : Unexpected behavior with OpenStack UI

   • At times, after the overcloud deployment, the OpenStack UI encounters issues, leading to unexpected behavior in certain functionalities.

   • Workaround:

     1. Login to the Horizon container and run the following commands: podman exec -it -u root horizon /bin/bash /usr/bin/python3 /usr/share/openstack-dashboard/manage.py collectstatic --clear --noinput /usr/bin/python3 /usr/share/openstack-dashboard/manage.py compress --force

     2. Restart the Horizon container : podman restart horizon

3. Horizon pod keeps restarting on RHOSP 16.2.4

   • It has been observed that on a specific RHOSP minor version, i.e. 16.2.4, the horizon pod keeps restarting.

   • Either of the below workarounds can be performed on all the controller nodes where the issue seems to occur.

   • Workaround:

     1. Workaround 1 : Restart the memcached service on the controller using systemctl systemctl restart tripleo_memcached.service

     2. Workaround 2 : Restart the memcached pod podman restart memcached

4. get-importworkload-list and get-orphaned-workloads-list show incorrect workloads

   • The commands get-importworkload-list and get-orphaned-workloads-list show incorrect workloads in the output.

   • Workaround:

     • Use --project option with get-importworkload-list CLI command to get the list of workloads that can be imported on any particular project of the OpenStack. workloadmgr workload-get-importworkloads-list --project_id <project_id>

5. workload-importworklaods cli command throws gateway timed out error

   • workload-importworklaods CLI command throws gateway timed out error, however, import actually succeeds.

   • In the background, the import process continues to run in spite of timeout until the operation is complete.

   • Workaround:

     • To avoid the timeout error, the respective timeout server value can be increased inside /var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg for RHOSP

5.2.2 (2024-04-22)

What's New?

1. Support for Kolla Bobcat OpenStack added.

2. Fixed the issues reported by customers.

Fixed Bugs and Issues

1. [VMware Migration] Migration failure due to minor issues in handling vm and network names.

2. [VMware migration] boot options shows BIOS when the guest VM has unsecured efi boot.

3. [VMware Migration] Migration fails when the network name of the guest VM contains - and _ both.

4. Support multi ceph config in same /etc/ceph location.

5. Trilio looks for instance in wrong region if openstack is multi region.

6. T4O deployment on Kolla Antelope failing. DMAPI and horizon container fail to start.

7. Quota updation when admin user have access to projects of multiple domains

Known Issues

1. Encryption checkbox is disabled when creating 1st workload from UI

   • While creating 1st workload from the UI, the Enable Encryption checkbox on the Create workload page is disabled. Creating a non-encrypted workload first will resolve this issue and the user can start creating the encrypted workloads.

2. RHOSP16.1/16.2 : Unexpected behavior with OpenStack UI

   • At times, after the overcloud deployment, the OpenStack UI encounters issues, leading to unexpected behavior in certain functionalities.

   • Workaround:

     1. Login to the Horizon container and run the following commands: podman exec -it -u root horizon /bin/bash /usr/bin/python3 /usr/share/openstack-dashboard/manage.py collectstatic --clear --noinput /usr/bin/python3 /usr/share/openstack-dashboard/manage.py compress --force

     2. Restart the Horizon container : podman restart horizon

3. Horizon pod keeps restarting on RHOSP 16.2.4

   • It has been observed that on a specific RHOSP minor version, i.e. 16.2.4, the horizon pod keeps restarting.

   • Either of the below workarounds can be performed on all the controller nodes where the issue seems to occur.

   • Workaround:

     1. Workaround 1 : Restart the memcached service on the controller using systemctl systemctl restart tripleo_memcached.service

     2. Workaround 2 : Restart the memcached pod podman restart memcached

4. get-importworkload-list and get-orphaned-workloads-list show incorrect workloads

   • The commands get-importworkload-list and get-orphaned-workloads-list show incorrect workloads in the output.

   • Workaround:

     • Use --project option with get-importworkload-list CLI command to get the list of workloads that can be imported on any particular project of the OpenStack. workloadmgr workload-get-importworkloads-list --project_id <project_id>

5. workload-importworklaods cli command throws gateway timed out error

   • workload-importworklaods CLI command throws gateway timed out error, however, import actually succeeds.

   • In the background, the import process continues to run in spite of timeout until the operation is complete.

   • Workaround:

     • To avoid the timeout error, the respective timeout server value can be increased inside /var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg for RHOSP

5.2.1

What's New?

1. Software Driven Migration : VMware to OpenStack With T4O-5.2.1 release, Trilio now supports migration of VMs from VMware to OpenStack.

2. Fixed the issues reported by customers.

Fixed Bugs and issues

1. Support multi ceph config in same /etc/ceph location.

2. Encrypted incremental restore fails to diff AZ.

3. Trilio installation failing for RHOSP17.1 IPV6.

4. [RHOSP] memcache_servers parameter in triliovault wlm and datamover service conf files fixed for ipv6.

5. Deployment (with non-root user) failing on kolla with barbican.

6. T4O 5.1 failing on Kolla Zed rocky

7. 5.2 deployment failing on Antelope with ansible node on separate server.

8. WLM service issue with IPV6. Containers keep restarting due to Data too long for column ip_addresses at row 1.

Known Issues

1. RHOSP16.1/16.2 : Unexpected behavior with OpenStack UI

   • At times, after the overcloud deployment, the OpenStack UI encounters issues, leading to unexpected behavior in certain functionalities.

   • Workaround:

     1. Login to the Horizon container and run the following commands: podman exec -it -u root horizon /bin/bash /usr/bin/python3 /usr/share/openstack-dashboard/manage.py collectstatic --clear --noinput /usr/bin/python3 /usr/share/openstack-dashboard/manage.py compress --force

     2. Restart the Horizon container : podman restart horizon

2. Horizon pod keeps restarting on RHOSP 16.2.4

   • It has been observed that on a specific RHOSP minor version, i.e. 16.2.4, the horizon pod keeps restarting.

   • Either of the below workarounds can be performed on all the controller nodes where the issue seems to occur.

   • Workaround:

     1. Workaround 1 : Restart the memcached service on the controller using systemctl systemctl restart tripleo_memcached.service

     2. Workaround 2 : Restart the memcached pod podman restart memcached

3. get-importworkload-list and get-orphaned-workloads-list show incorrect workloads

   • The commands get-importworkload-list and get-orphaned-workloads-list show incorrect workloads in the output.

   • Workaround:

     • Use --project option with get-importworkload-list CLI command to get the list of workloads that can be imported on any particular project of the OpenStack. workloadmgr workload-get-importworkloads-list --project_id <project_id>

4. workload-importworklaods cli command throws gateway timed out error

   • workload-importworklaods CLI command throws gateway timed out error, however, import actually succeeds.

   • In the background, the import process continues to run in spite of timeout until the operation is complete.

   • Workaround:

     • To avoid the timeout error, the respective timeout server value can be increased inside /var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg for RHOSP

5.2.0

What's New?

• New OpenStack additions in Trilio support matrix: With T4O-5.2.0, Trilio now supports backup and DR for RHOSP17.1 and Kolla-Ansible OpenStack 2023.1 Antelope on Rocky Linux & Ubuntu Jammy.

Known Issues

1. RHOSP16.1/16.2 : Unexpected behavior with OpenStack UI

   • At times, after the overcloud deployment, the OpenStack UI encounters issues, leading to unexpected behavior in certain functionalities.

   • Workaround:

     1. Login to the Horizon container and run the following commands: podman exec -it -u root horizon /bin/bash /usr/bin/python3 /usr/share/openstack-dashboard/manage.py collectstatic --clear --noinput /usr/bin/python3 /usr/share/openstack-dashboard/manage.py compress --force

     2. Restart the Horizon container : podman restart horizon

2. Horizon pod keeps restarting on RHOSP 16.2.4

   • It has been observed that on a specific RHOSP minor version, i.e. 16.2.4, the horizon pod keeps restarting.

   • Either of the below workarounds can be performed on all the controller nodes where the issue seems to occur.

   • Workaround:

     1. Workaround 1 : Restart the memcached service on the controller using systemctl systemctl restart tripleo_memcached.service

     2. Workaround 2 : Restart the memcached pod podman restart memcached

3. get-importworkload-list and get-orphaned-workloads-list show incorrect workloads

   • The commands get-importworkload-list and get-orphaned-workloads-list show incorrect workloads in the output.

   • Workaround:

     • Use --project option with get-importworkload-list CLI command to get the list of workloads that can be imported on any particular project of the OpenStack. workloadmgr workload-get-importworkloads-list --project_id <project_id>

4. workload-importworklaods cli command throws gateway timed out error

   • workload-importworklaods CLI command throws gateway timed out error, however, import actually succeeds.

   • In the background, the import process continues to run in spite of timeout until the operation is complete.

   • Workaround:

     • To avoid the timeout error, the respective timeout server value can be increased inside /var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg for RHOSP

5.1.0

What's New?

• True Native and Containerized Deployment: In the T4O-5.1.0 release, Trilio has embraced a transformative shift. Departing from the traditional method of shipping appliance images, T4O has evolved into a comprehensive backup and recovery solution that is fully containerized. Within this iteration, all T4O services have been elegantly encapsulated within containers, seamlessly aligning with OpenStack through its innate deployment process.

• New OpenStack additions in Trilio support matrix: With T4O-5.1.0, Trilio now supports backup and DR for Kolla-Ansible OpenStack Zed on Rocky Linux and Ubuntu Jammy.

Known Issues

1. Horizon pod keeps restarting on RHOSP 16.2.4

   • It has been observed that on a specific RHOSP minor version, i.e. 16.2.4, the horizon pod keeps restarting.

   • Either of the below workarounds can be performed on all the controller nodes where the issue seems to occur.

   • Workaround:

     1. Workaround 1 : Restart the memcached service on the controller using systemctl systemctl restart tripleo_memcached.service

     2. Workaround 2 : Restart the memcached pod podman restart memcached

2. get-importworkload-list and get-orphaned-workloads-list show incorrect workloads

   • The commands get-importworkload-list and get-orphaned-workloads-list show incorrect workloads in the output.

   • Workaround:

     • Use --project option with get-importworkload-list CLI command to get the list of workloads that can be imported on any particular project of the OpenStack. workloadmgr workload-get-importworkloads-list --project_id <project_id>

3. workload-importworklaods cli command throws gateway timed out error

   • workload-importworklaods CLI command throws gateway timed out error, however, import actually succeeds.

   • In the background, the import process continues to run in spite of timeout until the operation is complete.

   • Workaround:

     • To avoid the timeout error, the respective timeout server value can be increased inside /var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg for RHOSP

5.0.0

What's New?

• True Native and Containerized Deployment: In the T4O-5.0.0 release, Trilio has embraced a transformative shift. Departing from the traditional method of shipping appliance images, T4O has evolved into a comprehensive backup and recovery solution that is fully containerized. Within this iteration, all T4O services have been elegantly encapsulated within containers, seamlessly aligning with OpenStack through its innate deployment process.

• Faster Snapshot operation (Removal of Workload types: Serial and Parallel): In a move to boost Snapshot operation efficiency, we've introduced a streamlined method for creating workloads, eliminating the need for users to select between different workload types. T4O will now intelligently manage workloads in the background, ensuring optimal performance for backup operations.

• Improved Database Storage: T4O has made significant enhancements to its database management, focusing on efficient data garbage collection. By discarding unnecessary data and optimizing the utilization of DB storage within the task flow, these improvements have led to a leaner and more streamlined storage approach for its DB tables.

• Better Troubleshooting: Enhanced troubleshooting capabilities come through the implementation of a unified logging format across all T4O services/containers. This improvement grants users the freedom to adjust the format or log levels for individual services/containers as needed.

Known Issues

1. Horizon pod keeps restarting on RHOSP 16.2.4

   • It has been observed that on a specific RHOSP minor version, i.e. 16.2.4, the horizon pod keeps restarting.

   • Either of the below workarounds can be performed on all the controller nodes where the issue seems to occur.

   • Workaround:

     1. Workaround 1 : Restart the memcachedservice on the controller using systemctl systemctl restart tripleo_memcached.service

     2. Workaround 2 : Restart the memcached pod podman restart memcached

2. get-importworkload-list and get-orphaned-workloads-list show incorrect workloads

   • Commands get-importworkload-list and get-orphaned-workloads-list show incorrect workloads in the output.

   • Workaround:

     • Use --project option with get-importworkload-list CLI command to get the list of workloads that can be imported on any particular project of the OpenStack. workloadmgr workload-get-importworkloads-list --project_id <project_id>

3. workload-importworklaods cli command throws gateway timed out error

   • workload-importworkloads CLI command throws gateway timed out error, however, import succeeds.

   • In the background, the import process continues to run despite of timeout until the operation is complete.

   • Workaround:

     • To avoid the timeout error, the respective timeout server value can be increased inside /var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg for RHOSP

Trilio Branch

Branch of the repository to be used for DevOps scripts.

trilio_branch : 5.2.4

Deployment Scripts

git clone -b 5.2.4 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/redhat-director-scripts/rhosp16/

git clone -b 5.2.4 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/redhat-director-scripts/rhosp17/

git clone -b 5.2.4 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/kolla-ansible/ansible/

Container/Revision Information

Package Repositories & Versions

All packages are Python 3 (py3.6 - py3.10) compatible only.

https://pypi.fury.io/trilio-5-2/

deb [trusted=yes] https://apt.fury.io/trilio-5-2/ /

https://yum.fury.io/trilio-5-2/

[trilio-fury]
name=Trilio Gemfury Private Repo
baseurl=https://yum.fury.io/trilio-5-2/
enabled=1
gpgcheck=0

VM Migration Tool

Docker Container Tag

trilio/trilio-migration-vm2os:5.2.4

Each T4O release includes a set of artifacts such as version-tagged containers, package repositories, and distribution packages.

To help users quickly identify the resources associated with each release, we have added dedicated sub-pages corresponding to a specific release version.

Trilio Data, Inc

Trilio Data, Inc. is the leader in providing backup and recovery solutions for cloud-native applications. Established in 2013, its flagship products, Trilio for OpenStack(T4O) and Trilio for Kubernetes(T4K) are used by a wide number of large corporations around the world.

About Trilio for OpenStack

T4O, by Trilio Data, is a native OpenStack service-based solution that provides policy-based comprehensive backup and recovery for OpenStack workloads. It captures point-in-time workloads (Application, OS, Compute, Network, Configurations, Data, and Metadata of an environment) as full or incremental snapshots. These snapshots can be held in a variety of storage environments including NFS, AWS S3, and other S3-compatible storages. With Trilio and its one-click recovery, organizations can improve Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO). With Trilio, IT departments are enabled to fully deploy OpenStack solutions and provide business assurance through enhanced data retention, protection, and integrity.

With the use of Trilio’s VAST (Virtual Snapshot Technology), Enterprise IT and Cloud Service Providers can now deploy backup and disaster recovery as a service to prevent data loss or data corruption through point-in-time snapshots and seamless one-click recovery. Trilio takes point-in-time backup of the entire workload consisting of compute resources, network configurations, and storage data as one unit. It also takes incremental backups that capture only the changes that were made since the last backup. Incremental snapshots save a considerable amount of storage space as the backup only includes changes since the last backup. The benefits of using VAST for backup and restore could be summarized below:

About this Documentation

This documentation serves as the end-user technical resource to accompany Trilio for OpenStack. You will learn about the architecture, installation, and vast number of operations of this product. The intended audience is anyone who wants to understand the value, operations, and nuances of protecting their cloud-native applications with Trilio for OpenStack.

Trilio seamlessly integrates with OpenStack, functioning exclusively through APIs utilizing the OpenStack Endpoints. Furthermore, Trilio establishes its own set of OpenStack endpoints. Additionally, both the Trilio appliance and compute nodes interact with the backup target, impacting the network strategy for a Trilio installation.

Existing endpoints in OpenStack

OpenStack comprises three endpoint groupings:

• Public Endpoints

Public endpoints are meant to be used by the OpenStack end-users to work with OpenStack.

• Internal Endpoints

Internal endpoints are intended to be used by the OpenStack services to communicate with each

• Admin Endpoints

Admin endpoints are meant to be used by OpenStack administrators.

Among these three endpoint categories, it's important to note that the admin endpoint occasionally hosts APIs not accessible through any other type of endpoint.

To learn more about OpenStack endpoints please visit the official OpenStack documentation.

OpenStack endpoints required by Trilio

Trilio communicates with all OpenStack services through a designated endpoint type, determined and configured during the deployment of Trilio's services.

The following network requirements can be identified this way:

• Trilio services need access to the Keystone admin endpoint on the admin endpoint network if it is available.

• Trilio services need access to all endpoints of the set endpoint type during deployment.

Recommendation: Provide access to all OpenStack Endpoint types

Trilio recommends granting comprehensive access to all OpenStack endpoints for all Trilio services, aligning with OpenStack's established standards and best practices.

Additionally, Trilio generates its own endpoints, which are integrated within the same network as other OpenStack API services.

To adhere to OpenStack's prescribed standards and best practices, it's advisable that Trilio containers operate on the same network as other OpenStack containers.

• The public endpoint to be used by OpenStack users when using Trilio CLI or API

• The internal endpoint to communicate with the OpenStack services

• The admin endpoint to use the required admin-only APIs of Keystone

Backup target access required by Trilio

The Trilio solution uses backup target storage to place the backup data securely. Trilio divides its backup data into two parts:

1. Metadata

2. Volume Disk Data

The first type of data is generated by the Trilio Workloadmgr services through communication with the OpenStack Endpoints. All metadata that is stored together with a backup is written by the Trilio Workloadmgr services to the backup target in the JSON format.

The second type of data is generated by the Trilio Datamover service running on the compute nodes. The Datamover service reads the Volume Data from the Cinder or Nova storage and transfers this data as a qcow2 image to the backup target. Each Datamover service is hereby responsible for the VMs running on its compute node.

The network requirements are therefor:

• Every Trilio Workloadmgr service containers need access to the backup target

• Every Trilio Datamover service containers need access to the backup target

Before embarking on the installation process for Trilio in your OpenStack environment, it is highly advisable to carefully consider several key elements. These considerations will not only streamline the installation procedure but also ensure the optimal setup and functionality of Trilio's solutions within your OpenStack infrastructure.

Tenant Quotas

Trilio leverages Cinder snapshots to facilitate the computation of both full and incremental backups.

When executing full backups, Trilio orchestrates the generation of Cinder snapshots for all volumes included in the backup job. These Cinder snapshots remain intact for subsequent incremental backup image calculations.

During incremental backup operations, Trilio generates fresh Cinder snapshots and computes the altered blocks between these new snapshots and the earlier retained snapshots from full or previous backups. The old snapshots are subsequently deleted, while the newly generated snapshots are preserved.

Consequently, it becomes imperative for each tenant benefiting from Trilio's backup functionality to possess adequate Cinder snapshot quotas capable of accommodating these supplementary snapshots. As a guiding principle, it is recommended to append 2 snapshots for each volume incorporated into the backup quotas for the respective tenant. Additionally, a commensurate increase in volume quotas for the tenant is advisable, as Trilio briefly materializes a volume from the snapshot to access data for backup purposes.

During the restoration process, Trilio generates supplementary instances and Cinder volumes. To facilitate seamless restore operations, tenants should maintain adequate quota levels for Nova instances and Cinder volumes. Failure to meet these quota requirements may lead to disruptions in restoration procedures.

AWS S3 Eventual Consistency

The AWS S3 object consistency model includes:

1. Read-after-write

2. Read-after-update

3. Read-after-delete

Each of these models explains how an object becomes consistent after being created, updated, or deleted. None of these methods ensures strong consistency, leading to a delay before an object becomes fully consistent.

Although Trilio has introduced measures to address AWS S3's eventual consistency limitations, the exact time an object achieves consistency cannot be predicted through deterministic means.

There is no official statement from AWS on how long it takes for an object to reach a consistent state. However, read-after-write has a shorter time to reach consistency compared to other IO patterns. Therefore, our solution is designed to maximize the read-after-write IO pattern.

AWS Region Considerations

The time in which an object reaches eventual consistency also depends on the AWS region.

For instance, the AWS-standard region doesn't offer the same level of strong consistency as regions like us-east or us-west. Opting for these regions when setting up S3 buckets for Trilio is advisable. While fully avoiding the read-after-update IO pattern is complex, we've introduced significant access delays for objects to achieve consistency over longer periods. On rare occasions when this does happen, it will cause a backup failure and require a retry.

Trilio Cluster

Trilio can be deployed as a single node or a three-node cluster. It is highly recommended that Trilio is deployed as a three-node cluster for fault tolerance and load balancing. Starting with the 3.0 release, Trilio requires additional IP or FQDN for the cluster and is required for both single-node and three-node deployments. Cluster IP a.k.a virtual IP is used for managing clusters and is used to register the Trilio service endpoint in the keystone service catalog.

Deployment Scripts

git clone -b 5.2.1 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/redhat-director-scripts/rhosp16/

git clone -b 5.2.1 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/redhat-director-scripts/rhosp17/

git clone -b 5.2.1 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/kolla-ansible/ansible/

Container Information

Package Repositories & Versions

All packages are Python 3 (py3.6 - py3.10) compatible only.

https://pypi.fury.io/trilio-5-2/

deb [trusted=yes] https://apt.fury.io/trilio-5-2/ /

https://yum.fury.io/trilio-5-2/

[trilio-fury]
name=Trilio Gemfury Private Repo
baseurl=https://yum.fury.io/trilio-5-2/
enabled=1
gpgcheck=0

Trilio for OpenStack Compatibility Matrix

1. Caracal support is currently for Rocky 9 only.

NFS & S3 Support

All versions of T4O-5.x releases support NFSv3 and S3 as backup targets on all the compatible distributions.

Encryption Support

All versions of T4O-5.x releases support encryption using Barbican service on all the compatible distributions.

QEMU Guest Agent

QEMU guest agent is highly recommended to be running inside the VM that being backed up to avoid data corruption during backup process. QEMU Guest Agent is a daemon that runs inside a virtual machine (VM) and communicates with the host system (the hypervisor) to provide enhanced management and control of the VM. It is an essential component in virtualized environments specially OpenStack.

Trilio Branch

Branch of the repository to be used for DevOps scripts.

trilio_branch : 5.2.3

Deployment Scripts

git clone -b 5.2.3 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/redhat-director-scripts/rhosp16/

git clone -b 5.2.3 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/redhat-director-scripts/rhosp17/

git clone -b 5.2.3 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/kolla-ansible/ansible/

Container Information

Package Repositories & Versions

All packages are Python 3 (py3.6 - py3.10) compatible only.

https://pypi.fury.io/trilio-5-2/

deb [trusted=yes] https://apt.fury.io/trilio-5-2/ /

https://yum.fury.io/trilio-5-2/

[trilio-fury]
name=Trilio Gemfury Private Repo
baseurl=https://yum.fury.io/trilio-5-2/
enabled=1
gpgcheck=0

VM Migration Tool

Docker Container Tag

trilio/trilio-migration-vm2os:5.2.3

Deployment Scripts

Container Information

Package Repositories & Versions

All packages are Python 3 (py3.6 - py3.10) compatible only.

Deployment Scripts

Container Information

Package Repositories & Versions

All packages are Python 3 (py3.6 - py3.10) compatible only.

Trilio and Canonical have started a partnership to ensure a native deployment of Trilio using JuJu Charms.

Those JuJu Charms are publicly available as Open Source Charms.

Juju charms for OpenStack Antelope & Bobcat release

Prerequisite

A Canonical OpenStack base setup deployed for a required release like Jammy Antelope/Bobcat. Refer

Steps to install the Trilio charms

1. Export the OpenStack base bundle

2. Create a Trilio overlay bundle as per the OpenStack setup release using the charms given above.

Alternatively, triliovault-cfg-scripts repository can be cloned to get the sample overlay bundles.

Following table provides the details of various values to be updated in overlay bundle.

3. T4O Deployment

3.1] Do a dry run to check if the Trilio bundle is working

3.2] Trigger deployment

3.3] Wait until all the Trilio units are deployed successfully. Check the status via juju status command.

4. Post Deployment Steps

4.1] Once the deployment is complete, perform the below operations:

a. Create cloud admin trust & add licence

Note: Reach out to the Trilio support team for the license file.

5. Verify the T4O Deployment

5.1] After T4O deployment steps are over, it can take sometime for all units to get deployed successfully. Deployment is considered successful when all the units show Unit is Ready in the message column.

To verify the same, following command (& sample output) can be used to fetch the Trilio units/applications.

6. Troubleshooting T4O Deployment

6.1] To debug any specific unit : juju debug-log --include <UNIT_NAME_IN_ERROR>

Eg. If trilio-wlm/6 unit is in 'error' state, it's logs can be fetched using following command. Specific correct unit number from respective deployment using juju status command.

juju debug log --include trilio-wlm/6

For multipath enabled environments, perform the following actions

1. log into each nova compute node

2. add uxsock_timeout with value as 60000 (i.e. 60 sec) in /etc/multipath.conf

3. restart tvault-contego service

After the installation and configuration of Trilio for Openstack did succeed the following steps can be done to verify that the Trilio installation is healthy.

On the Controller node

Make sure below containers are in a running state, triliovault-wlm-cron would be running on only one of the controllers in case of multi-controller setup.

• triliovault_datamover_api

• triliovault_wlm_api

• triliovault_wlm_scheduler

• triliovault_wlm_workloads

• triliovault-wlm-cron

If the containers are in a restarting state or not listed by the following command then your deployment is not done correctly. Please recheck if you followed the complete documentation.

After successful deployment, triliovault-wlm-cron service would get added in pcs cluster as a cluster resource, you can verify through pcs status command

Verify the HAproxy configuration under:

On Compute node

Make sure the Trilio Datamover container is in running state and no other Trilio container is deployed on compute nodes.

Check if provided backup target is mounted well on Compute host.

On the node with Horizon service

Make sure the horizon container is in a running state. Please note that the Horizon container is replaced with Trilio's Horizon container. This container will have the latest OpenStack horizon + Trilio horizon plugin.

If the Trilio Horizon container is in the restarted state on RHOSP 16.1.8/RHSOP 16.2.4 then use the below workaround

About the workloadmgr CLI client

The workloadmgr CLI client is provided as rpm and deb packages.

The following operating systems have been verified with the installation:

• CentOS8

• Ubuntu 18.04, Ubuntu 20.04

Installing the workloadmgr client will automatically install all required OpenStack clients as well.

Further, the installation of the workloadmgr client will integrate the client into the global OpenStack Python client, if available.

Install workloadmgr client rpm package on CentOS8

The Trilio workload manager CLI client has several requirements that need to be met before the client can be installed without dependency issues.

Preparing the workloadmgr client installation

The following steps need to be done to prepare the installation of the workloadmgr client:

1. Add required repositories

   1. epel-release

   2. centos-release-openstack-train

2. install base packages

   1. yum -y install epel-release

   2. yum -y install centos-release-openstack-train

Installing the workloadmgr client

Installing from the Trilio online repository

To install the latest available workloadmgr package for a Trilio release from the Trilio repository the following steps need to be done:

Create the Trilio yum repository file /etc/yum.repos.d/trilio.repo Enter the following details into the repository file:

[trilio]
name=Trilio Repository
baseurl=https://yum.fury.io/trilio-5-0/
enabled=1
gpgcheck=0

Install the workloadmgr client issuing the following command:

yum install python-3-workloadmgrclient-el8

Install workloadmgr client deb packages on Ubuntu

The Trilio workloadmgr client packages for Ubuntu are only available from the online repository.

Preparing the workloadmgr client installation

There is no preparation required. All dependencies are automatically resolved by the standard repositories provided by Ubuntu.

Installing the Workloadmgr client

Installing from the Trilio online repository

To install the latest available workloadmgr package for a Trilio release from the Trilio repository the following steps need to be done:

Create the Trilio yum repository file /etc/apt/sources.list.d/fury.list Enter the following details into the repository file:

deb [trusted=yes] https://apt.fury.io/trilio-5-0/ /

run apt update to make the new repository available.

The apt package manager is used to install the workloadmgr client package:

apt-get install python3-workloadmgrclient

Trilio uses a widely used tool virt-v2v for migration which is one of the open-source tools by Red Hat. The below-mentioned limitations are from this tool and thus a limitation for Trilio as well:

• virt-v2v cannot change the default kernel in the GRUB2 configuration, and the kernel configured in the VM is not changed during the conversion, even if a more optimal version of the kernel is available on the VM.

• After converting a virtual machine to KVM, the name of the VM's network interface may change, and thus requires manual configuration.

• Trilio does not support nested virtualization OpenStack deployments for VMware migration.

• For Windows VMs, post migration all the non system disk would be in offline state. You would need to login to the guest VM and manually bring all the attached disk online, follow the below steps:

1. To bring a disk online, run command prompt as an administrator.
2. Run command diskmgmt.msc
3. Right click on the offline disk and change it to Online.

• Windows VMs migrated with the Dry-Run type won't be bootable on OpenStack, as complete boot information won't be available until the guest VM is shut down.

Please refer to Post-Conversion Tasks for more information.

Other Limitations specific to VM Migration:

• If any VM is having independent (i.e. not mounted on "/") /usr partition, then migration of such VM will fail with below error:

virt-v2v: error: inspection could not detect the source guest (or physical
machine).

Assuming that you are running virt-v2v/virt-p2v on a source which is
supported (and not, for example, a blank disk), then this should not
happen.

Inspection field 'i_arch' was 'unknown'.

Check the supported OpenStack distributions here and follow their deployment guides to install or upgrade to the latest version of T4O.

Enable Migration Tab on Horizon

Follow below steps to enable Migration Tab on Horizon.

1. Login to horizon container.

2. set DISABLED = False in following panel files.

_1806_migration_panel.py
_1807_migration_panel_group.py

These files can be found at <openstack_dashboard_installed_package_path>/enabled/

Example: /usr/share/openstack-dashboard/openstack_dashboard/local/enabled/ for RHOSP17.1

1. Restart horizon container.

The high-level process is the same for all Distributions.

1. Uninstall the Horizon Plugin or the Trilio Horizon container

2. Uninstall the datamover-api container

3. Uninstall the datamover container

4. Uninstall the workloadmgr container

Introduction

VM Migration Tool is a stand alone utility provided by Trilio for effective migration of VMs from VMWare to OpenStack. This will assist users in planning the migration, creating any missing artifacts, and providing a holistic view of the migration process at an organizational level.

Current document provides the steps to be followed for deploying respective tool.

1] Pre-requisites

A Linux VM (CentOS OR Ubuntu will do) with docker installed on it.

2] Prepare working dir and contents

2.1] Clone public repository triliovault-cfg-scripts on the standalone VM (at any convenient location) created for deploying the VM Migration tool.

git clone -b {{ trilio_branch }} https://github.com/trilioData/triliovault-cfg-scripts.git

2.2] On the VM, copy nginx, env, docker-compose.yml & vmosmapping.conf from triliovault-cfg-scripts to /opt directory.

cp -r triliovault-cfg-scripts/migration-vm2os/nginx /opt
cp triliovault-cfg-scripts/migration-vm2os/env /opt
cp triliovault-cfg-scripts/migration-vm2os/docker-compose.yml /opt
cp triliovault-cfg-scripts/migration-vm2os/vmosmapping.conf /opt

2.3] Set values against params in /opt/env file.

2.4] Set values against params in /opt/vmosmapping.conf file.

3] Deploy VM Migration Tool

3.1] To deploy the VM Migration Tool on standalone VM, please execute below command in background mode.

docker compose -f /opt/docker-compose.yml --env-file /opt/env up &

3.2] Deployment checks

• Run docker ps -a ; 4 containers should be running, viz redis, nginx, trilio_vm2os & opt-worker-1.

• Sample output below (Forwarded port number can be different as per the value provided in /opt/env file)

CONTAINER ID   IMAGE                                              COMMAND                  CREATED         STATUS         PORTS                                               NAMES
1ed651c084d4   nginx:latest                                       "/docker-entrypoint.…"   5 seconds ago   Up 3 seconds   80/tcp, 0.0.0.0:5085->5085/tcp, :::5085->5085/tcp   nginx
8ff9f81b9913   trilio/trilio-migration-vm2os:5.2.3-dev-maint3-3   "gunicorn --config g…"   5 seconds ago   Up 4 seconds                                                       trilio_vm2os
1f91f14ad011   trilio/trilio-migration-vm2os:5.2.3-dev-maint3-3   "celery -A run.celer…"   5 seconds ago   Up 4 seconds                                                       opt-worker-1
0bb5574ad56b   redis:latest                                       "docker-entrypoint.s…"   5 seconds ago   Up 4 seconds   0.0.0.0:6379->6379/tcp, :::6379->6379/tcp           redis

4] Access Tool Dashboard

Respective Migration tool can be accessed with URL : <VM_IP>:<NGINX_PORT> Example

http://192.168.6.25:5085/

For logging into dashboard, use the credentials provided under [user] section in /opt/vmosmapping.conf file

After all Trilio components are installed, the license can be applied.

The license can be applied either through the Admin tab in Horizon or the CLI

Apply license through Horizon

To apply for the license through Horizon follow these steps:

1. Login to Horizon using admin user.

2. Click on the Admin Tab.

3. Navigate to Backups-Admin

4. Navigate to Trilio

5. Navigate to License

6. Click "Update License"

7. Read the license agreement

8. Click on "I accept the terms in the License Agreement"

9. Click on "Next"

10. Click "Choose File"

11. Choose license file on the client system

12. Click "Apply"

Apply license through CLI

Read and accept the End User License Agreement to complete the license application.

OpenStack distributions

Guest VMs

• Refer to the "Supporting Conversions" section of this document.

Preparing OpenStack + Trilio

You will need to be ready with below-mentioned resources before starting with the deployment:

VMware Resources

vCenter access information

• vCenter access URL

• vCenter access Username and Password

• SSL certificate for secured connection to VCenter (if not available, the SSL verification has to be marked false during configuration)

• The minimum set of permissions required for a vCenter user to successfully migrate VMs are as follows:

VMware vSphere Virtual Disk Development Kit 7.0

• Trilio requires this kit for the VM migration feature to work. You can download it from the VMware Customer Connect portal by accessing the below link

• https://customerconnect.vmware.com/downloads/get-download?downloadGroup=VDDK700

• Make sure you download the below-mentioned file

  Product	VMware vSphere Virtual Disk Development Kit 7.0

Windows VMs Preparation

• VMware tools need to be installed and should be in running state on all the windows VMs which are to be migrated. Alternatively, the guest OS may be shutdown but they should not be in power off state

OpenStack Resources

Glance Images

• For migrating UEFI booted VMware VMs, the OpenStack requires to have the instances with UEFI boot information and so as Trilio.

• So before doing the migration, the user needs to create a Glance image (any image) with following metadata properties:

  1. set hw_firmware_type property to UEFI

  2. set hw_machine_type property to q35

• This is not required for BIOS booted VMware VMs

Instace Flavors

• Trilio does not create any flavor for the migrating VMs instead it uses the available flavors.

• To avoid any booting issues post-migration, the user needs to have the correct flavors created and available for selection while initiating the migration.

Preparing the Guest VMs on ESXi

To leverage the Dry-run and Warm migration features, the vCenter and ESXi should support the Change Block Tracking(CBT) feature. Make sure that all the data disks attached to the VM, have the CBT enabled.

To enable the CBT, follow the instructions at VMware Knowledge Base

Also understand the limitations Trilio has described in this page.

Deployment Scripts

git clone -b 5.2.0 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/redhat-director-scripts/rhosp16/

git clone -b 5.2.0 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/redhat-director-scripts/rhosp17/

git clone -b 5.2.0 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/kolla-ansible/ansible/

Container Information

Package Repositories & Versions

All packages are Python 3 (py3.6 - py3.10) compatible only.

https://pypi.fury.io/trilio-5-2/

deb [trusted=yes] https://apt.fury.io/trilio-5-2/ /

https://yum.fury.io/trilio-5-2/

[trilio-fury]
name=Trilio Gemfury Private Repo
baseurl=https://yum.fury.io/trilio-5-2/
enabled=1
gpgcheck=0

1] Plan for Deployment

Refer to the below-mentioned acceptable values for the placeholders triliovault_tag, trilio_branch and kolla_base_distro , in this document as per the Openstack environment:

Trilio requires OpenStack CLI to be installed and available for use on the Kolla Ansible Control node.

1.1] Select backup target type

Backup target storage is used to store backup images taken by Trilio and details needed for configuration:

Following backup target types are supported by Trilio. Select one of them and get it ready before proceeding to the next step.

a) NFS

Need NFS share path

b) Amazon S3

- S3 Access Key - Secret Key - Region - Bucket name

c) Other S3 compatible storage (Like, Ceph based S3)

- S3 Access Key - Secret Key - Region - Endpoint URL (Valid for S3 other than Amazon S3) - Bucket name

2] Clone Trilio Deployment Scripts

Clone triliovault-cfg-scripts GitHub repository on Kolla ansible server at '/root' or any other directory of your preference. Afterwards, copy Trilio Ansible role into Kolla-ansible roles directory

git clone -b {{ trilio_branch }} https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/kolla-ansible/
mkdir -p /usr/local/share/kolla-ansible/ansible/roles/triliovault

# For Rocky and Ubuntu Zed and Antelope
cp -R ansible/roles/triliovault /usr/local/share/kolla-ansible/ansible/roles/

# For Rocky and Ubuntu Bobcat and Caracal
cp -R ansible/roles/triliovault-bobcat/* /usr/local/share/kolla-ansible/ansible/roles/triliovault/

3] Hook Trilio deployment scripts to Kolla-ansible deploy scripts

3.1] Add Trilio global variables to globals.yml

## For Rocky and Ubuntu
- Take backup of globals.yml
cp /etc/kolla/globals.yml /opt/

- Append Trilio global variables to globals.yml for Zed
cat ansible/triliovault_globals_zed.yml >> /etc/kolla/globals.yml

- Append Trilio global variables to globals.yml for Antelope
cat ansible/triliovault_globals_2023.1.yml >> /etc/kolla/globals.yml

- Append Trilio global variables to globals.yml for Bobcat
cat ansible/triliovault_globals_2023.2.yml >> /etc/kolla/globals.yml

- Append Trilio global variables to globals.yml for Caracal
cat ansible/triliovault_globals_2024.1.yml >> /etc/kolla/globals.yml

3.2] Add Trilio passwords to kolla passwords.yaml

Generate triliovault passwords and append triliovault_passwords.yml to /etc/kolla/passwords.yml.

cd ansible
./scripts/generate_password.sh

## For Rocky and Ubuntu
- Take backup of passwords.yml
cp /etc/kolla/passwords.yml /opt/

- Append Trilio global variables to passwords.yml 
cat triliovault_passwords.yml >> /etc/kolla/passwords.yml

3.3] Append Trilio site.yml content to kolla ansible’s site.yml

# For Rocky and Ubuntu
- Take backup of site.yml
cp /usr/local/share/kolla-ansible/ansible/site.yml /opt/

- Append Trilio site variables to site.yml for Zed
cat ansible/triliovault_site_yoga.yml >> /usr/local/share/kolla-ansible/ansible/site.yml    

- Append Trilio site variables to site.yml for Antelope
cat ansible/triliovault_site_2023.1.yml >> /usr/local/share/kolla-ansible/ansible/site.yml

- Append Trilio site variables to site.yml for Bobcat
cat ansible/triliovault_site_2023.2.yml >> /usr/local/share/kolla-ansible/ansible/site.yml

- Append Trilio site variables to site.yml for Caracal
cat ansible/triliovault_site_2024.1.yml >> /usr/local/share/kolla-ansible/ansible/site.yml

3.4] Append triliovault_inventory.txt to your cloud’s kolla-ansible inventory file.

For example:
If your inventory file name path '/root/multinode' then use following command.

cat ansible/triliovault_inventory.txt >> /root/multinode

3.5] Configure multi-IP NFS

On kolla-ansible server node, change directory

cd triliovault-cfg-scripts/common/

Edit file 'triliovault_nfs_map_input.yml' in the current directory and provide compute host and NFS share/ip map.

vi triliovault_nfs_map_input.yml

The triliovault_nfs_map_imput.yml is explained here.

Update PyYAML on the kolla-ansible server node only

pip3 install -U pyyaml

Expand the map file to create one to one mapping of compute and nfs share.

python ./generate_nfs_map.py

Result will be in file - 'triliovault_nfs_map_output.yml'

Validate output map file

Open file 'triliovault_nfs_map_output.yml

vi triliovault_nfs_map_output.yml

available in the current directory and validate that all compute nodes are covered with all necessary nfs shares.

Append this output map file to 'triliovault_globals.yml' File Path: '/home/stack/triliovault-cfg-scripts/kolla-ansible/ansible/triliovault_globals.yml’