Release Versions

Packages

Containers and Gitbranch

Changelog

Fixed issues and bugs

Snapshot getting hung in upload phase for multipath environments

An issue has been fixed, which lead to Snapshots being stuck in the upload phase in the case of using an FC multipath connected Cinder Storage.

Backups fail due to an old Trilio Cinder Snapshot not being in available state

An issue has been fixed, which lead to Trilio-created Cinder Snapshots, which are not in the state available, blocking the creation of new Cinder Snapshots.

Latin characters in restore names lead to failed restores

The support for Latin characters has been enhanced to no longer impact restores.

Trilio 4.2 introduces new features and capabilities:

• Backup and Recovery of encrypted Cinder volumes (Barbican support)

• Encryption of Workloads (Barbican support)

• Support for multi-IP NFS backup targets

• Database clean up utility

• Backup rebase utility

Trilio 4.2.64

Release Versions

Packages

Containers and Gitbranch

Backup and Recovery of encrypted Cinder Volumes (Barbican support)

The OpenStack Barbican service enables the OpenStack Cinder service to provide encrypted Volumes. These volumes are software encrypted by the Cinder service with the secret used for the encryption being managed by the Barbican service.

Trilio for OpenStack 4.2 is integrating into OpenStack Barbican to enable T4O to provide native backup and recovery of the encrypted Cinder volume.

Any workload containing an encrypted Cinder volume has to create encrypted backups too. It is not possible to create unencrypted Workloads for encrypted Cinder Volumes.

T4O 4.2 is providing encryption on the Workload level. All VMs that are part of an encrypted Workload will have their Cinder Volume data encrypted.

Encryption of Workloads (Barbican support)

The integration of Trilio for OpenStack 4.2 into Barbican enables T4O to provide encryption for the qcow2-data component of the Trilio backups. The json files containing the backed-up OpenStack metadata stays unencrypted.

This functionality requires the OpenStack Barbican service to be present. Without the Barbican service, the possibilities to encrypt Workloads will not be shown inside Horizon.

T4O 4.2 is only consuming secrets from Barbican. It is not creating, editing, or deleting any secret inside Barbican.

To utilize encrypted Workloads the Trilio trustee role needs to be able to engage with the Barbican service to read and fetch secrets from Barbican. The only default roles enabled with these permissions are the admin and the creator roles.

The possibility to encrypt a Workload is only provided during Workload creation. Once a workload has been created it is no longer possible to change, whether the Workload is encrypted or not.

Every encrypted Workload is consuming a unique Barbican secret. It is not possible to assign the same secret to two Workloads.

The following secret configurations are supported.

A default Barbican installation will generate secrets of the following type:

• Alghorithm: AES-256

• Mode: cbc

• content type: application/octet-stream

• payload-content-encoding: base64

• secret type: opaque

• payload: plaintext

Encrypted Workloads can be migrated to different projects or clouds just like normal Workloads. The Barbican secrets used for the encryption need to be made available inside the target environment. This already applies when the Workload gets assigned to a different owner.

Support for Multi-IP NFS backends

Many Trilio customers are using software-defined storage solutions for a scalable backup target. These software-defined storage solutions often provide the capability to spread read and write operations over multiple nodes of the storage cluster. Each of these nodes has its own access IP to interact with. All nodes are still writing to the same logical volume, which is available through the NFS protocol.

Trilio for Openstack 4.2 is supporting such solutions by enabling the have different IPs for the same backup target for each Datamover.

Every Datamover and the Trilio appliance are still consuming one NFS path per Volume.

This is achieved by changing the method to calculate the T4O mount point. It is now only considering the volume path instead of the complete NFS path. Example below.

# echo -n /Trilio_Backup | base64
L1RyaWxpb19CYWNrdXA=
/var/triliovault-mounts/L1RyaWxpb19CYWNrdXA=/

Database cleanup utility

Trilio for OpenStack is following an older OpenStack database schema and model. This model contains, that no data inside the database ever gets truly deleted. Only a flag is set to showcase that the dataset is no longer considered active.

This has the advantage that it is always possible to trace back any objects and their existence timeline inside the database for analytical purposes.

The big disadvantage of this model is that the database itself is ever-growing with every activity that T4O is doing.

Over time it is possible for the database to reach sizes, that normal activities like listing Workloads are taking so long, that other tasks like taking backups are impacted.

To counter this issue does Trilio provide a new utility, which will delete all no longer required database entries to reduce the load of the work when the T4O solution needs to access the database.

Backup rebase utility

Trilio for OpenStack is providing full synthetic backups utilizing the backing-file functionality of qcow2 images.

This functionality enables Trilio backups to run incremental forever, without having to restore every incremental backup. Instead only the latest backup needs to be restored as all missing blocks are fetched through the backing file chain from older backups.

The backing file of a particular backup is hardcoded inside the qcow2 image itself. These backing files need to consist of the full path and can't use relative paths. An example is provided below.

qemu-img info 85b645c5-c1ea-4628-b5d8-1faea0e9d549
image: 85b645c5-c1ea-4628-b5d8-1faea0e9d549
file format: qcow2
virtual size: 1.0G (1073741824 bytes)
disk size: 21M
cluster_size: 65536
backing file: /var/triliovault-mounts/MTAuMTAuMi4yMDovdXBzdHJlYW0=/workload_3c2fbee5-ad90-4448-b009-5047bcffc2ea/snapshot_f4874ed7-fe85-4d7d-b22b-082a2e068010/vm_id_9894f013-77dd-4514-8e65-818f4ae91d1f/vm_res_id_9ae3a6e7-dffe-4424-badc-bc4de1a18b40_vda/a6289269-3e72-4085-adca-e228ba656984
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

T4O is using a base64 hash value inside this path for NFS backup targets. This backup target path needs to be resolvable for the qcow2 image to find its backing file.

The hash value of the mount path can change when the backup is moved to a different backup target or when the T4O calculation method is changed like it is for T4O 4.2.

Trilio is now providing a utility that will change the backing file path for a given workload.

Known issues

[Canonical OpenStack] Selective restore fails for migrated workloads

Observation:

• after migrating workloads to a different Canonical OpenStack environments restore fails with permission denied error in tmp directory

Workaround:

Run the {{sudo sysctl fs.protected_regular=0}} command on wlm units.

[Intermittent Canonical OpenStack Queens] In-Place restore doesn't work with ext3/ext4 file systems

Observation:

• In-Place restore succeeds logically

• Data is not getting replaced

Workaround:

• Running the In-Place restore a second time

• Running a selective restore and reattaching the restored Volume

[intermittent] [Canonical OpenStack] Retention not honored after mount/unmount operation

Observation:

• After a mount or unmount operation the ownership of the backup stays qemu:qemu which leads to T4O no longer being able to run required merge commands

Workaround:

• identify the workload with failed retention policy

• run: chown -R nova:nova <WL_DIR>

• verify next backup applies the retention policy

CLI commands get-importworkload-list and get-orphaned-workloads-list show wrong list of workloads

Observation:

• independent of the command all workloads located on the backup target are listed

Workaround:

• use project id in command to show only workloads that can be imported for that project workloadmgr workload-get-importworkloads-list --project_id <project_id>

File-search not displaying files in lvm created logical volumes

Observation:

• File search returns an empty list for lvm controlled logical volumes

• fdisk created logical volumes work as desired

File-search not displaying files when root directory doesn't contain read permissions for group

Observation:

• File search returns an empty list when root doesn't have read permissions for groups

• File search is run as user nova

Unable to create encrypted Workload if T4O gets reconfigured with creator trustee role

Observation:

• T4O initially configured with trustee role member (not able to create encrypted workloads)

• After reconfiguration to trustee role creator encrypted workloads are still not creatable

Workaround:

• After reconfiguration create one unencrypted workload

• Then create encrypted workloads as required

Post restore of encrypted incremental snapshots CentOS instance is not getting booted

Observation:

• Restoring of a CentOS-based instance fails to start the instance in the case of restoring from an encrypted incremental snapshot

Workaround:

• Use only full backups for CentOS-based instances in combination with encrypted Workloads

Release Versions

Packages

Containers and Gitbranch

Changelog

New Qualifications

This Hotfix extends the Support Matrix of T4O 4.2 as follows:

• Canonical Openstack Wallaby based on Focal (20.04) Support

• Kolla Ansible Openstack Wallaby on Ubuntu 20.04 and CentOS Stream

• Openstack Ansible Wallaby on Ubuntu 20.04 and CentOS Stream

Fixed Bugs and issues

Rebase permission error

An issue has been fixed which prevented the correct rebase of T4O incremental backups in the case that root didn't have the required permissions on the backup target.

Prerequisites

To use this hotfix (4.2.HF3).

1. Customers (except Canonical Openstack) and having Openstack Ussuri OR Openstack Victoria need to have an already deployed and working TVO-4.2 GA.

2. Customers (except Canonical Openstack) and having Openstack Wallaby need to follow the T4O-4.2 GA deployment process and directly upgrade to 4.2.HF3 containers/packages. The high-level flow below:

   1. Deplo T4O-4.2 GA appliance.

   2. Upgrade to 4.2.HF3 packages on the appliance.

   3. Kolla

      1. Deploy Trilio components via 4.2.HF3 containers/packages on Openstack Wallaby.

   4. Openstack Ansible

      1. Deploy Trilio components Openstack Wallaby [This will deploy 4.2 GA packages]

      2. Upgrade TrilioVault packages to 4.2.HF3 on Openstack Wallaby.

   5. Configure the Trilio appliance.

3. Canonical users having Openstack Ussuri Or Openstack Victoria can either upgrade (on top of 4.2 GA) using Trilio upgrade documents OR do a new deployment using 4.2 Deployment documents.

4. Canonical users having Openstack Wallaby need to do a new deployment using 4.2 Deployment documents.

Release Scope

Current Hotfix release targets the following:

1. High-level Qualification (via Sanity & Functional suites' execution) of T4O with Ussuri, Victoria & Wallaby Openstack.

2. Verification of Jira issues targeted for 4.2. release.

3. As part of the new process, the delivery will be via packages; end users would need to do the rolling upgrade on top of 4.2 GA.

Release Artifacts

Branch Tag and Containers Tags

Resolved Issues

Deliverables

Package Added/Changed

Known Issues

Backup-as-a-Service

Trilio is an add on service to OpenStack cloud infrastructure and provides backup and disaster recovery functions for tenant workloads. Trilio is very similar to other OpenStack services including nova, cinder, glance, etc and adheres to all tenets of OpenStack. It is a stateless service that scales with your cloud.

Main Components

Trilio has four main software components:

1. Trilio ships as a QCOW2 image. User can instantiate one or more VMs from the QCOW2 image on a standalone KVM boxes.

2. Trilio API is a python module that is installed on all OpenStack controller nodes where the nova-api service is running.

3. Trilio Datamover is a python module that is installed on every OpenStack compute nodes

4. Trilio horizon plugin is installed as an add on to horizon servers. This module is installed on every server that runs horizon service.

Service Endpoints

Trilio is both a provider and consumer into OpenStack ecosystem. It uses other OpenStack services such as nova, cinder, glance, neutron, and keystone and provides its own service to OpenStack tenants. To accomodate all possible OpenStack deployments, Trilio can be configured to use either public or internal URLs of services. Likewise Trilio provides its own public, internal and admin URLs.

Network Topology

This figure represents a typical network topology. Trilio exposes its public URL endpoint on public network and Trilio virtual appliances and data movers typically use either internal network or dedicated backup network for storing and retrieving backup images from backup store.

Release Versions

Packages

Deliverables against T4O-4.2.8

Following packages changed/added in the current release

Containers and Gitbranch

Changelog

• Support for Kolla Zed (Ubuntu & Rocky) OpenStack

• Verification of Jira issues targeted for 4.2.8 release

Fixed Bugs and issues

1. Backup failing on cinder backend Fiber Channel.

2. Single VM backup (configurable) at a time out of workload having multiple VMs.

3. Snapshot and Selective/One Click Restore failing on quobyte.

4. Inplace restore fails saying no space left on device.

5. Specific workload fails to be restored; says VMDiskResourceSnap could not be found.

6. Restores are failing; error says security group rules already exist.

7. Snapshot intermittently fails with IO error while backup uploading.

8. Stopping multipath volumes is taking huge time.

9. Upgrading or deploying RHOSP 16.2.5 results in TrilioVaultWLM service and respective endpoints disappearing.

Known issues

Deleting snapshots show status as available in horizon UI

Observation : Snapshot for which delete operation is in-progress from UI , its status is showing as available instead deleting.

Workaround:

Wait for sometime to complete all the delete operations.Eventually all the snapshots will be deleted successfully.

Release Versions

Packages

Deliverables against TVO-4.2.HF5

Following packages changed/added in the current release

Containers and Gitbranch

Changelog

• End of support for Openstack Ansible from 4.2.HF5

• Bug fixes targeted for 4.2.HF5 release

Fixed Bugs and issues

1. Kolla openstack source based container for wallaby

2. Cinder volumes NFS folder not mounted to datamover container

3. Cinder has IBM GPFS mounted as NFS

4. RHOSP16.1 breaks horizon 4.2hf4

5. Backup/restore does not work when NFS has access to Nova user just NOVA

6. Snapshot for encrypted volume is failing with error

7. Cinder attached volume backup is taking time and also created 4 temporary volumes

8. Cinder booted VM backup failing

The Trilio Appliance is delivered as qcow2 image and runs as VM on top of a KVM Hypervisor.

Creating the cloud-init image

The Trilio appliance is utilizing cloud-init to provide the initial network and user configuration.

Cloud-init is reading it's information either from a metadata server or from a provided cd image. Trilio is utilizing the cd image.

Needed tools

To create the cloud-init image it is required to have genisoimage available.

#For RHEL and centos
yum install genisoimage
#For Ubuntu 
apt-get install genisoimage

Providing the Metadata

Cloud-init is using two files for it's metadata.

The first file is called meta-data and contains the information about the network configuration. Below is an example of this file.

[root@kvm]# cat meta-data
instance-id: triliovault
network-interfaces: |
   auto eth0
   iface eth0 inet static
   address 158.69.170.20
   netmask 255.255.255.0
   gateway 158.69.170.30

   dns-nameservers 11.11.0.51
local-hostname: localhost

The second file is called user-data and contains little scripts and information to set up for example the user passwords. Below is an example of this file.

[root@kvm]# cat user-data
#cloud-config
chpasswd:
  list: |
    root:password1
    stack:password2
  expire: False

creating the image file

The image is getting created using genisoimage follwing this general command:

genisoimage -output <name>.iso -volid cidata -joliet -rock </path/user-data> </path/meta-data>

An example of this command is shown below.

genisoimage  -output tvault-firstboot-config.iso -volid cidata -joliet -rock user-data meta-data

Spining up the Trilio appliance

After the cloud-init image has been created the TriloVault appliance can be spun up on the desired KVM server.

Extract the Trilio QCOW2 tar file using the following command :

tar Jxvf TrilioVault_file.tar.xz

See below an example command, how to spin up the Trilio appliance using virsh and the created iso image.

virt-install -n triliovault-vm  --memory 24576 --vcpus 8 \
--os-type linux \ 
--disk tvault-appliance-os-3.0.154.qcow2,device=disk,bus=virtio,size=40 \
--network bridge=virbr0,model=virtio \
--network bridge=virbr1,model=virtio \
--graphics none \
--import \
--disk path=tvault-firstboot-config.iso,device=cdrom

Uninstalling cloud-init after first boot

Once the Trilio appliance is up and running with it's initial configuration is it recommended to uninstall cloud-init.

To uninstall cloud-init, follow the example below.

sudo yum remove cloud-init

[RHOSP, TripleO and Kolla only] Verify nova UID/GID for nova user on the Appliance

Red Hat OpenStack, TripleO, and Kolla Ansible Openstack are using the nova UID/GID of 42436 inside their containers instead of 162:162 which is the standard in other Openstack environments.

Please verify that the nova UID/GID on the Trilio Appliance is still 42436,

[root@TVM1 ~]# id nova
uid=42436(nova) gid=42436(nova) groups=42436(nova),990(libvirt),36(kvm)

In case of the UID/GID being changed back to 162:162 follow these steps to set it back to 42436:42436.

1. Download the shell script that will change the user id

2. Assign executable permissions

3. Execute the script

4. Verify that the nova user and group id have changed to '42436'

## Download the shell script
$ curl -O https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/master/common/nova_userid.sh

## Assign executable permissions
$ chmod +x nova_userid.sh

## Execute the shell script to change 'nova' user and group id to '42436'
$ ./nova_userid.sh

## Ignore any errors and verify that 'nova' user and group id has changed to '42436'
$ id nova
   uid=42436(nova) gid=42436(nova) groups=42436(nova),990(libvirt),36(kvm)

Updating the appliance to the latest minor version

It is recommended to directly update the Trilio appliance to the latest version.

To do so follow the minor update guide provided here:

Trilio, by TrilioData, is a native OpenStack service that provides policy-based comprehensive backup and recovery for OpenStack workloads. The solution captures point-in-time workloads (Application, OS, Compute, Network, Configurations, Data and Metadata of an environment) as full or incremental snapshots. These snapshots can be held in a variety of storage environments including NFS AWS S3 compatible storage. With Trilio and its single click recovery, organizations can improve Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO). With Trilio, IT departments are enabled to fully deploy OpenStack solutions and provide business assurance through enhanced data retention, protection and integrity.

With the use of Trilio’s VAST (Virtual Snapshot Technology), Enterprise IT and Cloud Service Providers can now deploy backup and disaster recovery as a service to prevent data loss or data corruption through point-in-time snapshots and seamless one-click recovery. Trilio takes point-in-time backup of the entire workload consisting of compute resources, network configurations and storage data as one unit. It also takes incremental backups that only captures the changes that were made since last backup. Incremental snapshots save time and storage space as the backup only includes changes since the last backup. The benefits of using VAST for backup and restore could be summarized as below:

1. Efficient capture and storage of snapshots. Since our full backups only include data that is committed to storage volume and the incremental backups only include changed blocks of data since last backup, our backup processes are efficient and storages backup images efficiently on the backup media

2. Faster and reliable recovery. When your applications become complex that snap multiple VMs and storage volumes, our efficient recovery process will bring your application from zero to operational with just click of button

3. Easy migration of workloads between clouds. Trilio captures all the details of your application and hence our migration includes your entire application stack without leaving any thing for guess work.

4. Through policy and automation lower the Total Cost of Ownership. Our tenant driven backup process and automation eliminates the need for dedicated backup administrators, there by improves your total cost of ownership.

Release Versions

Packages

Deliverables against TVO-4.2.6

Following packages changed/added in the current release

Containers and Gitbranch

Changelog

• Verification of Jira issues targeted for 4.2.6 release

• Cohesity NFS/S3 storage backend support

Fixed Bugs and issues

1. Mounting a snapshot fails if the first disk of the File Recovery Manager VM is a CDROM

2. TVM HA configuration fails when one of the chosen Controller's hostname is part of any other TVM's hostname

3. Validation of keystone and s3 endpoint is stuck on the TVM UI

4. Temp volume creation taking too long

5. The httpd service is in a failed state on a freshly deployed T4O

6. The sshd option UseDNS should be set to "no" to avoid issues

Known issues

[RHOSP 16.1.8 and RHOSP 16.2.4] Trilio Horizon container in reboot loop

Observation : Post upgrade from a previous release/hotfix or on fresh deployment, the Trilio Horizon container is in a reboot loop

Workaround:

Either of the below workarounds should be performed on the controller where the issue occurs for the horizon pod.

option-1: Restart the memcached service on controller using systemctl (command: systemctl restart tripleo_memcached.service).

option-2: Restart the memcached pod (command: podman restart memcached).

[Snapshot mount] Unable to see correct content after mounting snapshot

Observation : Snapshot mount using RHEL8 File recovery manager image is not showing any mounted device

Workaround:

Use RHEL7 image for File recovery manager vm instead RHEL8.

[Backup failure for HPE nimble storage volumes with timeout receiving packet error

Observation : Snapshot is failing for VM's that have volumes backed by a Nimble iSCSI SAN with a "timeout receiving packet" error

Workaround: add uxsock_timeout parameter

log into respective datamover container and add uxsock_timeout with value as 60000 (i.e. 60 sec) in /etc/multipath.conf. Restart datamover container

[Input/Output error while writing to Cohesity NFS share]

Observation : Input/Output error during qemu-img convert operation while writing to Cohesity NFS share

Workaround: For cohesity NFS use below nfs options during Trilio configuration and datamover deployment and if issue still persists then increase timeo and retrans parameter values in nfs options

nfs_options = nolock,soft,timeo=600,intr,lookupcache=none,nfsvers=3,retrans=10

Release Versions

Packages

Deliverables against TVO-4.2.HF4

Following packages changed/added in the current release

Juju charms for Openstack Yoga release

Containers and Gitbranch

Changelog

New Qualifications

This Hotfix extends the Support Matrix of T4O 4.2 as follows:

• Fresh 4.2HF4 Trilio appliance build with wlm services on Python3.8 and multiple vulnerability fixes

• Kolla Ansible Openstack Yoga CentOS Stream 8, Ubuntu 20.04

• Canonical Yoga Ubuntu 20.04, Ubuntu 22.04

Fixed Bugs and issues

1. Irods nfs T4O not compatible

2. Backups failing with "Unable to call vast_instance"

3. Backup/restore does not work when NFS has access to Nova user just NOVA

4. Failed ansible-tvault-contego-extension : create trilio.filters for mount and unmount task

5. lxc packages not installed when using bare metal install

6. dmapi_all also includes bare metal hosts on non LXC deployments

7. Config failed with "iptables: Nothing to save" when TVM utilizing IPv6 | Tmobile/Red Hat | SFDC#2881

Trilio for OpenStack Compatibility Matrix

NFS & S3 Support:

All versions of Trilio for OpenStack support NFSv3 and S3 as backup targets.

Barbican Support:

Supported in RHOSP 16.1, 16.2, 17.0, Canonical Ussuri, Victoria, Wallaby, Yoga, Zed, Kolla Victoria, Wallaby, Yoga.

Not Supported in RHOSP 13, Canonical Queens, Stein, Train, TripleO Train.

Supported OS:

RHEL7, RHEL8, RHEL9, Ubuntu 18.04, Ubuntu 20.04, Ubuntu 20.04 source image, Ubuntu 22.04, CentOS7, CentOS Stream 8, CentOS Linux 8, Ubuntu 20.04 binary image, CentOS Stream 8 binary image, CentOS Stream 8 source image.

Deployment:

Deploy using the distributions corresponding deployment toolsets, including Red Hat Director, JuJu Charms, Ansible, and Director.

Compatibility Matrix Detailed View

Release Versions

Packages

Deliverables against T4O-4.2.7

Following packages changed/added in the current release

Containers and Gitbranch

Changelog

• Support for RHOSP17

• Support for Canonical zed Openstack

• Support for physical delete from DB against all delete operations

• Verification of Jira issues targeted for 4.2.7 release

Fixed Bugs and issues

1. workloadmgr trust-create command is failing with error - There are multiple role entities named 'admin'

2. Selective restore failing: Invalid volume

3. Backport DB cleanup code from 5.0 to 4.2

Known issues

Deleting snapshots show status as available in horizon UI

Observation : Snapshot for which delete operation is in-progress from UI , its status is showing as available instead deleting.

Workaround:

Wait for sometime to complete all the delete operations.Eventually all the snapshots will be deleted successfully.

Once the Trilio VM or the Cluster of Trilio VMs has been spun, the actual installation process can begin. This process contains of the following steps:

1. Install the Trilio dm-api service on the control plane

2. Install the Trilio datamover service on the compute plane

3. Install the Trilio Horizon plugin into the Horizon service

How these steps look in detail is depending on the Openstack distribution Trilio is installed in. Each supported Openstack distribution has its own deployment tools. Trilio is integrating into these deployment tools to provide a native integration from the beginning to the end.

• Test text