Release Versions

Packages

Containers and Gitbranch

Changelog

Fixed issues and bugs

Snapshot getting hung in upload phase for multipath environments

An issue has been fixed, which lead to Snapshots being stuck in the upload phase in the case of using an FC multipath connected Cinder Storage.

Backups fail due to an old Trilio Cinder Snapshot not being in available state

An issue has been fixed, which lead to Trilio-created Cinder Snapshots, which are not in the state available, blocking the creation of new Cinder Snapshots.

Latin characters in restore names lead to failed restores

The support for Latin characters has been enhanced to no longer impact restores.

Release Versions

Packages

Containers and Gitbranch

Changelog

New Qualifications

This Hotfix extends the Support Matrix of T4O 4.2 as follows:

• Canonical Openstack Wallaby based on Focal (20.04) Support

• Kolla Ansible Openstack Wallaby on Ubuntu 20.04 and CentOS Stream

• Openstack Ansible Wallaby on Ubuntu 20.04 and CentOS Stream

Fixed Bugs and issues

Rebase permission error

An issue has been fixed which prevented the correct rebase of T4O incremental backups in the case that root didn't have the required permissions on the backup target.

Trilio 4.2 introduces new features and capabilities:

• Backup and Recovery of encrypted Cinder volumes (Barbican support)

• Encryption of Workloads (Barbican support)

• Support for multi-IP NFS backup targets

• Database clean up utility

• Backup rebase utility

Trilio 4.2.64

Release Versions

Packages

Containers and Gitbranch

Backup and Recovery of encrypted Cinder Volumes (Barbican support)

The OpenStack Barbican service enables the OpenStack Cinder service to provide encrypted Volumes. These volumes are software encrypted by the Cinder service with the secret used for the encryption being managed by the Barbican service.

Trilio for OpenStack 4.2 is integrating into OpenStack Barbican to enable T4O to provide native backup and recovery of the encrypted Cinder volume.

Any workload containing an encrypted Cinder volume has to create encrypted backups too. It is not possible to create unencrypted Workloads for encrypted Cinder Volumes.

T4O 4.2 is providing encryption on the Workload level. All VMs that are part of an encrypted Workload will have their Cinder Volume data encrypted.

Encryption of Workloads (Barbican support)

The integration of Trilio for OpenStack 4.2 into Barbican enables T4O to provide encryption for the qcow2-data component of the Trilio backups. The json files containing the backed-up OpenStack metadata stays unencrypted.

This functionality requires the OpenStack Barbican service to be present. Without the Barbican service, the possibilities to encrypt Workloads will not be shown inside Horizon.

T4O 4.2 is only consuming secrets from Barbican. It is not creating, editing, or deleting any secret inside Barbican.

To utilize encrypted Workloads the Trilio trustee role needs to be able to engage with the Barbican service to read and fetch secrets from Barbican. The only default roles enabled with these permissions are the admin and the creator roles.

The possibility to encrypt a Workload is only provided during Workload creation. Once a workload has been created it is no longer possible to change, whether the Workload is encrypted or not.

Every encrypted Workload is consuming a unique Barbican secret. It is not possible to assign the same secret to two Workloads.

The following secret configurations are supported.

A default Barbican installation will generate secrets of the following type:

• Alghorithm: AES-256

• Mode: cbc

• content type: application/octet-stream

• payload-content-encoding: base64

• secret type: opaque

• payload: plaintext

Encrypted Workloads can be migrated to different projects or clouds just like normal Workloads. The Barbican secrets used for the encryption need to be made available inside the target environment. This already applies when the Workload gets assigned to a different owner.

Support for Multi-IP NFS backends

Many Trilio customers are using software-defined storage solutions for a scalable backup target. These software-defined storage solutions often provide the capability to spread read and write operations over multiple nodes of the storage cluster. Each of these nodes has its own access IP to interact with. All nodes are still writing to the same logical volume, which is available through the NFS protocol.

Trilio for Openstack 4.2 is supporting such solutions by enabling the have different IPs for the same backup target for each Datamover.

Every Datamover and the Trilio appliance are still consuming one NFS path per Volume.

This is achieved by changing the method to calculate the T4O mount point. It is now only considering the volume path instead of the complete NFS path. Example below.

# echo -n /Trilio_Backup | base64
L1RyaWxpb19CYWNrdXA=
/var/triliovault-mounts/L1RyaWxpb19CYWNrdXA=/

Database cleanup utility

Trilio for OpenStack is following an older OpenStack database schema and model. This model contains, that no data inside the database ever gets truly deleted. Only a flag is set to showcase that the dataset is no longer considered active.

This has the advantage that it is always possible to trace back any objects and their existence timeline inside the database for analytical purposes.

The big disadvantage of this model is that the database itself is ever-growing with every activity that T4O is doing.

Over time it is possible for the database to reach sizes, that normal activities like listing Workloads are taking so long, that other tasks like taking backups are impacted.

To counter this issue does Trilio provide a new utility, which will delete all no longer required database entries to reduce the load of the work when the T4O solution needs to access the database.

Backup rebase utility

Trilio for OpenStack is providing full synthetic backups utilizing the backing-file functionality of qcow2 images.

This functionality enables Trilio backups to run incremental forever, without having to restore every incremental backup. Instead only the latest backup needs to be restored as all missing blocks are fetched through the backing file chain from older backups.

The backing file of a particular backup is hardcoded inside the qcow2 image itself. These backing files need to consist of the full path and can't use relative paths. An example is provided below.

qemu-img info 85b645c5-c1ea-4628-b5d8-1faea0e9d549
image: 85b645c5-c1ea-4628-b5d8-1faea0e9d549
file format: qcow2
virtual size: 1.0G (1073741824 bytes)
disk size: 21M
cluster_size: 65536
backing file: /var/triliovault-mounts/MTAuMTAuMi4yMDovdXBzdHJlYW0=/workload_3c2fbee5-ad90-4448-b009-5047bcffc2ea/snapshot_f4874ed7-fe85-4d7d-b22b-082a2e068010/vm_id_9894f013-77dd-4514-8e65-818f4ae91d1f/vm_res_id_9ae3a6e7-dffe-4424-badc-bc4de1a18b40_vda/a6289269-3e72-4085-adca-e228ba656984
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

T4O is using a base64 hash value inside this path for NFS backup targets. This backup target path needs to be resolvable for the qcow2 image to find its backing file.

The hash value of the mount path can change when the backup is moved to a different backup target or when the T4O calculation method is changed like it is for T4O 4.2.

Trilio is now providing a utility that will change the backing file path for a given workload.

Known issues

[Canonical OpenStack] Selective restore fails for migrated workloads

Observation:

• after migrating workloads to a different Canonical OpenStack environments restore fails with permission denied error in tmp directory

Workaround:

Run the {{sudo sysctl fs.protected_regular=0}} command on wlm units.

[Intermittent Canonical OpenStack Queens] In-Place restore doesn't work with ext3/ext4 file systems

Observation:

• In-Place restore succeeds logically

• Data is not getting replaced

Workaround:

• Running the In-Place restore a second time

• Running a selective restore and reattaching the restored Volume

[intermittent] [Canonical OpenStack] Retention not honored after mount/unmount operation

Observation:

• After a mount or unmount operation the ownership of the backup stays qemu:qemu which leads to T4O no longer being able to run required merge commands

Workaround:

• identify the workload with failed retention policy

• run: chown -R nova:nova <WL_DIR>

• verify next backup applies the retention policy

CLI commands get-importworkload-list and get-orphaned-workloads-list show wrong list of workloads

Observation:

• independent of the command all workloads located on the backup target are listed

Workaround:

• use project id in command to show only workloads that can be imported for that project workloadmgr workload-get-importworkloads-list --project_id <project_id>

File-search not displaying files in lvm created logical volumes

Observation:

• File search returns an empty list for lvm controlled logical volumes

• fdisk created logical volumes work as desired

File-search not displaying files when root directory doesn't contain read permissions for group

Observation:

• File search returns an empty list when root doesn't have read permissions for groups

• File search is run as user nova

Unable to create encrypted Workload if T4O gets reconfigured with creator trustee role

Observation:

• T4O initially configured with trustee role member (not able to create encrypted workloads)

• After reconfiguration to trustee role creator encrypted workloads are still not creatable

Workaround:

• After reconfiguration create one unencrypted workload

• Then create encrypted workloads as required

Post restore of encrypted incremental snapshots CentOS instance is not getting booted

Observation:

• Restoring of a CentOS-based instance fails to start the instance in the case of restoring from an encrypted incremental snapshot

Workaround:

• Use only full backups for CentOS-based instances in combination with encrypted Workloads

Release Versions

Packages

Deliverables against T4O-4.2.8

Following packages changed/added in the current release

Containers and Gitbranch

Changelog

• Support for Kolla Zed (Ubuntu & Rocky) OpenStack

• Verification of Jira issues targeted for 4.2.8 release

Fixed Bugs and issues

1. Backup failing on cinder backend Fiber Channel.

2. Single VM backup (configurable) at a time out of workload having multiple VMs.

3. Snapshot and Selective/One Click Restore failing on quobyte.

4. Inplace restore fails saying no space left on device.

5. Specific workload fails to be restored; says VMDiskResourceSnap could not be found.

6. Restores are failing; error says security group rules already exist.

7. Snapshot intermittently fails with IO error while backup uploading.

8. Stopping multipath volumes is taking huge time.

9. Upgrading or deploying RHOSP 16.2.5 results in TrilioVaultWLM service and respective endpoints disappearing.

Known issues

Deleting snapshots show status as available in horizon UI

Observation : Snapshot for which delete operation is in-progress from UI , its status is showing as available instead deleting.

Workaround:

Wait for sometime to complete all the delete operations.Eventually all the snapshots will be deleted successfully.

It is recommended to think about the following elements prior to the installation of Trilio for Openstack.

Tenant Quotas

Trilio uses Cinder snapshots for calculating full and incremental backups. For full backups, Trilio creates Cinder snapshots for all the volumes in the backup job. It then leaves these Cinder snapshots behind for calculating the incremental backup image during next backup. During an incremental backup operation it creates new Cinder snapshots, calculates the changed blocks between the new snapshots and the old snapshots that were left behind during full/previous backups. It then deletes the old snapshots but leaves the newly created snapshots behind. So, it is important that each tenant who is availing Trilio backup functionality has sufficient Cinder snapshot quotas to accommodate these additional snapshots. The guideline is to add 2 snapshots for every volume that is added to backups to volume snapshot quotas for that tenant. You may also increase the volume quotas for the tenant by the same amount because Trilio briefly creates a volume from snapshot to read data from the snapshot for backup purposes. During a restore process, Trilio creates additional instances and Cinder volumes. To accommodate restore operations, a tenant should have sufficient quota for Nova instances and Cinder volumes. Otherwise restore operations will result in failures.

AWS S3 eventual consistency¶

AWS S3 object consistency model includes:

1. Read-after-write

2. Read-after-update

3. Read-after-delete

Each of them describes how an object will reach its consistent state after an object is created/updated or deleted. None of them provides strong consistency and there is a lag time for an object to reach the consistent state. Though Trilio employed mechanisms to work around the limitations of eventual consistency of AWS S3, when an object reach its consistency state is not deterministic. There is no official statement from AWS on how long it takes for an object to reach consistent state. However read-after-write has a shorter time to reach consistency compared to other IO patterns. Our solution is designed to maximize read-after-write IO pattern. The time in which an object reaches eventual consistency also depends on the AWS region. For example, aws-standard region does not have strong consistency model compared to us-east or us-west. We suggest to use these regions when creating s3 buckets for Trilio. Though read-after-update IO pattern is hard to avoid completely, we employed ample delays in accessing objects to accommodate larger durations for objects to get into consistent state. However in rare occasions, backups may still fail and need to restarted.

Trilio Cluster¶

Trilio can be deployed as a single node or a three node cluster. It is highly recommended that Trilio is deployed as three node cluster for fault tolerance and load balancing. Starting with 3.0 release, Trilio requires additional IP for cluster and is required for both single node and three node deployments. Cluster ip a.k.a virtual ip is used for managing cluster and is used to register Trilio service endpoint in the keystone sevice catalog.

Once the Trilio VM or the Cluster of Trilio VMs has been spun, the actual installation process can begin. This process contains of the following steps:

1. Install the Trilio dm-api service on the control plane

2. Install the Trilio datamover service on the compute plane

3. Install the Trilio Horizon plugin into the Horizon service

How these steps look in detail is depending on the Openstack distribution Trilio is installed in. Each supported Openstack distribution has its own deployment tools. Trilio is integrating into these deployment tools to provide a native integration from the beginning to the end.

• Test text

Trilio, by TrilioData, is a native OpenStack service that provides policy-based comprehensive backup and recovery for OpenStack workloads. The solution captures point-in-time workloads (Application, OS, Compute, Network, Configurations, Data and Metadata of an environment) as full or incremental snapshots. These snapshots can be held in a variety of storage environments including NFS AWS S3 compatible storage. With Trilio and its single click recovery, organizations can improve Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO). With Trilio, IT departments are enabled to fully deploy OpenStack solutions and provide business assurance through enhanced data retention, protection and integrity.

With the use of Trilio’s VAST (Virtual Snapshot Technology), Enterprise IT and Cloud Service Providers can now deploy backup and disaster recovery as a service to prevent data loss or data corruption through point-in-time snapshots and seamless one-click recovery. Trilio takes point-in-time backup of the entire workload consisting of compute resources, network configurations and storage data as one unit. It also takes incremental backups that only captures the changes that were made since last backup. Incremental snapshots save time and storage space as the backup only includes changes since the last backup. The benefits of using VAST for backup and restore could be summarized as below:

1. Efficient capture and storage of snapshots. Since our full backups only include data that is committed to storage volume and the incremental backups only include changed blocks of data since last backup, our backup processes are efficient and storages backup images efficiently on the backup media

2. Faster and reliable recovery. When your applications become complex that snap multiple VMs and storage volumes, our efficient recovery process will bring your application from zero to operational with just click of button

3. Easy migration of workloads between clouds. Trilio captures all the details of your application and hence our migration includes your entire application stack without leaving any thing for guess work.

4. Through policy and automation lower the Total Cost of Ownership. Our tenant driven backup process and automation eliminates the need for dedicated backup administrators, there by improves your total cost of ownership.

The Trilio Appliance is delivered as qcow2 image and runs as VM on top of a KVM Hypervisor.

Creating the cloud-init image

The Trilio appliance is utilizing cloud-init to provide the initial network and user configuration.

Cloud-init is reading it's information either from a metadata server or from a provided cd image. Trilio is utilizing the cd image.

Needed tools

To create the cloud-init image it is required to have genisoimage available.

#For RHEL and centos
yum install genisoimage
#For Ubuntu 
apt-get install genisoimage

Providing the Metadata

Cloud-init is using two files for it's metadata.

The first file is called meta-data and contains the information about the network configuration. Below is an example of this file.

[root@kvm]# cat meta-data
instance-id: triliovault
network-interfaces: |
   auto eth0
   iface eth0 inet static
   address 158.69.170.20
   netmask 255.255.255.0
   gateway 158.69.170.30

   dns-nameservers 11.11.0.51
local-hostname: localhost

The second file is called user-data and contains little scripts and information to set up for example the user passwords. Below is an example of this file.

[root@kvm]# cat user-data
#cloud-config
chpasswd:
  list: |
    root:password1
    stack:password2
  expire: False

creating the image file

The image is getting created using genisoimage follwing this general command:

genisoimage -output <name>.iso -volid cidata -joliet -rock </path/user-data> </path/meta-data>

An example of this command is shown below.

genisoimage  -output tvault-firstboot-config.iso -volid cidata -joliet -rock user-data meta-data

Spining up the Trilio appliance

After the cloud-init image has been created the TriloVault appliance can be spun up on the desired KVM server.

Extract the Trilio QCOW2 tar file using the following command :

tar Jxvf TrilioVault_file.tar.xz

See below an example command, how to spin up the Trilio appliance using virsh and the created iso image.

virt-install -n triliovault-vm  --memory 24576 --vcpus 8 \
--os-type linux \ 
--disk tvault-appliance-os-3.0.154.qcow2,device=disk,bus=virtio,size=40 \
--network bridge=virbr0,model=virtio \
--network bridge=virbr1,model=virtio \
--graphics none \
--import \
--disk path=tvault-firstboot-config.iso,device=cdrom

Uninstalling cloud-init after first boot

Once the Trilio appliance is up and running with it's initial configuration is it recommended to uninstall cloud-init.

To uninstall cloud-init, follow the example below.

sudo yum remove cloud-init

[RHOSP, TripleO and Kolla only] Verify nova UID/GID for nova user on the Appliance

Red Hat OpenStack, TripleO, and Kolla Ansible Openstack are using the nova UID/GID of 42436 inside their containers instead of 162:162 which is the standard in other Openstack environments.

Please verify that the nova UID/GID on the Trilio Appliance is still 42436,

[root@TVM1 ~]# id nova
uid=42436(nova) gid=42436(nova) groups=42436(nova),990(libvirt),36(kvm)

In case of the UID/GID being changed back to 162:162 follow these steps to set it back to 42436:42436.

1. Download the shell script that will change the user id

2. Assign executable permissions

3. Execute the script

4. Verify that the nova user and group id have changed to '42436'

## Download the shell script
$ curl -O https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/master/common/nova_userid.sh

## Assign executable permissions
$ chmod +x nova_userid.sh

## Execute the shell script to change 'nova' user and group id to '42436'
$ ./nova_userid.sh

## Ignore any errors and verify that 'nova' user and group id has changed to '42436'
$ id nova
   uid=42436(nova) gid=42436(nova) groups=42436(nova),990(libvirt),36(kvm)

Updating the appliance to the latest minor version

It is recommended to directly update the Trilio appliance to the latest version.

To do so follow the minor update guide provided here:

Backup-as-a-Service

Trilio is an add on service to OpenStack cloud infrastructure and provides backup and disaster recovery functions for tenant workloads. Trilio is very similar to other OpenStack services including nova, cinder, glance, etc and adheres to all tenets of OpenStack. It is a stateless service that scales with your cloud.

Main Components

Trilio has four main software components:

1. Trilio ships as a QCOW2 image. User can instantiate one or more VMs from the QCOW2 image on a standalone KVM boxes.

2. Trilio API is a python module that is installed on all OpenStack controller nodes where the nova-api service is running.

3. Trilio Datamover is a python module that is installed on every OpenStack compute nodes

4. Trilio horizon plugin is installed as an add on to horizon servers. This module is installed on every server that runs horizon service.

Service Endpoints

Trilio is both a provider and consumer into OpenStack ecosystem. It uses other OpenStack services such as nova, cinder, glance, neutron, and keystone and provides its own service to OpenStack tenants. To accomodate all possible OpenStack deployments, Trilio can be configured to use either public or internal URLs of services. Likewise Trilio provides its own public, internal and admin URLs.

Network Topology

This figure represents a typical network topology. Trilio exposes its public URL endpoint on public network and Trilio virtual appliances and data movers typically use either internal network or dedicated backup network for storing and retrieving backup images from backup store.

Release Versions

Packages

Deliverables against TVO-4.2.6

Following packages changed/added in the current release

Containers and Gitbranch

Changelog

• Verification of Jira issues targeted for 4.2.6 release

• Cohesity NFS/S3 storage backend support

Fixed Bugs and issues

1. Mounting a snapshot fails if the first disk of the File Recovery Manager VM is a CDROM

2. TVM HA configuration fails when one of the chosen Controller's hostname is part of any other TVM's hostname

3. Validation of keystone and s3 endpoint is stuck on the TVM UI

4. Temp volume creation taking too long

5. The httpd service is in a failed state on a freshly deployed T4O

6. The sshd option UseDNS should be set to "no" to avoid issues

Known issues

[RHOSP 16.1.8 and RHOSP 16.2.4] Trilio Horizon container in reboot loop

Observation : Post upgrade from a previous release/hotfix or on fresh deployment, the Trilio Horizon container is in a reboot loop

Workaround:

Either of the below workarounds should be performed on the controller where the issue occurs for the horizon pod.

option-1: Restart the memcached service on controller using systemctl (command: systemctl restart tripleo_memcached.service).

option-2: Restart the memcached pod (command: podman restart memcached).

[Snapshot mount] Unable to see correct content after mounting snapshot

Observation : Snapshot mount using RHEL8 File recovery manager image is not showing any mounted device

Workaround:

Use RHEL7 image for File recovery manager vm instead RHEL8.

[Backup failure for HPE nimble storage volumes with timeout receiving packet error

Observation : Snapshot is failing for VM's that have volumes backed by a Nimble iSCSI SAN with a "timeout receiving packet" error

Workaround: add uxsock_timeout parameter

log into respective datamover container and add uxsock_timeout with value as 60000 (i.e. 60 sec) in /etc/multipath.conf. Restart datamover container

[Input/Output error while writing to Cohesity NFS share]

Observation : Input/Output error during qemu-img convert operation while writing to Cohesity NFS share

Workaround: For cohesity NFS use below nfs options during Trilio configuration and datamover deployment and if issue still persists then increase timeo and retrans parameter values in nfs options

nfs_options = nolock,soft,timeo=600,intr,lookupcache=none,nfsvers=3,retrans=10

Release Versions

Packages

Deliverables against T4O-4.2.7

Following packages changed/added in the current release

Containers and Gitbranch

Changelog

• Support for RHOSP17

• Support for Canonical zed Openstack

• Support for physical delete from DB against all delete operations

• Verification of Jira issues targeted for 4.2.7 release

Fixed Bugs and issues

1. workloadmgr trust-create command is failing with error - There are multiple role entities named 'admin'

2. Selective restore failing: Invalid volume

3. Backport DB cleanup code from 5.0 to 4.2

Known issues

Deleting snapshots show status as available in horizon UI

Observation : Snapshot for which delete operation is in-progress from UI , its status is showing as available instead deleting.

Workaround:

Wait for sometime to complete all the delete operations.Eventually all the snapshots will be deleted successfully.

Release Versions

Packages

Deliverables against TVO-4.2.HF4

Following packages changed/added in the current release

Juju charms for Openstack Yoga release

Containers and Gitbranch

Changelog

New Qualifications

This Hotfix extends the Support Matrix of T4O 4.2 as follows:

• Fresh 4.2HF4 Trilio appliance build with wlm services on Python3.8 and multiple vulnerability fixes

• Kolla Ansible Openstack Yoga CentOS Stream 8, Ubuntu 20.04

• Canonical Yoga Ubuntu 20.04, Ubuntu 22.04

Fixed Bugs and issues

1. Irods nfs T4O not compatible

2. Backups failing with "Unable to call vast_instance"

3. Backup/restore does not work when NFS has access to Nova user just NOVA

4. Failed ansible-tvault-contego-extension : create trilio.filters for mount and unmount task

5. lxc packages not installed when using bare metal install

6. dmapi_all also includes bare metal hosts on non LXC deployments

7. Config failed with "iptables: Nothing to save" when TVM utilizing IPv6 | Tmobile/Red Hat | SFDC#2881

Trilio for OpenStack Compatibility Matrix

NFS & S3 Support:

All versions of Trilio for OpenStack support NFSv3 and S3 as backup targets.

Barbican Support:

Supported in RHOSP 16.1, 16.2, 17.0, Canonical Ussuri, Victoria, Wallaby, Yoga, Zed, Kolla Victoria, Wallaby, Yoga.

Not Supported in RHOSP 13, Canonical Queens, Stein, Train, TripleO Train.

Supported OS:

RHEL7, RHEL8, RHEL9, Ubuntu 18.04, Ubuntu 20.04, Ubuntu 20.04 source image, Ubuntu 22.04, CentOS7, CentOS Stream 8, CentOS Linux 8, Ubuntu 20.04 binary image, CentOS Stream 8 binary image, CentOS Stream 8 source image.

Deployment:

Deploy using the distributions corresponding deployment toolsets, including Red Hat Director, JuJu Charms, Ansible, and Director.

Compatibility Matrix Detailed View

Trilio has four main software components:

1. Trilio ships as a QCOW2 image. User can instantiate one or more VMs from the QCOW2 image on a standalone KVM boxes.

2. Trilio API is a python module that is an extension to nova api service. This module is installed on all OpenStack controller nodes

3. Trilio Datamover is a python module that is installed on every OpenStack compute nodes

4. Trilio horizon plugin is installed as an add on to horizon servers. This module is installed on every server that runs horizon service.

System requirements Trilio Appliance

The Trilio Appliance gets delivered as a qcow2 image, which gets attached to a virtual machine.

Trilio supports KVM-based hypervisors on x86 architectures, with the following properties:

The recommended size of the VM for the Trilio Appliance is:

The qcow2 image itself defines the 40GB disk size of the VM.

Software Requirements

In addition to the Trilio Appliance does Trilio contain components, which are installed directly into the Openstack itself.

Additionally, it is necessary to have the nfs-common packages installed on the compute nodes in case of using the NFS protocol for the backup target.

OpenStack Barbican for encryption

Since Trilio for OpenStack 4.2 is T4O capable of providing encrypted backups

The Trilio for OpenStack solution is leveraging the OpenStack Barbican service to provide encryption capabilities for its backups.

To be precise, T4O is using secrets provided by Barbican to encrypt and decrypt backups. Barbican is therefore required to utilize this feature.

Trilio integrates natively with Openstack. This includes that Trilio communicates completely through APIs using the Openstack Endpoints. Trilio is also generating its own Openstack endpoints. In addition, is the Trilio appliance and the compute nodes writing to and reading from the backup target. These points affect the network planning for the Trilio installation.

Existing endpoints in Openstack

Openstack knows 3 types of endpoints:

• Public Endpoints

• Internal Endpoints

• Admin Endpoints

Each of these endpoint types is designed for a specific purpose. Public endpoints are meant to be used by the Openstack end-users to work with Openstack. Internal endpoints are meant to be used by the Openstack services to communicate with each other. Admin endpoints are meant to be used by Openstack administrators.

Out of those 3 endpoint types does only the admin endpoint sometimes contain APIs which are not available on any other endpoint type.

To learn more about Openstack endpoints please visit the official Openstack documentation.

Openstack endpoints required by Trilio

Trilio is communicating with all services of Openstack on a defined endpoint type. Which endpoint type Trilio is using to communicate with Openstack is decided during the configuration of the Trilio appliance.

The following network requirement can be identified this way:

• Trilio appliance needs access to the Keystone admin endpoint on the admin endpoint network

• Trilio appliance needs access to all endpoints of one type

Recommendation: Provide access to all Openstack Endpoint types

Trilio is recommending providing full access to all Openstack endpoints to the Trilio appliance to follow the Openstack standards and best practices.

Trilio is generating its own endpoints as well. These endpoints are pointing towards the Trilio Appliance directly. This means that using those endpoints will not send the API calls towards the Openstack Controller nodes first, but directly to the Trilio VM.

Following the Openstack standards and best practices, it is therefore recommended to put the Trilio endpoints on the same networks as the already existing Openstack endpoints. This allows to extend the purpose of each endpoint type to the Trilio service:

• The public endpoint to be used by Openstack users when using Trilio CLI or API

• The internal endpoint to communicate with the Openstack services

• The admin endpoint to use the required admin only APIs of Keystone

Backup target access required by Trilio

The Trilio solution is using backup target storage to securely place the backup data. Trilio is dividing its backup data into two parts:

1. Metadata

2. Volume Disk Data

The first type of data is generated by the Trilio appliance through communicating with the Openstack Endpoints. All metadata that is stored together with a backup is written by the Trilio Appliance to the backup target in the json format.

The second type of data is generated by the Trilio Datamover service running on the compute nodes. The Datamover service is reading the Volume Data from the Cinder or Nova storage and transferring this data as qcow2 image to the backup target. Each Datamover service is hereby responsible for the VMs running on its compute node.

The network requirements are therefor:

• The Trilio appliance needs access to the backup target

• Every compute node needs access to the backup target

Example of a typical Trilio network integration

Most Trilio customers are following the Openstack standards and best practices to have the public, internal, and admin endpoints on separate networks. They also typically don't have any network yet, which can access the desired backup target.

The starting network configuration typically looks like this:

Following the Openstack standards and Trilio's recommendation will the Trilio Appliance be placed on all those 3 networks. Further is the access to the backup target required by Trilio Appliance and Compute nodes. Here done by adding a 4th network.

The resulting network configuration would look like this:

It is of course possible to combine networks as necessary. As long as the required network access is available will Trilio work.

Other examples of Trilio network integrations

Each Openstack installation is different and so is the network configuration. There are endless possibilities of how to configure the Openstack network and how to implement the Trilio appliance into this network. The following three examples have been seen in production:

The first example is from a manufacturing company, which wanted to split the networks by function and decided to put the Trilio backup target on the internal network as the backup and recovery function was identified as an Openstack internal solution. This example looks complex but integrates Trilio just as recommended.

The second example is from a financial institute that wanted to be sure that the Openstack Users have no direct uncontrolled network access to the Openstack infrastructure. Following this example requires additional work as the internal HA-Proxy needs to be configured to correctly translates the API calls towards the Trilio

The third example is from a service company that was forced to treat Trilio as an external 3rd party solution, as we require a virtual machine running outside of Openstack. This kind of network configuration requires good planning on the Trilio endpoints and firewall rules.

Ceph is the most common OpenSource solution to provide block storage through OpenStack Cinder.

Ceph is a very flexible solution. The possibilities of Ceph require additional steps to the Trilio solution.

It is possible to configure Cinder to have multiple configurations and keyrings for CEPH.

In this case, the Trilio Datamover file needs to be extended with the CEPH information.

For Trilio to be able to work in such an environment it is required to put copies of each of these configurations and keyrings into a separate directory, which is then made known to the Trilio Datamover inside a [ceph] block in the tvault-contego.conf.

A tvault-contego.conf file with the extended [ceph] block would look like this.

[DEFAULT]

vault_storage_type = nfs
vault_storage_nfs_export = 192.168.1.34:/mnt/tvault/tvm5
vault_storage_nfs_options = nolock,soft,timeo=180,intr,lookupcache=none


vault_data_directory_old = /var/triliovault
vault_data_directory = /var/trilio/triliovault-mounts
log_file = /var/log/kolla/triliovault-datamover/tvault-contego.log
debug = False
verbose = True
max_uploads_pending = 3
max_commit_pending = 3

dmapi_transport_url = rabbit://openstack:BtsxtLmLM9EvMBduewZOX23rNusiqZqCbhihGIib@192.168.0.51:5672,openstack:BtsxtLmLM9EvMBduewZOX23rNusiqZqCbhihGIib@192.168.0.52:5672,openstack:BtsxtLmLM9EvMBduewZOX23rNusiqZqCbhihGIib@192.168.0.53:5672//

[dmapi_database]
connection = mysql+pymysql://dmapi:x5nvYXnAn4rXmCHfWTK8h3wwShA4vxMq3gE2jH57@kolla-victoriaR-internal.triliodata.demo:3306/dmapi


[libvirt]
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = volumes

[ceph]
keyring_ext = .volumes.keyring
ceph_dir = /etc/ceph/directory1/,/etc/ceph/directory2/

[contego_sys_admin]
helper_command = sudo /usr/bin/privsep-helper


[conductor]
use_local = True

[oslo_messaging_rabbit]
ssl = false

[cinder]
http_retries = 10

The uninstallation of Trilio is depending on the Openstack Distribution it is installed in. The high-level process is the same for all Distributions.

1. Uninstall the Horizon Plugin or the Trilio Horizon container

2. Uninstall the datamover-api container

3. Uninstall the datamover

4. Delete the Trilio Appliance Cluster

1. Prepare for deployment

1.1] Select 'backup target' type

Backup target storage is used to store backup images taken by Trilio and details needed for configuration:

The following backup target types are supported by Trilio

a) NFS

Need NFS share path

b) Amazon S3

- S3 Access Key - Secret Key - Region - Bucket name

c) Other S3 compatible storage (Like, Ceph based S3)

- S3 Access Key - Secret Key - Region - Endpoint URL (Valid for S3 other than Amazon S3) - Bucket name

1.2] Clone triliovault-cfg-scripts repository

The following steps are to be done on 'undercloud' node on an already installed RHOSP environment. The overcloud-deploy command has to be run successfully already and the overcloud should be available.

The following command clones the triliovault-cfg-scripts github repository.

cd /home/stack
git clone -b TVO/4.2.8 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/

1.3] If the backup target type is 'Ceph based S3' with SSL:

If your backup target is ceph S3 with SSL and SSL certificates are self-signed or authorized by a private CA, then the user needs to provide a CA chain certificate to validate the SSL requests. For that, the user needs to rename his ca chain cert file to s3-cert.pem and copy it into the directory triliovault-cfg-scripts/redhat-director-scripts/redhat-director-scripts/<RHOSP_RELEASE_Directory/puppet/trilio/files

cp s3-cert.pem /home/stack/triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/puppet/trilio/files/

2] Upload Trilio puppet module

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/scripts/
chmod +x *.sh
./upload_puppet_module.sh

## Output of the above command looks like the following.
Creating tarball...
Tarball created.
Creating heat environment file: /home/stack/.tripleo/environments/puppet-modules-url.yaml
Uploading file to swift: /tmp/puppet-modules-8Qjya2X/puppet-modules.tar.gz
+-----------------------+---------------------+----------------------------------+
| object                | container           | etag                             |
+-----------------------+---------------------+----------------------------------+
| puppet-modules.tar.gz | overcloud-artifacts | 368951f6a4d39cfe53b5781797b133ad |
+-----------------------+---------------------+----------------------------------+

## Above command creates the following file.
ls -ll /home/stack/.tripleo/environments/puppet-modules-url.yaml

3] Update overcloud roles data file to include Trilio services

Trilio contains multiple services. Add these services to your roles_data.yaml.

Add the following services to the roles_data.yaml

3.1] Add Trilio Datamover Api Service to role data file

This service needs to share the same role as the keystone and database service. In the case of the pre-defined roles will these services run on the role Controller. In the case of custom-defined roles, it is necessary to use the same role where OS::TripleO::Services::Keystone service is installed.

Add the following line to the identified role:

'OS::TripleO::Services::TrilioDatamoverApi'

3.2] Add Trilio Datamover Service to role data file

This service needs to share the same role as the nova-compute service. In the case of the pre-defined roles will the nova-compute service run on the role Compute. In the case of custom-defined roles, it is necessary to use the role the nova-compute service is used.

Add the following line to the identified role:

'OS::TripleO::Services::TrilioDatamover'

3.3] Add Trilio Horizon Service role data file4] Prepare Trilio container images

This service needs to share the same role as the OpenStack Horizon server. In the case of the pre-defined roles, Horizon service runs on the role Controller. Add the following line to the identified role:

'OS::TripleO::Services::TrilioHorizon'

Trilio containers are pushed to 'Dockerhub'. Registry URL: 'docker.io'. Container pull URLs are given below.

CentOS7

Trilio Datamove container:        docker.io/trilio/tripleo-train-centos7-trilio-datamover:<HOTFIX-TAG-VERSION>-tripleo
Trilio Datamover Api Container:   docker.io/trilio/tripleo-train-centos7-trilio-datamover-api:<HOTFIX-TAG-VERSION>-tripleo
Trilio horizon plugin:            docker.io/trilio/tripleo-train-centos7-trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-tripleo

There are two registry methods available in TripleO Openstack Platform.

1. Remote Registry

2. Local Registry

4.1] Remote Registry

Follow this section when 'Remote Registry' is used.

For this method, it is not necessary to pull the containers in advance. It is only necessary to populate the trilio_env.yaml file with the Trilio container URLs from the Dockerhub registry.

Populate the trilio_env.yaml with container URLs for:

• Trilio Datamover container

• Trilio Datamover api container

• Trilio Horizon Plugin

# For TripleO Train Centos7
$ grep 'Image' trilio_env.yaml
   DockerTrilioDatamoverImage: docker.io/trilio/tripleo-train-centos7-trilio-datamover:<HOTFIX-TAG-VERSION>-tripleo
   DockerTrilioDmApiImage: docker.io/trilio/tripleo-train-centos7-trilio-datamover-api:<HOTFIX-TAG-VERSION>-tripleo
   DockerHorizonImage: docker.io/trilio/tripleo-train-centos7-trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-tripleo

4.2] Local Registry

Follow this section when 'local registry' is used on the undercloud.

Run the following script. Script pulls the triliovault containers and updates the triliovault environment file with URLs.

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/scripts/

sudo ./prepare_trilio_images.sh <undercloud_registry_hostname_or_ip> <OS_platform> <4.1-TRIPLEO-CONTAINER> <container_tool_available_on_undercloud>

Options OS_platform: [centos7]
Options container_tool_available_on_undercloud: [docker, podman]

## To get undercloud registry hostname/ip, we have two approaches. Use either one.
1. openstack tripleo container image list

2. find your 'containers-prepare-parameter.yaml' (from overcloud deploy command) and search for 'push_destination'
cat /home/stack/containers-prepare-parameter.yaml | grep push_destination
 - push_destination: "undercloud.ctlplane.ooo.prod1:8787"

Here, 'undercloud.ctlplane.ooo.prod1' is undercloud registry hostname. Use it in our command like following example.

# Command Example:
sudo ./prepare_trilio_images.sh undercloud.ctlplane.ooo.prod1 centos7 <HOTFIX-TAG-VERSION>-tripleo podman

## Verify changes
# For TripleO Train Centos7
$ grep '<HOTFIX-TAG-VERSION>-tripleo' ../environments/trilio_env.yaml
   DockerTrilioDatamoverImage: prod1-undercloud.demo:8787/trilio/tripleo-train-centos7-trilio-datamover:<HOTFIX-TAG-VERSION>-tripleo
   DockerTrilioDmApiImage: prod1-undercloud.demo:8787/trilio/tripleo-train-centos7-trilio-datamover-api:<HOTFIX-TAG-VERSION>-tripleo
   DockerHorizonImage: prod1-undercloud.demo:8787/trilio/tripleo-train-centos7-trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-tripleo

The changes can be verified using the following commands.

## For Centos7 Train

(undercloud) [stack@undercloud redhat-director-scripts/tripleo-train]$ openstack tripleo container image list | grep trilio
| docker://undercloud.ctlplane.localdomain:8787/trilio/tripleo-train-centos7-trilio-datamover:<HOTFIX-TAG-VERSION>-tripleo                       |                        |
| docker://undercloud.ctlplane.localdomain:8787/trilio/tripleo-train-centos7-trilio-datamover-api:<HOTFIX-TAG-VERSION>-tripleo                  |                   |
| docker://undercloud.ctlplane.localdomain:8787/trilio/tripleo-train-centos7-trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-tripleo                  |

5] Configure multi-IP NFS

On Undercloud node, change the directory

cd triliovault-cfg-scripts/common/

Edit file 'triliovault_nfs_map_input.yml' in the current directory and provide compute host and NFS share/IP map.

Get the compute hostnames from the following command. Check the 'Name' column. Use exact hostnames in 'triliovault_nfs_map_input.yml' file.

(undercloud) [stack@ucqa161 ~]$ openstack server list
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+
| ID                                   | Name                          | Status | Networks             | Image          | Flavor  |
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+
| 8c3d04ae-fcdd-431c-afa6-9a50f3cb2c0d | overcloudtrain1-controller-2  | ACTIVE | ctlplane=172.30.5.18 | overcloud-full | control |
| 103dfd3e-d073-4123-9223-b8cf8c7398fe | overcloudtrain1-controller-0  | ACTIVE | ctlplane=172.30.5.11 | overcloud-full | control |
| a3541849-2e9b-4aa0-9fa9-91e7d24f0149 | overcloudtrain1-controller-1  | ACTIVE | ctlplane=172.30.5.25 | overcloud-full | control |
| 74a9f530-0c7b-49c4-9a1f-87e7eeda91c0 | overcloudtrain1-novacompute-0 | ACTIVE | ctlplane=172.30.5.30 | overcloud-full | compute |
| c1664ac3-7d9c-4a36-b375-0e4ee19e93e4 | overcloudtrain1-novacompute-1 | ACTIVE | ctlplane=172.30.5.15 | overcloud-full | compute |
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+

vi triliovault_nfs_map_input.yml

Update pyYAML on the undercloud node only

## On Python3 env
sudo pip3 install PyYAML==5.1

## On Python2 env
sudo pip install PyYAML==5.1

Expand the map file to create a one-to-one mapping of the compute nodes and the NFS shares.

## On Python3 env
python3 ./generate_nfs_map.py

## On Python2 env
python ./generate_nfs_map.py

The result will be in file - 'triliovault_nfs_map_output.yml'

Validate output map file

Open file 'triliovault_nfs_map_output.yml

vi triliovault_nfs_map_output.yml

available in the current directory and validate that all compute nodes are covered with all the necessary NFS shares.

Append this output map file to 'triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_nfs_map.yaml'

grep ':.*:' triliovault_nfs_map_output.yml >> ../redhat-director-scripts/tripleo-train/environments/trilio_nfs_map.yaml

Validate the changes in file triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_nfs_map.yaml

Include the environment file (trilio_nfs_map.yaml) in overcloud deploy command with '-e' option as shown below.

-e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/environments/trilio_nfs_map.yaml

Ensure that MultiIPNfsEnabled is set to true in trilio_env.yaml file and that NFS is used as the backup target.

6] Fill in Trilio environment details

Fill Trilio details in the file /home/stack/triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/environments/trilio_env.yaml, triliovault environment file is self-explanatory. Fill in details of the backup target, verify image URLs, and other details.

7] Install Trilio on Overcloud

Use the following heat environment file and roles data file in overcloud deploy command

1. trilio_env.yaml: This environment file contains Trilio backup target details and Trilio container image locations

2. roles_data.yaml: This file contains overcloud roles data with Trilio roles added.

3. Use the correct trilio endpoint map file as per your keystone endpoint configuration. - Instead of tls-endpoints-public-dns.yaml this file, use ‘environments/trilio_env_tls_endpoints_public_dns.yaml’ - Instead of tls-endpoints-public-ip.yaml this file, use ‘environments/trilio_env_tls_endpoints_public_ip.yaml’ - Instead of tls-everywhere-endpoints-dns.yaml this file, use ‘environments/trilio_env_tls_everywhere_dns.yaml’

Deploy command with triliovault environment file looks like the following.

openstack overcloud deploy --templates \
  -e /home/stack/templates/node-info.yaml \
  -e /home/stack/templates/overcloud_images.yaml \
  -e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/environments/trilio_env.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-tls.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/inject-trust-anchor.yaml \
  -e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/environments/trilio_env_tls_endpoints_public_dns.yaml \
  --ntp-server 192.168.1.34 \
  --libvirt-type qemu \
  --log-file overcloud_deploy.log \
  -r /home/stack/templates/roles_data.yaml

8] Verify the deployment

8.1] On the Controller node

Make sure Trilio dmapi and horizon containers are in a running state and no other Trilio container is deployed on controller nodes. When the role for these containers is not "controller" check on respective nodes according to configured roles_data.yaml.

[root@overcloud-controller-0 heat-admin]# podman ps | grep trilio
26fcb9194566  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-tripleo        kolla_start           5 days ago  Up 5 days ago         trilio_dmapi
094971d0f5a9  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-tripleo       kolla_start           5 days ago  Up 5 days ago         horizon

Verify the haproxy configuration under:

/var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg

8.2] On Compute node

Make sure Trilio datamover container is in running state and no other Trilio container is deployed on compute nodes.

[root@overcloud-novacompute-0 heat-admin]# podman ps | grep trilio
b1840444cc59  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-tripleo                    kolla_start           5 days ago  Up 5 days ago         trilio_datamover

8.3] On the node with Horizon service

Make sure horizon container is in running state. Please note that 'Horizon' container is replaced with Trilio Horizon container. This container will have the latest OpenStack horizon + Trilio's horizon plugin.

[root@overcloud-controller-0 heat-admin]# podman ps | grep horizon
094971d0f5a9  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-tripleo       kolla_start           5 days ago  Up 5 days ago         horizon

10] Troubleshooting for overcloud deployment failures

Trilio components will be deployed using puppet scripts.

openstack stack failures list overcloud
heat stack-list --show-nested -f "status=FAILED"
heat resource-list --nested-depth 5 overcloud | grep FAILED

##=> If trilio datamover api containers does not start well or in restarting state, use following logs to debug.
docker logs trilio_dmapi
tailf /var/log/containers/trilio-datamover-api/dmapi.log

##=> If trilio datamover containers does not start well or in restarting state, use following logs to debug.
docker logs trilio_datamover
tailf /var/log/containers/trilio-datamover/tvault-contego.log

Trilio and Canonical have started a partnership to ensure a native deployment of Trilio using JuJu Charms.

Those JuJu Charms are publicly available as Open Source Charms.

The documentation of the charms can be found here:

Juju charms for OpenStack Yoga release onwards

Juju charms for other supported OpenStack releases upto wallaby

Prerequisite

Have a canonical OpenStack base setup deployed for a required release like Jammy Zed/Yoga, Focal yoga/Wallaby/Victoria/Ussuri, or Bionic Ussuri/Queens.

Steps to install the Trilio charms

1. Export the OpenStack base bundle

juju export-bundle --filename openstack_base_file.yaml

2. Create a Trilio overlay bundle as per the OpenStack setup release using the charms given above.

3. If file search functionality is required, provision any additional node(s) that will be required for deploying the Trilio Workload manager (trilio-wlm) as a VM instead of lxd container(s).

4. Commission the additional node from MAAS UI.

5. Do a dry run to check if the Trilio bundle is working

juju deploy --dry-run ./openstack_base_file.yaml --overlay <Trilio bundle path>

6. Do the deployment

juju deploy ./openstack_base_file.yaml --overlay <Trilio bundle path>

7. Wait till all the Trilio units are deployed successfully. Check the status via juju status command.

8. Once the deployment is complete, perform the below operations:

1. Create cloud admin trust

juju run-action --wait trilio-wlm/leader create-cloud-admin-trust password=<openstack admin password>

juju run --wait trilio-wlm/leader create-cloud-admin-trust password=<openstack admin password>

1. Add license

juju attach-resource trilio-wlm license=<Path to trilio license file>

juju run-action --wait trilio-wlm/leader create-license

juju run --wait trilio-wlm/leader create-license

Note: Reach out to the Trilio support team for the license file.

For multipath enabled environments, perform the following actions

1. log into each nova compute node

2. add uxsock_timeout with value as 60000 (i.e. 60 sec) in /etc/multipath.conf

3. restart tvault-contego service

Sample Trilio overlay bundles

Trilio is integrating natively into the RHOSP Director. Manual deployment methods are not supported for RHOSP.

1. Prepare for deployment

1.1] Select 'backup target' type

Backup target storage is used to store backup images taken by Trilio and details needed for configuration:

Following backup target types are supported by Trilio

a) NFS

Need NFS share path

b) Amazon S3

- S3 Access Key - Secret Key - Region - Bucket name

c) Other S3 compatible storage (Like, Ceph based S3)

- S3 Access Key - Secret Key - Region - Endpoint URL (Valid for S3 other than Amazon S3) - Bucket name

1.2] Clone triliovault-cfg-scripts repository

The following steps are to be done on 'undercloud' node on an already installed RHOSP environment. The overcloud-deploy command has to be run successfully already and the overcloud should be available.

The following command clones the triliovault-cfg-scripts github repository.

cd /home/stack
git clone -b TVO/4.2.8 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/

Next access the Red Hat Director scripts according to the used RHOSP version.

RHOSP 13

cd triliovault-cfg-scripts/redhat-director-scripts/rhosp13/

RHOSP 16.1

cd triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/

RHOSP 16.2

cd triliovault-cfg-scripts/redhat-director-scripts/rhosp16.2/

RHOSP 17.0

cd triliovault-cfg-scripts/redhat-director-scripts/rhosp17.0/

cd triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/

1.3] If backup target type is 'Ceph based S3' with SSL:

If your backup target is ceph S3 with SSL and SSL certificates are self signed or authorized by private CA, then user needs to provide CA chain certificate to validate the SSL requests. For that, user needs to rename his ca chain cert file to 's3-cert.pem' and copy it into directory - 'triliovault-cfg-scripts/redhat-director-scripts/redhat-director-scripts/<RHOSP_RELEASE___Directory/puppet/trilio/files'

cp s3-cert.pem /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/puppet/trilio/files

2] Upload trilio puppet module

The following commands upload the Trilio puppet module to the overcloud registry. The actual upload happens upon the next deployment.

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/scripts/
./upload_puppet_module.sh

## Output of above command looks like following for RHOSP13, RHOSP16.1 and RHOSP16.2
Creating tarball...
Tarball created.
Creating heat environment file: /home/stack/.tripleo/environments/puppet-modules-url.yaml
Uploading file to swift: /tmp/puppet-modules-8Qjya2X/puppet-modules.tar.gz
+-----------------------+---------------------+----------------------------------+
| object                | container           | etag                             |
+-----------------------+---------------------+----------------------------------+
| puppet-modules.tar.gz | overcloud-artifacts | 368951f6a4d39cfe53b5781797b133ad |
+-----------------------+---------------------+----------------------------------+

## Above command creates following file.
ls -ll /home/stack/.tripleo/environments/puppet-modules-url.yaml

## Command is same for RHOSP17.0 but the command output and file content would be different

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp17.0/scripts/
./upload_puppet_module.sh

## Output of above command looks like following for RHOSP17.0
Creating tarball...
Tarball created.
renamed '/tmp/puppet-modules-P3duCg9/puppet-modules.tar.gz' -> '/var/lib/tripleo/artifacts/overcloud-artifacts/puppet-modules.tar.gz'
Creating heat environment file: /home/stack/.tripleo/environments/puppet-modules-url.yaml

## Above command creates following file.
ls -ll /home/stack/.tripleo/environments/puppet-modules-url.yaml