1 of 93

T4O-4.3

About Trilio for OpenStack

Trilio, by TrilioData, is a native OpenStack service that provides policy-based comprehensive backup and recovery for OpenStack workloads. The solution captures point-in-time workloads (Application, OS, Compute, Network, Configurations, Data and Metadata of an environment) as full or incremental snapshots. These snapshots can be held in a variety of storage environments including NFS AWS S3 compatible storage. With Trilio and its single click recovery, organizations can improve Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO). With Trilio, IT departments are enabled to fully deploy OpenStack solutions and provide business assurance through enhanced data retention, protection and integrity.

With the use of Trilio’s VAST (Virtual Snapshot Technology), Enterprise IT and Cloud Service Providers can now deploy backup and disaster recovery as a service to prevent data loss or data corruption through point-in-time snapshots and seamless one-click recovery. Trilio takes point-in-time backup of the entire workload consisting of compute resources, network configurations and storage data as one unit. It also takes incremental backups that only captures the changes that were made since last backup. Incremental snapshots save time and storage space as the backup only includes changes since the last backup. The benefits of using VAST for backup and restore could be summarized as below:

Efficient capture and storage of snapshots. Since our full backups only include data that is committed to storage volume and the incremental backups only include changed blocks of data since last backup, our backup processes are efficient and storages backup images efficiently on the backup media
Faster and reliable recovery. When your applications become complex that snap multiple VMs and storage volumes, our efficient recovery process will bring your application from zero to operational with just click of button
Easy migration of workloads between clouds. Trilio captures all the details of your application and hence our migration includes your entire application stack without leaving any thing for guess work.
Through policy and automation lower the Total Cost of Ownership. Our tenant driven backup process and automation eliminates the need for dedicated backup administrators, there by improves your total cost of ownership.

Trilio for OpenStack Architecture

Backup-as-a-Service

Trilio is an add on service to OpenStack cloud infrastructure and provides backup and disaster recovery functions for tenant workloads. Trilio is very similar to other OpenStack services including nova, cinder, glance, etc and adheres to all tenets of OpenStack. It is a stateless service that scales with your cloud.

Main Components

Trilio has four main software components:

Trilio ships as a QCOW2 image. User can instantiate one or more VMs from the QCOW2 image on a standalone KVM boxes.
Trilio API is a python module that is installed on all OpenStack controller nodes where the nova-api service is running.
Trilio Datamover is a python module that is installed on every OpenStack compute nodes
Trilio horizon plugin is installed as an add on to horizon servers. This module is installed on every server that runs horizon service.

Service Endpoints

Trilio is both a provider and consumer into OpenStack ecosystem. It uses other OpenStack services such as nova, cinder, glance, neutron, and keystone and provides its own service to OpenStack tenants. To accomodate all possible OpenStack deployments, Trilio can be configured to use either public or internal URLs of services. Likewise Trilio provides its own public, internal and admin URLs.

Network Topology

This figure represents a typical network topology. Trilio exposes its public URL endpoint on public network and Trilio virtual appliances and data movers typically use either internal network or dedicated backup network for storing and retrieving backup images from backup store.

Trilio 4.3 Release Notes

T4O 4.3.0 (GA)

Release Versions

Packages

Deliverables against T4O-4.3.0

Containers and Gitbranch

Changelog

Improvement of workload import performance.
Issues reported by customers.

Fixed Bugs and issues

Trilio backup deleted after completion without manual intervention and backup not visible under workload.
S3 mountpoint gets stalled when the datamover container is stopped.
TVM uses outdated TLS protocols when HTTPS is activated for the Trilio API.

Known issues

1. Import of multiple workloads don't get segregated across all nodes evenly

Observation : When import of multiple workloads is triggered, it is expected that system must evenly divide the load on all available nodes in round robin fashion. However, current it is running the import on any one of the available nodes.

2. After clicking on snapshot or any restore against Imported WL, UI become unresponsive for first time only

Observation : For workloads with a large number of VMs and with a large number of networks attached to those VMs, then the import of the snapshot details may take more than 1 minute (connection timeout) and hence this issue might be observed.

Comments:

If a user hits this issue, then the import of the snapshot has already been triggered and now the user needs to wait till the snapshot import is done.
The users are advised to wait for a couple of minutes before they recheck the snapshot details.
Once the details of the snapshot are visible, the restore operation can be carried out.

3. Import of ALL workloads without specifying any workload id NOT recommended

Observation: If the user uses the workload import command without any workload id, thereby expecting that all eligible workloads will get imported, then the command execution takes longer than expected time.

Comments:

As long as the import command is not returning, it is expected to be running; if successful, it will return the job ID, if not, it will throw an error msg.
The execution time may vary based on the number of workloads present in the backend target.
It is recommended to run this command with specific workload IDs.
To import ALL workloads at once, all WL IDs to be provided as parameter to the import CLI command. Procedure for the same mentioned below.

4. Deleting snapshots show status as available in horizon UI

Observation: Snapshot may show as available instead of deleting when the deletion is in progress.

Workaround:

Wait until all the delete operations have completed. Eventually, all the snapshots will be deleted successfully.

T4O 4.3.1

Release Versions

Packages

Deliverables against T4O-4.3.1

Containers and Gitbranch

Changelog

Issues reported by customers.

Fixed Bugs and issues

Backup and Selective restore failure with quobyte cinder volume.
Trilio takes user time and not the dashboard time.
Documentation Change : Steps to be followed from Trilio end while minor upgrade from RHOSP 16.x to 16.y.
Canonical/Juju - python-os-brick_5.2.2-0ubuntu1.3 will disrupt Trilio snapshots completely, if using multipath.
Restores failing with Error : "security group rule does not exist".
When VMs of the workloads are deleted, snapshots fail (with a very misleading message).
"Timeout uploading data" when backing up a VM with a 23TB volume.

Known issues

1. Import of multiple workloads don't get segregated across all nodes evenly

2. After clicking on snapshot or any restore against Imported WL, UI become unresponsive for first time only

Comments:

If a user hits this issue, then the import of the snapshot has already been triggered and now the user needs to wait till the snapshot import is done.
The users are advised to wait for a couple of minutes before they recheck the snapshot details.
Once the details of the snapshot are visible, the restore operation can be carried out.

3. Import of ALL workloads without specifying any workload id NOT recommended

Observation : If user hits workload import command without any workload id, thereby expecting that all eligible workloads will get imported, then the command execution takes longer than expected time.

Comments:

As long as the import command is not returning, it is expected to be running; if successful, it will return the job ID, if not, it will throw an error msg.
The execution time may vary based on the number of workloads present in the backend target.
It is recommended to run this command with specific workload IDs.
To import ALL workloads at once, all WL IDs to be provided as parameter to the import CLI command. Procedure for the same mentioned below.

4. Deleting snapshots show status as available in horizon UI

Observation : Snapshot for which delete operation is in-progress from UI , its status is showing as available instead deleting.

Workaround:

Wait for sometime to complete all the delete operations.Eventually all the snapshots will be deleted successfully.

T4O 4.3.2

Release Versions

Packages

Deliverables against T4O-4.3.2

Containers and Gitbranch

Changelog

Issues reported by customers.

Fixed Bugs and issues

Snapshot listing (workload open) on UI doesn't happen if snapshot count is > 1000.
FileManager caches old snapshot details.
Custom horizon integration - Canonical.
Restore failing with error Failed restoring snapshot: RetryError[<Future at 0x7fc607693c18 state=finished raised StorageFailure>.
Duplicate security groups get created post restore of a VM.
kolla ansible yoga 4.3.0 deployment fails with error "Unsupported parameters for (kolla_container_facts) module: container_engine. Supported parameters include: name, api_version".

Known issues

1. Import of multiple workloads don't get segregated across all nodes evenly

2. After clicking on snapshot or any restore against Imported WL, UI become unresponsive for first time only

Comments:

If a user hits this issue, then the import of the snapshot has already been triggered and now the user needs to wait till the snapshot import is done.
The users are advised to wait for a couple of minutes before they recheck the snapshot details.
Once the details of the snapshot are visible, the restore operation can be carried out.

3. Import of ALL workloads without specifying any workload id NOT recommended

Comments:

As long as the import command is not returning, it is expected to be running; if successful, it will return the job ID, if not, it will throw an error msg.
The execution time may vary based on the number of workloads present in the backend target.
It is recommended to run this command with specific workload IDs.
To import ALL workloads at once, all WL IDs to be provided as parameter to the import CLI command. Procedure for the same mentioned below.

4. Deleting snapshots show status as available in horizon UI

Observation : Snapshot for which delete operation is in-progress from UI , its status is showing as available instead deleting.

Workaround:

Wait for sometime to complete all the delete operations.Eventually all the snapshots will be deleted successfully.

Deployment Guide

Compatibility Matrix

Learn about Trilio Support for OpenStack Distributions

The CentOS community has moved over to the CentOS stream.

CentOS7 is still supported and maintained till June 30th 2024

Kolla Ansible environments running on CentOS8 are receiving continuous limited support. This means that future updates from Trilio for Kolla Ansible environments on CentOS8 will use the latest available CentOS8 base containers and only the Trilio for OpenStack code gets updated. When the Kolla Ansible community provides CentOS Stream based containers, Trilio will provide CentOS Stream based containers as well.

Ansible Openstack is at its End Of Support from Trilio starting 2nd March 2023

Trilio for OpenStack Compatibility Matrix

NFS & S3 Support:

All versions of Trilio for OpenStack support NFSv3 and S3 as backup targets.

Barbican Support:

Supported in RHOSP 16.1, 16.2, 17.0, Canonical Ussuri, Victoria, Wallaby, Yoga, Zed, Kolla Victoria, Wallaby, Yoga.

Not Supported in RHOSP 13, Canonical Queens, Stein, Train, TripleO Train.

Supported OS:

RHEL7, RHEL8, RHEL9, Ubuntu 18.04, Ubuntu 20.04, Ubuntu 20.04 source image, Ubuntu 22.04, CentOS7, CentOS Stream 8, CentOS Linux 8, Ubuntu 20.04 binary image, CentOS Stream 8 binary image, CentOS Stream 8 source image.

Deployment:

Deploy using the distributions corresponding deployment toolsets, including Red Hat Director, JuJu Charms, Ansible, and Director.

Compatibility Matrix Detailed View

Requirements

Trilio has four main software components:

Trilio ships as a QCOW2 image. User can instantiate one or more VMs from the QCOW2 image on a standalone KVM boxes.
Trilio API is a python module that is an extension to nova api service. This module is installed on all OpenStack controller nodes
Trilio Datamover is a python module that is installed on every OpenStack compute nodes
Trilio horizon plugin is installed as an add on to horizon servers. This module is installed on every server that runs horizon service.

System requirements Trilio Appliance

The Trilio Appliance is not supported as instance inside Openstack.

The Trilio Appliance gets delivered as a qcow2 image, which gets attached to a virtual machine.

Trilio supports KVM-based hypervisors on x86 architectures, with the following properties:

Software

Supported

libvirt

2.0.0 and above

QEMU

2.0.0 and above

qemu-img

2.6.0 and above

The recommended size of the VM for the Trilio Appliance is:

When running Trilio in production, a 3-node cluster of the Trilio appliance is recommended for high availability and load balancing.

Ressource

Value

vCPU

RAM

24 GB

The qcow2 image itself defines the 40GB disk size of the VM.

In the rare case of the Trilio Appliance database or log files getting larger than 40GB disk, contact or open a ticket with Trilio Customer Success to attach another drive to the Trilio Appliance.

Software Requirements

In addition to the Trilio Appliance does Trilio contain components, which are installed directly into the Openstack itself.

Additionally, it is necessary to have the nfs-common packages installed on the compute nodes in case of using the NFS protocol for the backup target.

OpenStack Barbican for encryption

Since Trilio for OpenStack 4.2 is T4O capable of providing encrypted backups

The Trilio for OpenStack solution is leveraging the OpenStack Barbican service to provide encryption capabilities for its backups.

To be precise, T4O is using secrets provided by Barbican to encrypt and decrypt backups. Barbican is therefore required to utilize this feature.

Trilio network considerations

Trilio integrates natively with Openstack. This includes that Trilio communicates completely through APIs using the Openstack Endpoints. Trilio is also generating its own Openstack endpoints. In addition, is the Trilio appliance and the compute nodes writing to and reading from the backup target. These points affect the network planning for the Trilio installation.

Existing endpoints in Openstack

Openstack knows 3 types of endpoints:

Public Endpoints
Internal Endpoints
Admin Endpoints

Each of these endpoint types is designed for a specific purpose. Public endpoints are meant to be used by the Openstack end-users to work with Openstack. Internal endpoints are meant to be used by the Openstack services to communicate with each other. Admin endpoints are meant to be used by Openstack administrators.

Out of those 3 endpoint types does only the admin endpoint sometimes contain APIs which are not available on any other endpoint type.

To learn more about Openstack endpoints please visit the official Openstack documentation.

Openstack endpoints required by Trilio

Trilio is communicating with all services of Openstack on a defined endpoint type. Which endpoint type Trilio is using to communicate with Openstack is decided during the configuration of the Trilio appliance.

There is one exception: The Trilio Appliance always requires access to the Keystone admin endpoint.

The following network requirement can be identified this way:

Trilio appliance needs access to the Keystone admin endpoint on the admin endpoint network
Trilio appliance needs access to all endpoints of one type

Recommendation: Provide access to all Openstack Endpoint types

Trilio is recommending providing full access to all Openstack endpoints to the Trilio appliance to follow the Openstack standards and best practices.

Trilio is generating its own endpoints as well. These endpoints are pointing towards the Trilio Appliance directly. This means that using those endpoints will not send the API calls towards the Openstack Controller nodes first, but directly to the Trilio VM.

Following the Openstack standards and best practices, it is therefore recommended to put the Trilio endpoints on the same networks as the already existing Openstack endpoints. This allows to extend the purpose of each endpoint type to the Trilio service:

The public endpoint to be used by Openstack users when using Trilio CLI or API
The internal endpoint to communicate with the Openstack services
The admin endpoint to use the required admin only APIs of Keystone

Backup target access required by Trilio

The Trilio solution is using backup target storage to securely place the backup data. Trilio is dividing its backup data into two parts:

Metadata
Volume Disk Data

The first type of data is generated by the Trilio appliance through communicating with the Openstack Endpoints. All metadata that is stored together with a backup is written by the Trilio Appliance to the backup target in the json format.

The second type of data is generated by the Trilio Datamover service running on the compute nodes. The Datamover service is reading the Volume Data from the Cinder or Nova storage and transferring this data as qcow2 image to the backup target. Each Datamover service is hereby responsible for the VMs running on its compute node.

The network requirements are therefor:

The Trilio appliance needs access to the backup target
Every compute node needs access to the backup target

Example of a typical Trilio network integration

Most Trilio customers are following the Openstack standards and best practices to have the public, internal, and admin endpoints on separate networks. They also typically don't have any network yet, which can access the desired backup target.

The starting network configuration typically looks like this:

Following the Openstack standards and Trilio's recommendation will the Trilio Appliance be placed on all those 3 networks. Further is the access to the backup target required by Trilio Appliance and Compute nodes. Here done by adding a 4th network.

The resulting network configuration would look like this:

It is of course possible to combine networks as necessary. As long as the required network access is available will Trilio work.

Other examples of Trilio network integrations

Each Openstack installation is different and so is the network configuration. There are endless possibilities of how to configure the Openstack network and how to implement the Trilio appliance into this network. The following three examples have been seen in production:

The first example is from a manufacturing company, which wanted to split the networks by function and decided to put the Trilio backup target on the internal network as the backup and recovery function was identified as an Openstack internal solution. This example looks complex but integrates Trilio just as recommended.

The second example is from a financial institute that wanted to be sure that the Openstack Users have no direct uncontrolled network access to the Openstack infrastructure. Following this example requires additional work as the internal HA-Proxy needs to be configured to correctly translates the API calls towards the Trilio

The third example is from a service company that was forced to treat Trilio as an external 3rd party solution, as we require a virtual machine running outside of Openstack. This kind of network configuration requires good planning on the Trilio endpoints and firewall rules.

Preparing the installation

It is recommended to think about the following elements prior to the installation of Trilio for Openstack.

Tenant Quotas

Trilio uses Cinder snapshots for calculating full and incremental backups. For full backups, Trilio creates Cinder snapshots for all the volumes in the backup job. It then leaves these Cinder snapshots behind for calculating the incremental backup image during next backup. During an incremental backup operation it creates new Cinder snapshots, calculates the changed blocks between the new snapshots and the old snapshots that were left behind during full/previous backups. It then deletes the old snapshots but leaves the newly created snapshots behind. So, it is important that each tenant who is availing Trilio backup functionality has sufficient Cinder snapshot quotas to accommodate these additional snapshots. The guideline is to add 2 snapshots for every volume that is added to backups to volume snapshot quotas for that tenant. You may also increase the volume quotas for the tenant by the same amount because Trilio briefly creates a volume from snapshot to read data from the snapshot for backup purposes. During a restore process, Trilio creates additional instances and Cinder volumes. To accommodate restore operations, a tenant should have sufficient quota for Nova instances and Cinder volumes. Otherwise restore operations will result in failures.

AWS S3 eventual consistency¶

AWS S3 object consistency model includes:

Read-after-write
Read-after-update
Read-after-delete

Each of them describes how an object will reach its consistent state after an object is created/updated or deleted. None of them provides strong consistency and there is a lag time for an object to reach the consistent state. Though Trilio employed mechanisms to work around the limitations of eventual consistency of AWS S3, when an object reach its consistency state is not deterministic. There is no official statement from AWS on how long it takes for an object to reach consistent state. However read-after-write has a shorter time to reach consistency compared to other IO patterns. Our solution is designed to maximize read-after-write IO pattern. The time in which an object reaches eventual consistency also depends on the AWS region. For example, aws-standard region does not have strong consistency model compared to us-east or us-west. We suggest to use these regions when creating s3 buckets for Trilio. Though read-after-update IO pattern is hard to avoid completely, we employed ample delays in accessing objects to accommodate larger durations for objects to get into consistent state. However in rare occasions, backups may still fail and need to restarted.

Trilio Cluster¶

Trilio can be deployed as a single node or a three node cluster. It is highly recommended that Trilio is deployed as three node cluster for fault tolerance and load balancing. Starting with 3.0 release, Trilio requires additional IP for cluster and is required for both single node and three node deployments. Cluster ip a.k.a virtual ip is used for managing cluster and is used to register Trilio service endpoint in the keystone sevice catalog.

Spinning up the Trilio VM

Learn about spinning up the Trilio VM

For Canonical Openstack it is not necessary to spin up the Trilio VM. However, Trilio File Search functionality requires that the Trilio Workload manager (trilio-wlm) be deployed as a virtual machine. File Search will not function if the Trilio Workload manager (trilio-wlm) is running as a lxd container(s).

The Trilio Appliance is delivered as qcow2 image and runs as VM on top of a KVM Hypervisor.

This guide shows the tested way to spin up the Trilio Appliance on a RHV Cluster. Please contact a RHV Administrator and Trilio Customer Success Agent in case of incompatibility with company standards.

Creating the cloud-init image

The Trilio appliance is utilizing cloud-init to provide the initial network and user configuration.

Cloud-init is reading it's information either from a metadata server or from a provided cd image. Trilio is utilizing the cd image.

Needed tools

To create the cloud-init image it is required to have genisoimage available.

Providing the Metadata

Cloud-init is using two files for it's metadata.

The first file is called meta-data and contains the information about the network configuration. Below is an example of this file.

Keep the hostname localhost. The hostname gets changed through the configuration step. Changing the hostname will lead to the tvault-config service not properly starting, blocking further configuration.

The instance-id has to match the VM name in virsh

The second file is called user-data and contains little scripts and information to set up for example the user passwords. Below is an example of this file.

creating the image file

Both files meta-data and user-data are needed. Even when one is empty, is it needed to create a working cloud-init image.

The image is getting created using genisoimage follwing this general command:

genisoimage -output <name>.iso -volid cidata -joliet -rock </path/user-data> </path/meta-data>

An example of this command is shown below.

Spining up the Trilio appliance

After the cloud-init image has been created the TriloVault appliance can be spun up on the desired KVM server.

Extract the Trilio QCOW2 tar file using the following command :

See below an example command, how to spin up the Trilio appliance using virsh and the created iso image.

It is of course possible to spin up the Trilio appliance without a cloud-init iso-image. It will spin up with default values.

Uninstalling cloud-init after first boot

Once the Trilio appliance is up and running with it's initial configuration is it recommended to uninstall cloud-init.

If cloud-init is not installed it will rerun the network configuration upon every boot. Setting the network configuration back to DHCP, if no metadata is provided.

To uninstall cloud-init, follow the example below.

[RHOSP, TripleO and Kolla only] Verify nova UID/GID for nova user on the Appliance

Red Hat OpenStack, TripleO, and Kolla Ansible Openstack are using the nova UID/GID of 42436 inside their containers instead of 162:162 which is the standard in other Openstack environments.

Please verify that the nova UID/GID on the Trilio Appliance is still 42436,

In case of the UID/GID being changed back to 162:162 follow these steps to set it back to 42436:42436.

Download the shell script that will change the user id
Assign executable permissions
Execute the script
Verify that the nova user and group id have changed to '42436'

Updating the appliance to the latest minor version

It is recommended to directly update the Trilio appliance to the latest version.

To do so follow the minor update guide provided here:

Installing Trilio Components

Once the Trilio VM or the Cluster of Trilio VMs has been spun, the actual installation process can begin. This process contains of the following steps:

Install the Trilio dm-api service on the control plane
Install the Trilio datamover service on the compute plane
Install the Trilio Horizon plugin into the Horizon service

How these steps look in detail is depending on the Openstack distribution Trilio is installed in. Each supported Openstack distribution has its own deployment tools. Trilio is integrating into these deployment tools to provide a native integration from the beginning to the end.

Test text

Installing on RHOSP

Trilio is integrating natively into the RHOSP Director. Manual deployment methods are not supported for RHOSP.

1. Prepare for deployment

1.1] Select 'backup target' type

Backup target storage is used to store backup images taken by Trilio and details needed for configuration:

Following backup target types are supported by Trilio

a) NFS

Need NFS share path

b) Amazon S3

- S3 Access Key - Secret Key - Region - Bucket name

c) Other S3 compatible storage (Like, Ceph based S3)

- S3 Access Key - Secret Key - Region - Endpoint URL (Valid for S3 other than Amazon S3) - Bucket name

1.2] Clone triliovault-cfg-scripts repository

The following steps are to be done on 'undercloud' node on an already installed RHOSP environment. The overcloud-deploy command has to be run successfully already and the overcloud should be available.

All commands need to be run as user 'stack' on undercloud node

The following command clones the triliovault-cfg-scripts github repository.

cd /home/stack
git clone -b 4.3.2 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/

Next access the Red Hat Director scripts according to the used RHOSP version.

RHOSP 13

cd triliovault-cfg-scripts/redhat-director-scripts/rhosp13/

RHOSP 16.1

cd triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/

RHOSP 16.2

cd triliovault-cfg-scripts/redhat-director-scripts/rhosp16.2/

RHOSP 17.0

cd triliovault-cfg-scripts/redhat-director-scripts/rhosp17.0/

The remaining documentation will use the following path for examples:

cd triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/

1.3] If backup target type is 'Ceph based S3' with SSL:

If your backup target is ceph S3 with SSL and SSL certificates are self signed or authorized by private CA, then user needs to provide CA chain certificate to validate the SSL requests. For that, user needs to rename his ca chain cert file to 's3-cert.pem' and copy it into directory - 'triliovault-cfg-scripts/redhat-director-scripts/redhat-director-scripts/<RHOSP_RELEASE___Directory/puppet/trilio/files'

cp s3-cert.pem /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/puppet/trilio/files

2] Upload trilio puppet module

The following commands upload the Trilio puppet module to the overcloud registry. The actual upload happens upon the next deployment.

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/scripts/
./upload_puppet_module.sh

## Output of above command looks like following for RHOSP13, RHOSP16.1 and RHOSP16.2
Creating tarball...
Tarball created.
Creating heat environment file: /home/stack/.tripleo/environments/puppet-modules-url.yaml
Uploading file to swift: /tmp/puppet-modules-8Qjya2X/puppet-modules.tar.gz
+-----------------------+---------------------+----------------------------------+
| object                | container           | etag                             |
+-----------------------+---------------------+----------------------------------+
| puppet-modules.tar.gz | overcloud-artifacts | 368951f6a4d39cfe53b5781797b133ad |
+-----------------------+---------------------+----------------------------------+

## Above command creates following file.
ls -ll /home/stack/.tripleo/environments/puppet-modules-url.yaml

## Command is same for RHOSP17.0 but the command output and file content would be different

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp17.0/scripts/
./upload_puppet_module.sh

## Output of above command looks like following for RHOSP17.0
Creating tarball...
Tarball created.
renamed '/tmp/puppet-modules-P3duCg9/puppet-modules.tar.gz' -> '/var/lib/tripleo/artifacts/overcloud-artifacts/puppet-modules.tar.gz'
Creating heat environment file: /home/stack/.tripleo/environments/puppet-modules-url.yaml

## Above command creates following file.
ls -ll /home/stack/.tripleo/environments/puppet-modules-url.yaml

Trilio puppet module is uploaded to overcloud as a swift deploy artifact with heat resource name 'DeployArtifactURLs'. If you check trilio's puppet module artifact file it looks like following.

## For RHOSP13, RHOSP16.1 and RHOSP16.2
(undercloud) [stack@ucloud161 ~]$ cat /home/stack/.tripleo/environments/puppet-modules-url.yaml
# Heat environment to deploy artifacts via Swift Temp URL(s)
parameter_defaults:
    DeployArtifactURLs:
    - 'http://172.25.0.103:8080/v1/AUTH_46ba596219d143c8b076e9fcc4139fed/overcloud-artifacts/puppet-modules.tar.gz?temp_url_sig=c3972b7ce75226c278ab3fa8237d31cc1f2115bd&temp_url_expires=1646738377'

## For RHOSP17.0
(undercloud) [stack@undercloud17-3 scripts]$ cat /home/stack/.tripleo/environments/puppet-modules-url.yaml
parameter_defaults:
  DeployArtifactFILEs:
  - /var/lib/tripleo/artifacts/overcloud-artifacts/puppet-modules.tar.gz

Note: If your overcloud deploy command using any other deploy artifact through a environment file, then you need to merge trilio deploy artifact url and your url in single file.

How to check if your overcloud deploy environment files using deploy artifacts? You need to check string 'DeployArtifactURLs' in your environment files (only those mentioned in overcloud deploy command with -e option). If you find it any such environment file that is mentioned in overcloud dpeloy command with '-e' option then your deploy command is using deploy artifact.
In that case you need to merge all deploy artifacts in single file. Refer following steps.

Let's say, your artifact file path is "/home/stack/templates/user-artifacts.yaml" then refer following steps to merge both urls in single file and pass that new file to overcloud deploy command with '-e' option.

(undercloud) [stack@ucloud161 ~]$ cat /home/stack/.tripleo/environments/puppet-modules-url.yaml | grep http >> /home/stack/templates/user-artifacts.yaml
(undercloud) [stack@ucloud161 ~]$ cat /home/stack/templates/user-artifacts.yaml
# Heat environment to deploy artifacts via Swift Temp URL(s)
parameter_defaults:
    DeployArtifactURLs:
    - 'http://172.25.0.103:8080/v1/AUTH_57ba596219d143c8b076e9fcc4139f3g/overcloud-artifacts/some-artifact.tar.gz?temp_url_sig=dc972b7ce75226c278ab3fa8237d31cc1f2115sc&temp_url_expires=3446738365'
    - 'http://172.25.0.103:8080/v1/AUTH_46ba596219d143c8b076e9fcc4139fed/overcloud-artifacts/puppet-modules.tar.gz?temp_url_sig=c3972b7ce75226c278ab3fa8237d31cc1f2115bd&temp_url_expires=1646738377'

3] Update overcloud roles data file to include Trilio services

Trilio contains multiple services. Add these services to your roles_data.yaml.

In the case of uncustomized roles_data.yaml can the default file be found on the undercloud at:

/usr/share/openstack-tripleo-heat-templates/roles_data.yaml

Add the following services to the roles_data.yaml

All commands need to be run as user 'stack'

3.1] Add Trilio Datamover Api Service to role data file

This service needs to share the same role as the keystone and database service. In case of the pre-defined roles will these services run on the role Controller. In case of custom defined roles, it is necessary to use the same role where 'OS::TripleO::Services::Keystone' service installed.

Add the following line to the identified role:

'OS::TripleO::Services::TrilioDatamoverApi'

3.2] Add Trilio Datamover Service to role data file

This service needs to share the same role as the nova-compute service. In case of the pre-defined roles will the nova-compute service run on the role Compute. In case of custom defined roles, it is necessary to use the role the nova-compute service is using.

Add the following line to the identified role:

'OS::TripleO::Services::TrilioDatamover'

3.3] Add Trilio Horizon Service role data file

This service needs to share the same role as the OpenStack Horizon server. In case of the pre-defined roles will the Horizon service run on the role Controller. Add the following line to the identified role:

OS::TripleO::Services::TrilioHorizon

4] Prepare Trilio container images

All commands need to be run as user 'stack'

Trilio containers are pushed to 'RedHat Container Registry'. Registry URL: 'registry.connect.redhat.com'. Container pull URLs are given below.

Please note that using the hotfix containers requires that the Trilio Appliance is getting upgraded to the desired hotfix level as well.

Refer to the word <HOTFIX-TAG-VERSION> as 4.3.2 in the below sections

RHOSP 13

Trilio Datamover container:        registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp13
Trilio Datamover Api Container:   registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp13
Trilio horizon plugin:            registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp13

RHOSP 16.1

Trilio Datamover container:        registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1
Trilio Datamover Api Container:   registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1
Trilio horizon plugin:            registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1

RHOSP 16.2

Trilio Datamover container:        registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.2
Trilio Datamover Api Container:   registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.2
Trilio horizon plugin:            registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.2

RHOSP 17.0

Trilio Datamover container:        registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp17.0
Trilio Datamover Api Container:   registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp17.0
Trilio horizon plugin:            registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp17.0

There are three registry methods available in RedHat Openstack Platform.

Remote Registry
Local Registry
Satellite Server

4.1] Remote Registry

Follow this section when 'Remote Registry' is used.

In this method, container images gets downloaded directly on overcloud nodes during overcloud deploy/update command execution. User can set remote registry to redhat registry or any other private registry that he wants to use. User needs to provide credentials of the registry in 'containers-prepare-parameter.yaml' file.

Make sure other openstack service images are also using the same method to pull container images. If it's not the case you can not use this method.
Populate 'containers-prepare-parameter.yaml' with content like following. Important parameters are 'push_destination: false', ContainerImageRegistryLogin: true and registry credentials. Trilio container images are published to registry 'registry.connect.redhat.com'. Credentials of registry 'registry.redhat.io' will work for 'registry.connect.redhat.com' registry too.

Note: This file -'containers-prepare-parameter.yaml'

File Name: containers-prepare-parameter.yaml

parameter_defaults:
  ContainerImagePrepare:
  - push_destination: false
    set:
      namespace: registry.redhat.io/...
      ...
  ...
  ContainerImageRegistryCredentials:
    registry.redhat.io:
      myuser: 'p@55w0rd!'
    registry.connect.redhat.com:
      myuser: 'p@55w0rd!'
  ContainerImageRegistryLogin: true

Note: File 'containers-prepare-parameter.yaml' gets created as output of command 'openstack tripleo container image prepare'. Refer above document by RedHat

3. Make sure you have network connectivity to above registries from all overcloud nodes. Otherwise image pull operation will fail.

4. User need to manually populate the trilio_env.yaml file with Trilio container image URLs as given below:

trilio_env.yaml file path:

cd triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/

# For RHOSP13
$ grep '<HOTFIX-TAG-VERSION>-rhosp13' trilio_env.yaml
   DockerTrilioDatamoverImage: registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp13
   DockerTrilioDmApiImage: registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp13
   DockerHorizonImage: registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp13

# For RHOSP16.1
$ grep '<HOTFIX-TAG-VERSION>-rhosp16.1' trilio_env.yaml
   DockerTrilioDatamoverImage: registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1
   DockerTrilioDmApiImage: registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1
   ContainerHorizonImage: registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1

# For RHOSP16.2
$ grep '<HOTFIX-TAG-VERSION>-rhosp16.2' trilio_env.yaml
   DockerTrilioDatamoverImage: registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.2
   DockerTrilioDmApiImage: registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.2
   ContainerHorizonImage: registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.2

# For RHOSP17.0
$ grep '<HOTFIX-TAG-VERSION>-rhosp17.0' trilio_env.yaml
   DockerTrilioDatamoverImage: registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp17.0
   DockerTrilioDmApiImage: registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp17.0
   ContainerHorizonImage: registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp17.0

At this step, you have configured Trilio image urls in the necessary environment file.

4.2] Local Registry

Follow this section when 'local registry' is used on the undercloud.

In this case it is necessary to push the Trilio containers to the undercloud registry. Trilio provides shell scripts which will pull the containers from 'registry.connect.redhat.com' and push them to the undercloud and updates the trilio_env.yaml.

RHOSP13

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp13/scripts/

./prepare_trilio_images.sh <undercloud_ip/hostname> <HOTFIX-TAG-VERSION>-rhosp13

## Verify changes
$ grep '<HOTFIX-TAG-VERSION>-rhosp13' ../environments/trilio_env.yaml
   DockerTrilioDatamoverImage: 172.25.2.2:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp13
   DockerTrilioDmApiImage: 172.25.2.2:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp13
   DockerHorizonImage: 172.25.2.2:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp13

$ docker image list | grep <HOTFIX-TAG-VERSION>-rhosp13
172.30.5.101:8787/trilio/trilio-datamover                  <HOTFIX-TAG-VERSION>-rhosp13        f2dfb36bb176        8 weeks ago         3.61 GB
registry.connect.redhat.com/trilio/trilio-datamover        <HOTFIX-TAG-VERSION>-rhosp13        f2dfb36bb176        8 weeks ago         3.61 GB
172.30.5.101:8787/trilio/trilio-datamover-api              <HOTFIX-TAG-VERSION>-rhosp13        5d62f572a00c        8 weeks ago         2.24 GB
registry.connect.redhat.com/trilio/trilio-datamover-api    <HOTFIX-TAG-VERSION>-rhosp13        5d62f572a00c        8 weeks ago         2.24 GB
registry.connect.redhat.com/trilio/trilio-horizon-plugin   <HOTFIX-TAG-VERSION>-rhosp13        27c4de28e5ae        2 months ago        2.27 GB
172.30.5.101:8787/trilio/trilio-horizon-plugin             <HOTFIX-TAG-VERSION>-rhosp13        27c4de28e5ae        2 months ago        2.27 GB

RHOSP 16.1

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/scripts/

sudo ./prepare_trilio_images.sh <undercloud_ip/hostname> <HOTFIX-TAG-VERSION>-rhosp16.1

## Verify changes
$ grep '<HOTFIX-TAG-VERSION>-rhosp16.1' ../environments/trilio_env.yaml
   DockerTrilioDatamoverImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1
   DockerTrilioDmApiImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1
   ContainerHorizonImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1

$ openstack tripleo container image list | grep <HOTFIX-TAG-VERSION>-rhosp16.1
| docker://tlsundercloud.ctlplane.trilio.local:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1 |
| docker://tlsundercloud.ctlplane.trilio.local:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1      |
| docker://tlsundercloud.ctlplane.trilio.local:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1  |
-----------------------------------------------------------------------------------------------------

RHOSP16.2

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.2/scripts/

sudo ./prepare_trilio_images.sh <undercloud_ip/hostname> <HOTFIX-TAG-VERSION>-rhosp16.2

## Verify changes
$ grep '<HOTFIX-TAG-VERSION>-rhosp16.2' ../environments/trilio_env.yaml
   DockerTrilioDatamoverImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.2
   DockerTrilioDmApiImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.2
   ContainerHorizonImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.2

$ openstack tripleo container image list | grep <HOTFIX-TAG-VERSION>-rhosp16.2
| docker://tlsundercloud.ctlplane.trilio.local:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.2 |
| docker://tlsundercloud.ctlplane.trilio.local:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.2      |
| docker://tlsundercloud.ctlplane.trilio.local:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.2  |

RHOSP17.0

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp17.0/scripts/

sudo ./prepare_trilio_images.sh <undercloud_ip/hostname> <HOTFIX-TAG-VERSION>-rhosp17.0

## Verify changes
$ grep '<HOTFIX-TAG-VERSION>-rhosp17.0' ../environments/trilio_env.yaml
   DockerTrilioDatamoverImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp17.0
   DockerTrilioDmApiImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp17.0
   ContainerHorizonImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp17.0

$ openstack tripleo container image list | grep <HOTFIX-TAG-VERSION>-rhosp17.0
| docker://tlsundercloud.ctlplane.trilio.local:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp17.0 |
| docker://tlsundercloud.ctlplane.trilio.local:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp17.0      |
| docker://tlsundercloud.ctlplane.trilio.local:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp17.0  |

At this step, you have downloaded triliovault container images and configured triliovault image urls in necessary environment file.

4.3] Red Hat Satellite Server

Follow this section when a Satellite Server is used for the container registry.

Populate the trilio_env.yaml with container urls.

RHOSP 13

$ grep '<HOTFIX-TAG-VERSION>-rhosp13' ../environments/trilio_env.yaml
   DockerTrilioDatamoverImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp13
   DockerTrilioDmApiImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp13
   DockerHorizonImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp13

RHOSP 16.1

$ grep '<HOTFIX-TAG-VERSION>-rhosp16.1' trilio_env.yaml
   DockerTrilioDatamoverImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1
   DockerTrilioDmApiImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1
   ContainerHorizonImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1

RHOSP16.2

$ grep '<HOTFIX-TAG-VERSION>-rhosp16.2' trilio_env.yaml
   DockerTrilioDatamoverImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.2
   DockerTrilioDmApiImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.2
   ContainerHorizonImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.2

RHOSP17.0

$ grep '<HOTFIX-TAG-VERSION>-rhosp17.0' trilio_env.yaml
   DockerTrilioDatamoverImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp17.0
   DockerTrilioDmApiImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp17.0
   ContainerHorizonImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp17.0

At this step, you have downloaded triliovault container images into RedHat sattelite server and configured triliovault image urls in necessary environment file.

5] Configure multi-IP NFS

This section is only required when the multi-IP feature for NFS is required.

This feature allows to set the IP to access the NFS Volume per datamover instead of globally.

On Undercloud node, change directory

cd triliovault-cfg-scripts/common/

Edit file 'triliovault_nfs_map_input.yml' in the current directory and provide compute host and NFS share/ip map.

Get the compute hostnames from the following command. Check ‘Name' column. Use exact hostnames in 'triliovault_nfs_map_input.yml' file.

Run this command on undercloud by sourcing 'stackrc'.

(undercloud) [stack@ucqa161 ~]$ openstack server list
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+
| ID                                   | Name                          | Status | Networks             | Image          | Flavor  |
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+
| 8c3d04ae-fcdd-431c-afa6-9a50f3cb2c0d | overcloudtrain1-controller-2  | ACTIVE | ctlplane=172.30.5.18 | overcloud-full | control |
| 103dfd3e-d073-4123-9223-b8cf8c7398fe | overcloudtrain1-controller-0  | ACTIVE | ctlplane=172.30.5.11 | overcloud-full | control |
| a3541849-2e9b-4aa0-9fa9-91e7d24f0149 | overcloudtrain1-controller-1  | ACTIVE | ctlplane=172.30.5.25 | overcloud-full | control |
| 74a9f530-0c7b-49c4-9a1f-87e7eeda91c0 | overcloudtrain1-novacompute-0 | ACTIVE | ctlplane=172.30.5.30 | overcloud-full | compute |
| c1664ac3-7d9c-4a36-b375-0e4ee19e93e4 | overcloudtrain1-novacompute-1 | ACTIVE | ctlplane=172.30.5.15 | overcloud-full | compute |
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+

vi triliovault_nfs_map_input.yml

Update pyyaml on the undercloud node only

If pip isn't available please install pip on the undercloud.

## On Python3 env
sudo pip3 install PyYAML==5.1 3

## On Python2 env
sudo pip install PyYAML==5.1

Expand the map file to create a one-to-one mapping of the compute nodes and the nfs shares.

## On Python3 env
python3 ./generate_nfs_map.py

## On Python2 env
python ./generate_nfs_map.py

The result will be in file - 'triliovault_nfs_map_output.yml'

Validate output map file

Open file 'triliovault_nfs_map_output.yml

vi triliovault_nfs_map_output.yml

available in the current directory and validate that all compute nodes are covered with all the necessary NFS shares.

Append this output map file to 'triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_nfs_map.yaml'

grep ':.*:' triliovault_nfs_map_output.yml >> ../redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_nfs_map.yaml

Validate the changes in file 'triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_nfs_map.yaml'

Include the environment file (trilio_nfs_map.yaml) in overcloud deploy command with '-e' option as shown below.

-e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_nfs_map.yaml

Ensure that MultiIPNfsEnabled is set to true in trilio_env.yaml file and that nfs is used as backup target.

6] Provide environment details in trilio-env.yaml

Provide backup target details and other necessary details in the provided environment file. This environment file will be used in the overcloud deployment to configure Trilio components. Container image names have already been populated in the preparation of the container images. Still it is recommended to verify the container URLs.

The following information are required additionally:

Network for the datamover api
datamover password
Backup target type {nfs/s3}
In case of NFS
- list of NFS Shares
- NFS options
- MultiIPNfsEnabled

NFS options for Cohesity NFS : nolock,soft,timeo=600,intr,lookupcache=none,nfsvers=3,retrans=10

In case of S3
- S3 type {amazon_s3/ceph_s3}
- S3 Access key
- S3 Secret key
- S3 Region name
- S3 Bucket
- S3 Endpoint URL
- S3 Signature Version
- S3 Auth Version
- S3 SSL Enabled {true/false}
- S3 SSL Cert

Use ceph_s3 for any non-aws S3 backup targets.

resource_registry:
  OS::TripleO::Services::TrilioDatamover: ../services/trilio-datamover.yaml
  OS::TripleO::Services::TrilioDatamoverApi: ../services/trilio-datamover-api.yaml
  OS::TripleO::Services::TrilioHorizon: ../services/trilio-horizon.yaml

  # NOTE: If there are addition customizations to the endpoint map (e.g. for
  # other integratiosn), this will need to be regenerated.
  OS::TripleO::EndpointMap: endpoint_map.yaml

parameter_defaults:

   ## Enable Trilio's quota functionality on horizon
   ExtraConfig:
     horizon::customization_module: 'dashboards.overrides'

   ## Define network map for trilio datamover api service
   ServiceNetMap:
       TrilioDatamoverApiNetwork: internal_api

   ## Trilio Datamover Password for keystone and database
   TrilioDatamoverPassword: "test1234"

   ## Trilio container pull urls
   DockerTrilioDatamoverImage: devundercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1
   DockerTrilioDmApiImage: devundercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1

   ## If you do not want Trilio's horizon plugin to replace your horizon container, just comment following line.
   ContainerHorizonImage: devundercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1

   ## Backup target type nfs/s3, used to store snapshots taken by triliovault
   BackupTargetType: 'nfs'

   ## If backup target NFS share support multiple IPs and you want to use those IPs(more than one) then
   ## set this parameter to True. Otherwise keep it False.
   MultiIPNfsEnabled: False

   ## For backup target 'nfs'
   NfsShares: '192.168.122.101:/opt/tvault'
   NfsOptions: 'nolock,soft,timeo=180,intr,lookupcache=none'

   ## For backup target 's3'
   ## S3 type: amazon_s3/ceph_s3
   S3Type: 'amazon_s3'

   ## S3 access key
   S3AccessKey: ''

   ## S3 secret key
   S3SecretKey: ''

   ## S3 region, if your s3 does not have any region, just keep the parameter as it is
   S3RegionName: ''

   ## S3 bucket name
   S3Bucket: ''

   ## S3 endpoint url, not required for Amazon S3, keep it as it is
   S3EndpointUrl: ''

   ## S3 signature version
   S3SignatureVersion: 'default'

   ## S3 Auth version
   S3AuthVersion: 'DEFAULT'

   ## If S3 backend is not Amazon S3 and SSL is enabled on S3 endpoint url then change it to 'True', otherwise keep it as 'False'
   S3SslEnabled: False

   ## If S3 backend is not Amazon S3 and SSL is enabled on S3 endpoint URL and SSL certificates are self signed, then
   ## user need to set this parameter value to: '/etc/tvault-contego/s3-cert.pem', otherwise keep it's value as empty string.
   S3SslCert: ''

   ## Configure 'dmapi_workers' parameter of '/etc/dmapi/dmapi.conf' file
   ## This parameter value used to spawn the number of dmapi processes to handle the incoming api requests.
   ## If your dmapi node has ‘n' cpu cores, It is recommended, to set this parameter to '4*n’.
   ## If dmapi_workers field is not present in config file. The Default value will be equals to number of cores present on the node
   DmApiWorkers: 16

   ## Don't edit following parameter
   EnablePackageInstall: True


   ## Load 'rbd' kernel module on all compute nodes
   ComputeParameters:
     ExtraKernelModules:
       rbd: {}

7] Advanced Settings/Configuration

7.1] Haproxy customized configuration for Trilio dmapi service __

The existing default haproxy configuration works fine with most of the environments. Only when timeout issues with the dmapi are observed or other reasons are known, change the configuration as described here.

Following is the haproxy conf file location on haproxy nodes of the overcloud. Trilio datamover api service haproxy configuration gets added to this file.

/var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg

Trilio datamover haproxy default configuration from the above file looks as follows:

listen trilio_datamover_api
  bind 172.25.0.107:13784 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 172.25.0.107:8784 transparent
  balance roundrobin
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  http-request set-header X-Forwarded-Port %[dst_port]
  maxconn 50000
  option httpchk
  option httplog
  retries 5
  timeout check 10m
  timeout client 10m
  timeout connect 10m
  timeout http-request 10m
  timeout queue 10m
  timeout server 10m
  server overcloud-controller-0.internalapi.localdomain 172.25.0.106:8784 check fall 5 inter 2000 rise 2

The user can change the following configuration parameter values.

retries 5
timeout http-request 10m
timeout queue 10m
timeout connect 10m
timeout client 10m
timeout server 10m
timeout check 10m
balance roundrobin
maxconn 50000

To change these default values, you need to do the following steps. i) On the undercloud node, open the following file for edit (Edit <RHOSP_RELEASE> with your cloud's release information. Valid values are - rhosp13, rhosp16, rhosp16.1)

For RHOSP13

/home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp13/services/trilio-datamover-api.yaml

For RHOSP16.1

/home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/services/trilio-datamover-api.yaml

For RHOSP16.2

/home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.2/services/trilio-datamover-api.yaml

For RHOSP17.0

/home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp17.0/services/trilio-datamover-api.yaml

ii) Search the following entries and edit as required

          tripleo::haproxy::trilio_datamover_api::options:
             'retries': '5'
             'maxconn': '50000'
             'balance': 'roundrobin'
             'timeout http-request': '10m'
             'timeout queue': '10m'
             'timeout connect': '10m'
             'timeout client': '10m'
             'timeout server': '10m'
             'timeout check': '10m'

iii) Save the changes.

7.2] Configure Custom Volume/Directory Mounts for the Trilio Datamover Service

If the user wants to add one or more extra volume/directory mounts to the Trilio Datamover Service container, they can use the following heat environment file. A variable named 'TrilioDatamoverOptVolumes' is available in this file.

triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE>/environments/trilio_datamover_opt_volumes.yaml

This variable 'TrilioDatamoverOptVolumes' accepts list of volume/bind mounts.
User needs to edit this file and add their volume mounts in below format.
For example:
In this volume mount "/mnt/dir2:/var/dir2", "/mnt/dir2" is a diretcory available on compute host and "/var/dir2" is the mount point inside datamover container.

parameter_defaults:
  TrilioDatamoverOptVolumes:
    - /opt/dir1:/opt/dir1
    - /mnt/dir2:/var/dir2

Next, User needs to pass this file to overcloud deploy command with '-e' option Like below.

openstack overcloud deploy --templates \
-e <> \
.
.
-e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/environments/trilio_datamover_opt_volumes.yaml

8] Deploy overcloud with trilio environment

Use the following heat environment file and roles data file in overcloud deploy command:

trilio_env.yaml
roles_data.yaml
Use correct Trilio endpoint map file as per available Keystone endpoint configuration
1. Instead of tls-endpoints-public-dns.yaml file, use environments/trilio_env_tls_endpoints_public_dns.yaml
2. Instead of tls-endpoints-public-ip.yaml file, useenvironments/trilio_env_tls_endpoints_public_ip.yaml
3. Instead of tls-everywhere-endpoints-dns.yaml file, useenvironments/trilio_env_tls_everywhere_dns.yaml
If activated use the correct trilio_nfs_map.yaml file

To include new environment files use '-e' option and for roles data file use '-r' option. An example overcloud deploy command is shown below:

openstack overcloud deploy --templates \
  -e /home/stack/templates/node-info.yaml \
  -e /home/stack/templates/overcloud_images.yaml \
  -e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/environments/trilio_env.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-tls.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/inject-trust-anchor.yaml \
  -e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/environments/trilio_env_tls_endpoints_public_dns.yaml \
  -e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/environments/trilio_nfs_map.yaml \
  --ntp-server 192.168.1.34 \
  --libvirt-type qemu \
  --log-file overcloud_deploy.log \
  -r /usr/share/openstack-tripleo-heat-templates/roles_data.yaml

Post deployment for multipath enabled environment, log into respective datamover container and add uxsock_timeout with value as 60000 (i.e. 60 sec) in /etc/multipath.conf. Restart datamover container

9] Verify deployment

If the containers are in restarting state or not listed by the following command then your deployment is not done correctly. Please recheck if you followed the complete documentation.

9.1] On Controller node

Make sure Trilio dmapi and horizon containers are in a running state and no other Trilio container is deployed on controller nodes. When the role for these containers is not "controller" check on respective nodes according to configured roles_data.yaml.

[root@overcloud-controller-0 heat-admin]# podman ps | grep trilio
26fcb9194566  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.2        kolla_start           5 days ago  Up 5 days ago         trilio_dmapi
094971d0f5a9  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.2       kolla_start           5 days ago  Up 5 days ago         horizon

Verify the haproxy configuration under:

/var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg

9.2] On Compute node

Make sure Trilio datamover container is in running state and no other Trilio container is deployed on compute nodes.

[root@overcloud-novacompute-0 heat-admin]# podman ps | grep trilio
b1840444cc59  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.2                    kolla_start           5 days ago  Up 5 days ago         trilio_datamover

9.3] On the node with Horizon service

Make sure horizon container is in running state. Please note that 'Horizon' container is replaced with Trilio Horizon container. This container will have latest OpenStack horizon + Trilio's horizon plugin.

[root@overcloud-controller-0 heat-admin]# podman ps | grep horizon
094971d0f5a9  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.2       kolla_start           5 days ago  Up 5 days ago         horizon

If Trilio Horizon container is in restarted state on RHOSP 16.1.8/RHSOP 16.2.4 then use below workaroud

## Either of the below workarounds should be performed on all the controller nodes where issue occurs for horizon pod.

option-1: Restart the memcached service on controller using systemctl (command: systemctl restart tripleo_memcached.service)

option-2: Restart the memcached pod (command: podman restart memcached)

10] Troubleshooting for overcloud deployment failures

Trilio components will be deployed using puppet scripts.

openstack stack failures list overcloud
heat stack-list --show-nested -f "status=FAILED"
heat resource-list --nested-depth 5 overcloud | grep FAILED

=> If trilio datamover api containers does not start well or in restarting state, use following logs to debug.


docker logs trilio_dmapi

tailf /var/log/containers/trilio-datamover-api/dmapi.log



=> If trilio datamover containers does not start well or in restarting state, use following logs to debug.


docker logs trilio_datamover

tailf /var/log/containers/trilio-datamover/tvault-contego.log

Installing on Canonical OpenStack

Trilio and Canonical have started a partnership to ensure a native deployment of Trilio using JuJu Charms.

Those JuJu Charms are publicly available as Open Source Charms.

Trilio is providing the JuJu Charms to deploy Trilio 4.2 in Canonical OpenStack from Yoga release onwards only. JuJu Charms to deploy Trilio 4.2 in Canonical OpenStack up to wallaby release are developed and maintained by Canonical.

Canonical OpenStack doesn't require the Trilio Cluster. The required services are installed and managed via JuJu Charms.

The documentation of the charms can be found here:

Juju charms for OpenStack Yoga release onwards

Juju charms for other supported OpenStack releases upto wallaby

Prerequisite

Have a canonical OpenStack base setup deployed for a required release like Jammy Zed/Yoga, Focal yoga/Wallaby/Victoria/Ussuri, or Bionic Ussuri/Queens.

Steps to install the Trilio charms

Export the OpenStack base bundle

2. Create a Trilio overlay bundle as per the OpenStack setup release using the charms given above.

NFS options for Cohesity NFS : nolock,soft,timeo=600,intr,lookupcache=none,nfsvers=3,retrans=10

Trilio File Search functionality requires that the Trilio Workload manager (trilio-wlm) be deployed as a virtual machine. File Search will not function if the Trilio Workload manager (trilio-wlm) is running as a lxd container(s).

3. If file search functionality is required, provision any additional node(s) that will be required for deploying the Trilio Workload manager (trilio-wlm) as a VM instead of lxd container(s).

4. Commission the additional node from MAAS UI.

5. Do a dry run to check if the Trilio bundle is working

6. Do the deployment

7. Wait till all the Trilio units are deployed successfully. Check the status via juju status command.

8. Once the deployment is complete, perform the below operations:

Create cloud admin trust

Add license

Note: Reach out to the Trilio support team for the license file.

For multipath enabled environments, perform the following actions

log into each nova compute node
add uxsock_timeout with value as 60000 (i.e. 60 sec) in /etc/multipath.conf
restart tvault-contego service

Sample Trilio overlay bundles

For bionic-queens openstack-origin parameter value for trilio-dm-api charm must be cloud:bionic-train

Installing on Kolla Openstack

This page lists all steps required to deploy Trilio components on Kolla-ansible deployed OpenStack cloud.

1] Plan for Deployment

Please ensure that the Trilio Appliance has been updated to the latest maintenance release before continuing the installation.

Refer to the below-mentioned acceptable values for the placeholders triliovault_tag and kolla_base_distro , in this document as per the Openstack environment:

1.1] Select backup target type

Backup target storage is used to store backup images taken by Trilio and details needed for configuration:

Following backup target types are supported by Trilio. Select one of them and get it ready before proceeding to the next step.

a) NFS

Need NFS share path

b) Amazon S3

- S3 Access Key - Secret Key - Region - Bucket name

c) Other S3 compatible storage (Like, Ceph based S3)

- S3 Access Key - Secret Key - Region - Endpoint URL (Valid for S3 other than Amazon S3) - Bucket name

2] Clone Trilio Deployment Scripts

Clone triliovault-cfg-scripts GitHub repository on Kolla ansible server at '/root' or any other directory of your preference. Afterwards, copy Trilio Ansible role into Kolla-ansible roles directory

3] Hook Trilio deployment scripts to Kolla-ansible deploy scripts

3.1] Add Trilio global variables to globals.yml

3.2] Add Trilio passwords to kolla passwords.yaml

Append triliovault_passwords.yml to /etc/kolla/passwords.yml. Passwords are empty. Set these passwords manually in the /etc/kolla/passwords.yml.

3.3] Append Trilio site.yml content to kolla ansible’s site.yml

3.4] Append triliovault_inventory.txt to your cloud’s kolla-ansible inventory file.

3.5] Configure multi-IP NFS

This step is only required when the multi-IP NFS feature is used to connect different datamovers to the same NFS volume through multiple IPs

On kolla-ansible server node, change directory

Edit file 'triliovault_nfs_map_input.yml' in the current directory and provide compute host and NFS share/ip map.

If IP addresses are used in the kolla-ansible inventory file then you should use same IP addresses in 'triliovault_nfs_map_input.yml' file too. If you used hostnames there then you need to use same hostnames here in nfs map input file.

Compute host names or IP addresses that you are using in nfs map input file here should match with kolla-ansible inventory file entries.

vi triliovault_nfs_map_input.yml

Update PyYAML on the kolla-ansible server node only

Expand the map file to create one to one mapping of compute and nfs share.

Result will be in file - 'triliovault_nfs_map_output.yml'

Validate output map file

Open file 'triliovault_nfs_map_output.yml

vi triliovault_nfs_map_output.yml

available in the current directory and validate that all compute nodes are covered with all necessary nfs shares.

Append this output map file to 'triliovault_globals.yml' File Path: '/home/stack/triliovault-cfg-scripts/kolla-ansible/ansible/triliovault_globals.yml’

Ensure to set multi_ip_nfs_enabled in __ triliovault_globals.yml file to yes

4] Edit globals.yml to set Trilio parameters

Edit /etc/kolla/globals.yml file to fill Trilio backup target and build details. You will find the Trilio related parameters at the end of globals.yml file. Details like Trilio build version, backup target type, backup target details, etc need to be filled out.

Following is the list of parameters that the usr needs to edit.

In the case of a different registry than docker hub, Trilio containers need to be pulled from docker.io and pushed to preferred registries.

Following are the triliovault container image URLs for 4.3 releases**.** Replace kolla_base_distro and triliovault_tag variables with their values.\

This {{ kolla_base_distro }} variable can be either 'centos' or 'ubuntu' depends on your base OpenStack distro

Trilio supports the Source-based containers from the OpenStack Yoga release **** onwards.

Below are the Source-based OpenStack deployment images

Below are the Binary-based OpenStack deployment images

5] Enable Trilio Snapshot mount feature

To enable Trilio's Snapshot mount feature it is necessary to make the Trilio Backup target available to the nova-compute and nova-libvirt containers.

Edit /usr/local/share/kolla-ansible/ansible/roles/nova-cell/defaults/main.yml and find nova_libvirt_default_volumes variables. Append the Trilio mount bind /var/trilio:/var/trilio:shared to the list of already existing volumes.

For a default Kolla installation, will the variable look as follows afterward:

Next, find the variable nova_compute_default_volumes in the same file and append the mount bind /var/trilio:/var/trilio:shared to the list.

After the change will the variable look for a default Kolla installation as follows:

In case of using Ironic compute nodes one more entry need to be adjusted in the same file. Find the variable nova_compute_ironic_default_volumes and append trilio mount /var/trilio:/var/trilio:shared to the list.

After the changes the variable will looks like the following:

6] Pull Trilio container images

Activate the login into dockerhub for Trilio tagged containers.

Please get the Dockerhub login credentials from Trilio Sales/Support team

Pull the Trilio container images from the dockerhub based on the existing inventory file. In the example is the inventory file named multinode.

7] Deploy Trilio

All that is left, is to run the deploy command using the existing inventory file. In the example is the inventory file named 'multinode'.

This is just an example command. You need to use your cloud deploy command.

Post deployment for multipath enabled environment, log into respective datamover container and add uxsock_timeout with value as 60000 (i.e. 60 sec) in /etc/multipath.conf. Restart datamover container

8] Verify Trilio deployment

Verify on the controller and compute nodes that the Trilio containers are in UP state.

Following is a sample output of commands from controller and compute nodes. triliovault_tag will have value as per the openstack release where deployment being done.

9] Troubleshooting Tips

9.1 ] Check Trilio containers and their startup logs

To see all TriloVault containers running on a specific node use the docker ps command.

To check the startup logs use the docker logs <container name> command.

9.2] Trilio Horizon tabs are not visible in Openstack

Verify that the Trilio Appliance is configured. The Horizon tabs are only shown, when a configured Trilio appliance is available.

Verify that the Trilio horizon container is installed and in a running state.

9.3] Trilio Service logs

Trilio datamover api service logs on datamover api node

Trilio datamover service logs on datamover node

10. Change the nova user id on the Trilio Nodes

Note: This step needs to be done on Trilio Appliance node. Not on OpenStack node.

Pre-requisite: You should have already launched Trilio appliance VM

In Kolla openstack distribution, 'nova' user id on nova-compute docker container is set to '42436'. The 'nova' user id on the Trilio nodes need to be set the same. Do the following steps on all Trilio nodes:

Download the shell script that will change the user id
Assign executable permissions
Execute the script
Verify that 'nova' user and group id has changed to '42436'
After this step, you can proceed to 'Configuring Trilio' section.

11. Advanced configurations - [Optional]

11.1] We are using cinder's ceph user for interacting with Ceph cinder storage. This user name is defined using parameter - 'ceph_cinder_user' in the file '/etc/kolla/globals.yaml'.

If user wants to edit this parameter value they can do it. Impact will be, cinder's ceph user and triliovault datamover's ceph user will be updated upon next kolla-ansible deploy command.

Installing on Ansible Openstack

Please ensure that the Trilio Appliance has been updated to the latest hotfix before continuing the installation.

Change the nova user id on the Trilio Nodes

The 'nova' user ID and Group ID on the Trilio nodes need to be set the same as in the compute node(s). Trilio is by default using the nova user ID (UID) and Group ID (GID) 162:162. Ansible OpenStack is not always 'nova' user id 162 on compute node. Do the following steps on all Trilio nodes in case of nova UID & GID are not in sync with the Compute Node(s)

Download the shell script that will change the user-id
Assign executable permissions
Edit script to use the correct nova id
Execute the script
Verify that 'nova' user and group id has changed to the desired value

Prepare deployment host

Clone triliovault-cfg-scripts from github repository on Ansible Host.

Available values for <branch>: hotfix-4-TVO/4.2

Copy Ansible roles and vars to required places.

In case of installing on OSA Victora or OSA Wallaby edit OPENSTACK_DIST in the file /etc/openstack_/user_tvault_vars.yml to victoria or wallaby respectively

Add Trilio playbook to /opt/openstack-ansible/playbooks/setup-openstack.ymlat the end of the file.

Add the following content at the end of the file /etc/openstack_deploy/user_variables.yml

Create the following file /opt/openstack-ansible/inventory/env.d/tvault-dmapi.yml

Add the following content to the created file.

Edit the file /etc/openstack_deploy/openstack_user_config.yml according to the example below to set host entries for Trilio components.

Edit the common editable parameter section in the file /etc/openstack_deploy/user_tvault_vars.yml

Append the required details like Trilio Appliance IP address, Openstack distribution, snapshot storage backend, SSL related information, etc.

Note:

From 4.2HF4 onwards default prefilled value i.e 4.2.64 will be used for TVAULT_PACKAGE_VERSION .
In case of more than one nova virtual environment If the user wants to install tvault-contego service in a specific nova virtual environment on compute node(s) then needs to uncomment var nova_virtual_env and then set the value of nova_virtual_env
In case of more than one horizon plugin configured on openstack user can specify under which horizon plugins to install Trilio Horizon Plugins by setting horizon_virtual_env parameter. Default value of horizon_virtual_env is ' /openstack/venvs/horizon*'\
NFS options for Cohesity NFS : nolock,soft,timeo=600,intr,lookupcache=none,nfsvers=3,retrans=10

Configure Multi-IP NFS

This step is only required when the multi-IP NFS feature is used to connect different datamovers to the same NFS volume through multiple IPs

New parameter added to /etc/openstack_deploy/user_tvault_vars.yml file for Mutli-IP NFS\

Change the Directory

Edit file 'triliovault_nfs_map_input.yml' in current directory and provide compute host and NFS share/IP map.

Update pyyaml on the Openstack Ansible server node only

Execute generate_nfs_map.py file to create one to one mapping of compute and nfs share.

Result will be in file - 'triliovault_nfs_map_output.yml' of the current directory

Validate output map file Open file 'triliovault_nfs_map_output.yml

available in current directory and validate that all compute nodes are mapped with all necessary nfs shares.

Append the content of triliovault_nfs_map_output.yml file to /etc/openstack_deploy/user_tvault_vars.yml

Deploy Trilio components

Run the following commands to deploy only Trilio components in case of an already deployed Ansible Openstack.

If Ansible Openstack is not already deployed then run the native Openstack deployment commands to deploy Openstack and Trilio Components together. An example for the native deployment command is given below:

Verify the Trilio deployment

Verify triliovault datamover api service deployed and started well. Run the below commands on controller node(s).

Verify triliovault datamover service deployed and started well on compute node(s). Run the following command oncompute node(s).

Verify that triliovault horizon plugin, contegoclient, and workloadmgrclient are installed on the Horizon container.

Run the following command on Horizon container.

Verify that haproxy setting on controller node using below commands.

Update to the latest hotfix

After the deployment has been verified it is recommended to update to the latest hotfix to ensure the best possible experience.

Installing on TripleO Train

1. Prepare for deployment

1.1] Select 'backup target' type

Backup target storage is used to store backup images taken by Trilio and details needed for configuration:

The following backup target types are supported by Trilio

a) NFS

Need NFS share path

b) Amazon S3

- S3 Access Key - Secret Key - Region - Bucket name

c) Other S3 compatible storage (Like, Ceph based S3)

- S3 Access Key - Secret Key - Region - Endpoint URL (Valid for S3 other than Amazon S3) - Bucket name

1.2] Clone triliovault-cfg-scripts repository

All commands need to be run as user 'stack' on undercloud node

TripleO CentOS8 is not supported anymore as CentOS Linux 8 has reached End of Life on December 31st,2021.

The following command clones the triliovault-cfg-scripts github repository.

cd /home/stack
git clone -b 4.3.2 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/

Please note that the Trilio Appliance needs to get updated to the latest HF as well.

1.3] If the backup target type is 'Ceph based S3' with SSL:

If your backup target is ceph S3 with SSL and SSL certificates are self-signed or authorized by a private CA, then the user needs to provide a CA chain certificate to validate the SSL requests. For that, the user needs to rename his ca chain cert file to s3-cert.pem and copy it into the directory triliovault-cfg-scripts/redhat-director-scripts/redhat-director-scripts/<RHOSP_RELEASE_Directory/puppet/trilio/files

cp s3-cert.pem /home/stack/triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/puppet/trilio/files/

2] Upload Trilio puppet module

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/scripts/
chmod +x *.sh
./upload_puppet_module.sh

## Output of the above command looks like the following.
Creating tarball...
Tarball created.
Creating heat environment file: /home/stack/.tripleo/environments/puppet-modules-url.yaml
Uploading file to swift: /tmp/puppet-modules-8Qjya2X/puppet-modules.tar.gz
+-----------------------+---------------------+----------------------------------+
| object                | container           | etag                             |
+-----------------------+---------------------+----------------------------------+
| puppet-modules.tar.gz | overcloud-artifacts | 368951f6a4d39cfe53b5781797b133ad |
+-----------------------+---------------------+----------------------------------+

## Above command creates the following file.
ls -ll /home/stack/.tripleo/environments/puppet-modules-url.yaml

3] Update overcloud roles data file to include Trilio services

Trilio contains multiple services. Add these services to your roles_data.yaml.

In the case of uncustomized roles_data.yaml can the default file be found on the undercloud at:

/usr/share/openstack-tripleo-heat-templates/roles_data.yaml

Add the following services to the roles_data.yaml

All commands need to be run as user 'stack'

3.1] Add Trilio Datamover Api Service to role data file

This service needs to share the same role as the keystone and database service. In the case of the pre-defined roles will these services run on the role Controller. In the case of custom-defined roles, it is necessary to use the same role where OS::TripleO::Services::Keystone service is installed.

Add the following line to the identified role:

'OS::TripleO::Services::TrilioDatamoverApi'

3.2] Add Trilio Datamover Service to role data file

This service needs to share the same role as the nova-compute service. In the case of the pre-defined roles will the nova-compute service run on the role Compute. In the case of custom-defined roles, it is necessary to use the role the nova-compute service is used.

Add the following line to the identified role:

'OS::TripleO::Services::TrilioDatamover'

3.3] Add Trilio Horizon Service role data file4] Prepare Trilio container images

This service needs to share the same role as the OpenStack Horizon server. In the case of the pre-defined roles, Horizon service runs on the role Controller. Add the following line to the identified role:

'OS::TripleO::Services::TrilioHorizon'

All commands need to be run as user 'stack'

Refer to the word <HOTFIX-TAG-VERSION> as 4.3.2 in the below sections

Trilio containers are pushed to 'Dockerhub'. Registry URL: 'docker.io'. Container pull URLs are given below.

CentOS7

Trilio Datamove container:        docker.io/trilio/tripleo-train-centos7-trilio-datamover:<HOTFIX-TAG-VERSION>-tripleo
Trilio Datamover Api Container:   docker.io/trilio/tripleo-train-centos7-trilio-datamover-api:<HOTFIX-TAG-VERSION>-tripleo
Trilio horizon plugin:            docker.io/trilio/tripleo-train-centos7-trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-tripleo

There are two registry methods available in TripleO Openstack Platform.

Remote Registry
Local Registry

4.1] Remote Registry

Follow this section when 'Remote Registry' is used.

For this method, it is not necessary to pull the containers in advance. It is only necessary to populate the trilio_env.yaml file with the Trilio container URLs from the Dockerhub registry.

Populate the trilio_env.yaml with container URLs for:

Trilio Datamover container
Trilio Datamover api container
Trilio Horizon Plugin

trilio_env.yaml will be available in __triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/environments

# For TripleO Train Centos7
$ grep 'Image' trilio_env.yaml
   DockerTrilioDatamoverImage: docker.io/trilio/tripleo-train-centos7-trilio-datamover:<HOTFIX-TAG-VERSION>-tripleo
   DockerTrilioDmApiImage: docker.io/trilio/tripleo-train-centos7-trilio-datamover-api:<HOTFIX-TAG-VERSION>-tripleo
   DockerHorizonImage: docker.io/trilio/tripleo-train-centos7-trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-tripleo

4.2] Local Registry

Follow this section when 'local registry' is used on the undercloud.

Run the following script. Script pulls the triliovault containers and updates the triliovault environment file with URLs.

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/scripts/

sudo ./prepare_trilio_images.sh <undercloud_registry_hostname_or_ip> <OS_platform> <4.1-TRIPLEO-CONTAINER> <container_tool_available_on_undercloud>

Options OS_platform: [centos7]
Options container_tool_available_on_undercloud: [docker, podman]

## To get undercloud registry hostname/ip, we have two approaches. Use either one.
1. openstack tripleo container image list

2. find your 'containers-prepare-parameter.yaml' (from overcloud deploy command) and search for 'push_destination'
cat /home/stack/containers-prepare-parameter.yaml | grep push_destination
 - push_destination: "undercloud.ctlplane.ooo.prod1:8787"

Here, 'undercloud.ctlplane.ooo.prod1' is undercloud registry hostname. Use it in our command like following example.

# Command Example:
sudo ./prepare_trilio_images.sh undercloud.ctlplane.ooo.prod1 centos7 <HOTFIX-TAG-VERSION>-tripleo podman

## Verify changes
# For TripleO Train Centos7
$ grep '<HOTFIX-TAG-VERSION>-tripleo' ../environments/trilio_env.yaml
   DockerTrilioDatamoverImage: prod1-undercloud.demo:8787/trilio/tripleo-train-centos7-trilio-datamover:<HOTFIX-TAG-VERSION>-tripleo
   DockerTrilioDmApiImage: prod1-undercloud.demo:8787/trilio/tripleo-train-centos7-trilio-datamover-api:<HOTFIX-TAG-VERSION>-tripleo
   DockerHorizonImage: prod1-undercloud.demo:8787/trilio/tripleo-train-centos7-trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-tripleo

The changes can be verified using the following commands.

## For Centos7 Train

(undercloud) [stack@undercloud redhat-director-scripts/tripleo-train]$ openstack tripleo container image list | grep trilio
| docker://undercloud.ctlplane.localdomain:8787/trilio/tripleo-train-centos7-trilio-datamover:<HOTFIX-TAG-VERSION>-tripleo                       |                        |
| docker://undercloud.ctlplane.localdomain:8787/trilio/tripleo-train-centos7-trilio-datamover-api:<HOTFIX-TAG-VERSION>-tripleo                  |                   |
| docker://undercloud.ctlplane.localdomain:8787/trilio/tripleo-train-centos7-trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-tripleo                  |

5] Configure multi-IP NFS

This section is only required when the multi-IP feature for NFS is required.

This feature allows setting the IP to access the NFS Volume per datamover instead of globally.

On Undercloud node, change the directory

cd triliovault-cfg-scripts/common/

Edit file 'triliovault_nfs_map_input.yml' in the current directory and provide compute host and NFS share/IP map.

Get the compute hostnames from the following command. Check the 'Name' column. Use exact hostnames in 'triliovault_nfs_map_input.yml' file.

Run this command on undercloud by sourcing 'stackrc'.

(undercloud) [stack@ucqa161 ~]$ openstack server list
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+
| ID                                   | Name                          | Status | Networks             | Image          | Flavor  |
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+
| 8c3d04ae-fcdd-431c-afa6-9a50f3cb2c0d | overcloudtrain1-controller-2  | ACTIVE | ctlplane=172.30.5.18 | overcloud-full | control |
| 103dfd3e-d073-4123-9223-b8cf8c7398fe | overcloudtrain1-controller-0  | ACTIVE | ctlplane=172.30.5.11 | overcloud-full | control |
| a3541849-2e9b-4aa0-9fa9-91e7d24f0149 | overcloudtrain1-controller-1  | ACTIVE | ctlplane=172.30.5.25 | overcloud-full | control |
| 74a9f530-0c7b-49c4-9a1f-87e7eeda91c0 | overcloudtrain1-novacompute-0 | ACTIVE | ctlplane=172.30.5.30 | overcloud-full | compute |
| c1664ac3-7d9c-4a36-b375-0e4ee19e93e4 | overcloudtrain1-novacompute-1 | ACTIVE | ctlplane=172.30.5.15 | overcloud-full | compute |
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+

vi triliovault_nfs_map_input.yml

Update pyYAML on the undercloud node only

If pip isn't available please install pip on the undercloud.

## On Python3 env
sudo pip3 install PyYAML==5.1

## On Python2 env
sudo pip install PyYAML==5.1

Expand the map file to create a one-to-one mapping of the compute nodes and the NFS shares.

## On Python3 env
python3 ./generate_nfs_map.py

## On Python2 env
python ./generate_nfs_map.py

The result will be in file - 'triliovault_nfs_map_output.yml'

Validate output map file

Open file 'triliovault_nfs_map_output.yml

vi triliovault_nfs_map_output.yml

available in the current directory and validate that all compute nodes are covered with all the necessary NFS shares.

Append this output map file to 'triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_nfs_map.yaml'

grep ':.*:' triliovault_nfs_map_output.yml >> ../redhat-director-scripts/tripleo-train/environments/trilio_nfs_map.yaml

Validate the changes in file triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_nfs_map.yaml

Include the environment file (trilio_nfs_map.yaml) in overcloud deploy command with '-e' option as shown below.

-e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/environments/trilio_nfs_map.yaml

Ensure that MultiIPNfsEnabled is set to true in trilio_env.yaml file and that NFS is used as the backup target.

6] Fill in Trilio environment details

Fill Trilio details in the file /home/stack/triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/environments/trilio_env.yaml, triliovault environment file is self-explanatory. Fill in details of the backup target, verify image URLs, and other details.

NFS options for Cohesity NFS : nolock,soft,timeo=600,intr,lookupcache=none,nfsvers=3,retrans=10

7] Install Trilio on Overcloud

Use the following heat environment file and roles data file in overcloud deploy command

trilio_env.yaml: This environment file contains Trilio backup target details and Trilio container image locations
roles_data.yaml: This file contains overcloud roles data with Trilio roles added.
Use the correct trilio endpoint map file as per your keystone endpoint configuration. - Instead of tls-endpoints-public-dns.yaml this file, use ‘environments/trilio_env_tls_endpoints_public_dns.yaml’ - Instead of tls-endpoints-public-ip.yaml this file, use ‘environments/trilio_env_tls_endpoints_public_ip.yaml’ - Instead of tls-everywhere-endpoints-dns.yaml this file, use ‘environments/trilio_env_tls_everywhere_dns.yaml’

Deploy command with triliovault environment file looks like the following.

openstack overcloud deploy --templates \
  -e /home/stack/templates/node-info.yaml \
  -e /home/stack/templates/overcloud_images.yaml \
  -e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/environments/trilio_env.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-tls.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/inject-trust-anchor.yaml \
  -e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/environments/trilio_env_tls_endpoints_public_dns.yaml \
  --ntp-server 192.168.1.34 \
  --libvirt-type qemu \
  --log-file overcloud_deploy.log \
  -r /home/stack/templates/roles_data.yaml

Post deployment for multipath enabled environment, log into respective datamover container and add uxsock_timeout with value as 60000 (i.e. 60 sec) in /etc/multipath.conf. Restart datamover container

8] Verify the deployment

If the containers are in restarting state or not listed by the following command then your deployment is not done correctly. Please recheck if you followed the complete documentation.

8.1] On the Controller node

[root@overcloud-controller-0 heat-admin]# podman ps | grep trilio
26fcb9194566  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-tripleo        kolla_start           5 days ago  Up 5 days ago         trilio_dmapi
094971d0f5a9  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-tripleo       kolla_start           5 days ago  Up 5 days ago         horizon

Verify the haproxy configuration under:

/var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg

8.2] On Compute node

Make sure Trilio datamover container is in running state and no other Trilio container is deployed on compute nodes.

[root@overcloud-novacompute-0 heat-admin]# podman ps | grep trilio
b1840444cc59  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-tripleo                    kolla_start           5 days ago  Up 5 days ago         trilio_datamover

8.3] On the node with Horizon service

Make sure horizon container is in running state. Please note that 'Horizon' container is replaced with Trilio Horizon container. This container will have the latest OpenStack horizon + Trilio's horizon plugin.

[root@overcloud-controller-0 heat-admin]# podman ps | grep horizon
094971d0f5a9  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-tripleo       kolla_start           5 days ago  Up 5 days ago         horizon

10] Troubleshooting for overcloud deployment failures

Trilio components will be deployed using puppet scripts.

openstack stack failures list overcloud
heat stack-list --show-nested -f "status=FAILED"
heat resource-list --nested-depth 5 overcloud | grep FAILED

##=> If trilio datamover api containers does not start well or in restarting state, use following logs to debug.
docker logs trilio_dmapi
tailf /var/log/containers/trilio-datamover-api/dmapi.log

##=> If trilio datamover containers does not start well or in restarting state, use following logs to debug.
docker logs trilio_datamover
tailf /var/log/containers/trilio-datamover/tvault-contego.log

Apply the Trilio license

After the Trilio VM has been configured and all components are installed can the license be applied.

The license can be applied either through the admin-tab in Horizon or the CLI

Apply license through Horizon

To apply the license through Horizon follow these steps:

Login to Horizon using admin user.
Click on Admin Tab.
Navigate to Backups-Admin
Navigate to Trilio
Navigate to License
Click "Update License"
Click "Choose File"
choose license-file on client system
click "Apply"

Apply license through CLI

Advanced Ceph configurations

Ceph is the most common OpenSource solution to provide block storage through OpenStack Cinder.

Ceph is a very flexible solution. The possibilities of Ceph require additional steps to the Trilio solution.

Upgrade OpenStack

Trilio Appliance Administration Guide

User Guide

Admin Guide

Troubleshooting

API GUIDE

Installing on RHOSP

The is the supported and recommended method to deploy and maintain any RHOSP installation.

Trilio is integrating natively into the RHOSP Director. Manual deployment methods are not supported for RHOSP.

1. Prepare for deployment

1.1] Select 'backup target' type

Backup target storage is used to store backup images taken by Trilio and details needed for configuration:

Following backup target types are supported by Trilio

a) NFS

Need NFS share path

b) Amazon S3

- S3 Access Key - Secret Key - Region - Bucket name

c) Other S3 compatible storage (Like, Ceph based S3)

- S3 Access Key - Secret Key - Region - Endpoint URL (Valid for S3 other than Amazon S3) - Bucket name

1.2] Clone triliovault-cfg-scripts repository

All commands need to be run as user 'stack' on undercloud node

The following command clones the triliovault-cfg-scripts github repository.

cd /home/stack
git clone -b 4.3.2 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/

Next access the Red Hat Director scripts according to the used RHOSP version.

RHOSP 13

cd triliovault-cfg-scripts/redhat-director-scripts/rhosp13/

RHOSP 16.1

cd triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/

RHOSP 16.2

cd triliovault-cfg-scripts/redhat-director-scripts/rhosp16.2/

RHOSP 17.0

cd triliovault-cfg-scripts/redhat-director-scripts/rhosp17.0/

The remaining documentation will use the following path for examples:

cd triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/

1.3] If backup target type is 'Ceph based S3' with SSL:

cp s3-cert.pem /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/puppet/trilio/files

2] Upload trilio puppet module

The following commands upload the Trilio puppet module to the overcloud registry. The actual upload happens upon the next deployment.

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/scripts/
./upload_puppet_module.sh

## Output of above command looks like following for RHOSP13, RHOSP16.1 and RHOSP16.2
Creating tarball...
Tarball created.
Creating heat environment file: /home/stack/.tripleo/environments/puppet-modules-url.yaml
Uploading file to swift: /tmp/puppet-modules-8Qjya2X/puppet-modules.tar.gz
+-----------------------+---------------------+----------------------------------+
| object                | container           | etag                             |
+-----------------------+---------------------+----------------------------------+
| puppet-modules.tar.gz | overcloud-artifacts | 368951f6a4d39cfe53b5781797b133ad |
+-----------------------+---------------------+----------------------------------+

## Above command creates following file.
ls -ll /home/stack/.tripleo/environments/puppet-modules-url.yaml

## Command is same for RHOSP17.0 but the command output and file content would be different

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp17.0/scripts/
./upload_puppet_module.sh

## Output of above command looks like following for RHOSP17.0
Creating tarball...
Tarball created.
renamed '/tmp/puppet-modules-P3duCg9/puppet-modules.tar.gz' -> '/var/lib/tripleo/artifacts/overcloud-artifacts/puppet-modules.tar.gz'
Creating heat environment file: /home/stack/.tripleo/environments/puppet-modules-url.yaml

## Above command creates following file.
ls -ll /home/stack/.tripleo/environments/puppet-modules-url.yaml

Trilio puppet module is uploaded to overcloud as a swift deploy artifact with heat resource name 'DeployArtifactURLs'. If you check trilio's puppet module artifact file it looks like following.

## For RHOSP13, RHOSP16.1 and RHOSP16.2
(undercloud) [stack@ucloud161 ~]$ cat /home/stack/.tripleo/environments/puppet-modules-url.yaml
# Heat environment to deploy artifacts via Swift Temp URL(s)
parameter_defaults:
    DeployArtifactURLs:
    - 'http://172.25.0.103:8080/v1/AUTH_46ba596219d143c8b076e9fcc4139fed/overcloud-artifacts/puppet-modules.tar.gz?temp_url_sig=c3972b7ce75226c278ab3fa8237d31cc1f2115bd&temp_url_expires=1646738377'

## For RHOSP17.0
(undercloud) [stack@undercloud17-3 scripts]$ cat /home/stack/.tripleo/environments/puppet-modules-url.yaml
parameter_defaults:
  DeployArtifactFILEs:
  - /var/lib/tripleo/artifacts/overcloud-artifacts/puppet-modules.tar.gz

Note: If your overcloud deploy command using any other deploy artifact through a environment file, then you need to merge trilio deploy artifact url and your url in single file.

How to check if your overcloud deploy environment files using deploy artifacts? You need to check string 'DeployArtifactURLs' in your environment files (only those mentioned in overcloud deploy command with -e option). If you find it any such environment file that is mentioned in overcloud dpeloy command with '-e' option then your deploy command is using deploy artifact.
In that case you need to merge all deploy artifacts in single file. Refer following steps.

(undercloud) [stack@ucloud161 ~]$ cat /home/stack/.tripleo/environments/puppet-modules-url.yaml | grep http >> /home/stack/templates/user-artifacts.yaml
(undercloud) [stack@ucloud161 ~]$ cat /home/stack/templates/user-artifacts.yaml
# Heat environment to deploy artifacts via Swift Temp URL(s)
parameter_defaults:
    DeployArtifactURLs:
    - 'http://172.25.0.103:8080/v1/AUTH_57ba596219d143c8b076e9fcc4139f3g/overcloud-artifacts/some-artifact.tar.gz?temp_url_sig=dc972b7ce75226c278ab3fa8237d31cc1f2115sc&temp_url_expires=3446738365'
    - 'http://172.25.0.103:8080/v1/AUTH_46ba596219d143c8b076e9fcc4139fed/overcloud-artifacts/puppet-modules.tar.gz?temp_url_sig=c3972b7ce75226c278ab3fa8237d31cc1f2115bd&temp_url_expires=1646738377'

3] Update overcloud roles data file to include Trilio services

Trilio contains multiple services. Add these services to your roles_data.yaml.

In the case of uncustomized roles_data.yaml can the default file be found on the undercloud at:

/usr/share/openstack-tripleo-heat-templates/roles_data.yaml

Add the following services to the roles_data.yaml

All commands need to be run as user 'stack'

3.1] Add Trilio Datamover Api Service to role data file

Add the following line to the identified role:

'OS::TripleO::Services::TrilioDatamoverApi'

3.2] Add Trilio Datamover Service to role data file

Add the following line to the identified role:

'OS::TripleO::Services::TrilioDatamover'

3.3] Add Trilio Horizon Service role data file

OS::TripleO::Services::TrilioHorizon

4] Prepare Trilio container images

All commands need to be run as user 'stack'

Trilio containers are pushed to 'RedHat Container Registry'. Registry URL: 'registry.connect.redhat.com'. Container pull URLs are given below.

Please note that using the hotfix containers requires that the Trilio Appliance is getting upgraded to the desired hotfix level as well.

Refer to the word <HOTFIX-TAG-VERSION> as 4.3.2 in the below sections

RHOSP 13

Trilio Datamover container:        registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp13
Trilio Datamover Api Container:   registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp13
Trilio horizon plugin:            registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp13

RHOSP 16.1

Trilio Datamover container:        registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1
Trilio Datamover Api Container:   registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1
Trilio horizon plugin:            registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1

RHOSP 16.2

Trilio Datamover container:        registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.2
Trilio Datamover Api Container:   registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.2
Trilio horizon plugin:            registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.2

RHOSP 17.0

Trilio Datamover container:        registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp17.0
Trilio Datamover Api Container:   registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp17.0
Trilio horizon plugin:            registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp17.0

There are three registry methods available in RedHat Openstack Platform.

Remote Registry
Local Registry
Satellite Server

4.1] Remote Registry

Follow this section when 'Remote Registry' is used.

Make sure other openstack service images are also using the same method to pull container images. If it's not the case you can not use this method.
Populate 'containers-prepare-parameter.yaml' with content like following. Important parameters are 'push_destination: false', ContainerImageRegistryLogin: true and registry credentials. Trilio container images are published to registry 'registry.connect.redhat.com'. Credentials of registry 'registry.redhat.io' will work for 'registry.connect.redhat.com' registry too.

Note: This file -'containers-prepare-parameter.yaml'

File Name: containers-prepare-parameter.yaml

parameter_defaults:
  ContainerImagePrepare:
  - push_destination: false
    set:
      namespace: registry.redhat.io/...
      ...
  ...
  ContainerImageRegistryCredentials:
    registry.redhat.io:
      myuser: 'p@55w0rd!'
    registry.connect.redhat.com:
      myuser: 'p@55w0rd!'
  ContainerImageRegistryLogin: true

Redhat document for remote registry method: \

Note: File 'containers-prepare-parameter.yaml' gets created as output of command 'openstack tripleo container image prepare'. Refer above document by RedHat

3. Make sure you have network connectivity to above registries from all overcloud nodes. Otherwise image pull operation will fail.

4. User need to manually populate the trilio_env.yaml file with Trilio container image URLs as given below:

trilio_env.yaml file path:

cd triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/

# For RHOSP13
$ grep '<HOTFIX-TAG-VERSION>-rhosp13' trilio_env.yaml
   DockerTrilioDatamoverImage: registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp13
   DockerTrilioDmApiImage: registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp13
   DockerHorizonImage: registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp13

# For RHOSP16.1
$ grep '<HOTFIX-TAG-VERSION>-rhosp16.1' trilio_env.yaml
   DockerTrilioDatamoverImage: registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1
   DockerTrilioDmApiImage: registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1
   ContainerHorizonImage: registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1

# For RHOSP16.2
$ grep '<HOTFIX-TAG-VERSION>-rhosp16.2' trilio_env.yaml
   DockerTrilioDatamoverImage: registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.2
   DockerTrilioDmApiImage: registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.2
   ContainerHorizonImage: registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.2

# For RHOSP17.0
$ grep '<HOTFIX-TAG-VERSION>-rhosp17.0' trilio_env.yaml
   DockerTrilioDatamoverImage: registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp17.0
   DockerTrilioDmApiImage: registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp17.0
   ContainerHorizonImage: registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp17.0

At this step, you have configured Trilio image urls in the necessary environment file.

4.2] Local Registry

Follow this section when 'local registry' is used on the undercloud.

RHOSP13

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp13/scripts/

./prepare_trilio_images.sh <undercloud_ip/hostname> <HOTFIX-TAG-VERSION>-rhosp13

## Verify changes
$ grep '<HOTFIX-TAG-VERSION>-rhosp13' ../environments/trilio_env.yaml
   DockerTrilioDatamoverImage: 172.25.2.2:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp13
   DockerTrilioDmApiImage: 172.25.2.2:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp13
   DockerHorizonImage: 172.25.2.2:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp13

$ docker image list | grep <HOTFIX-TAG-VERSION>-rhosp13
172.30.5.101:8787/trilio/trilio-datamover                  <HOTFIX-TAG-VERSION>-rhosp13        f2dfb36bb176        8 weeks ago         3.61 GB
registry.connect.redhat.com/trilio/trilio-datamover        <HOTFIX-TAG-VERSION>-rhosp13        f2dfb36bb176        8 weeks ago         3.61 GB
172.30.5.101:8787/trilio/trilio-datamover-api              <HOTFIX-TAG-VERSION>-rhosp13        5d62f572a00c        8 weeks ago         2.24 GB
registry.connect.redhat.com/trilio/trilio-datamover-api    <HOTFIX-TAG-VERSION>-rhosp13        5d62f572a00c        8 weeks ago         2.24 GB
registry.connect.redhat.com/trilio/trilio-horizon-plugin   <HOTFIX-TAG-VERSION>-rhosp13        27c4de28e5ae        2 months ago        2.27 GB
172.30.5.101:8787/trilio/trilio-horizon-plugin             <HOTFIX-TAG-VERSION>-rhosp13        27c4de28e5ae        2 months ago        2.27 GB

RHOSP 16.1

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/scripts/

sudo ./prepare_trilio_images.sh <undercloud_ip/hostname> <HOTFIX-TAG-VERSION>-rhosp16.1

## Verify changes
$ grep '<HOTFIX-TAG-VERSION>-rhosp16.1' ../environments/trilio_env.yaml
   DockerTrilioDatamoverImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1
   DockerTrilioDmApiImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1
   ContainerHorizonImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1

$ openstack tripleo container image list | grep <HOTFIX-TAG-VERSION>-rhosp16.1
| docker://tlsundercloud.ctlplane.trilio.local:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1 |
| docker://tlsundercloud.ctlplane.trilio.local:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1      |
| docker://tlsundercloud.ctlplane.trilio.local:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1  |
-----------------------------------------------------------------------------------------------------

RHOSP16.2

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.2/scripts/

sudo ./prepare_trilio_images.sh <undercloud_ip/hostname> <HOTFIX-TAG-VERSION>-rhosp16.2

## Verify changes
$ grep '<HOTFIX-TAG-VERSION>-rhosp16.2' ../environments/trilio_env.yaml
   DockerTrilioDatamoverImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.2
   DockerTrilioDmApiImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.2
   ContainerHorizonImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.2

$ openstack tripleo container image list | grep <HOTFIX-TAG-VERSION>-rhosp16.2
| docker://tlsundercloud.ctlplane.trilio.local:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.2 |
| docker://tlsundercloud.ctlplane.trilio.local:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.2      |
| docker://tlsundercloud.ctlplane.trilio.local:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.2  |

RHOSP17.0

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp17.0/scripts/

sudo ./prepare_trilio_images.sh <undercloud_ip/hostname> <HOTFIX-TAG-VERSION>-rhosp17.0

## Verify changes
$ grep '<HOTFIX-TAG-VERSION>-rhosp17.0' ../environments/trilio_env.yaml
   DockerTrilioDatamoverImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp17.0
   DockerTrilioDmApiImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp17.0
   ContainerHorizonImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp17.0

$ openstack tripleo container image list | grep <HOTFIX-TAG-VERSION>-rhosp17.0
| docker://tlsundercloud.ctlplane.trilio.local:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp17.0 |
| docker://tlsundercloud.ctlplane.trilio.local:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp17.0      |
| docker://tlsundercloud.ctlplane.trilio.local:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp17.0  |

At this step, you have downloaded triliovault container images and configured triliovault image urls in necessary environment file.

4.3] Red Hat Satellite Server

Follow this section when a Satellite Server is used for the container registry.

Pull the Trilio containers on the Red Hat Satellite using the given

Populate the trilio_env.yaml with container urls.

RHOSP 13

$ grep '<HOTFIX-TAG-VERSION>-rhosp13' ../environments/trilio_env.yaml
   DockerTrilioDatamoverImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp13
   DockerTrilioDmApiImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp13
   DockerHorizonImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp13

RHOSP 16.1

$ grep '<HOTFIX-TAG-VERSION>-rhosp16.1' trilio_env.yaml
   DockerTrilioDatamoverImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1
   DockerTrilioDmApiImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1
   ContainerHorizonImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1

RHOSP16.2

$ grep '<HOTFIX-TAG-VERSION>-rhosp16.2' trilio_env.yaml
   DockerTrilioDatamoverImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.2
   DockerTrilioDmApiImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.2
   ContainerHorizonImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.2

RHOSP17.0

$ grep '<HOTFIX-TAG-VERSION>-rhosp17.0' trilio_env.yaml
   DockerTrilioDatamoverImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp17.0
   DockerTrilioDmApiImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp17.0
   ContainerHorizonImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp17.0

At this step, you have downloaded triliovault container images into RedHat sattelite server and configured triliovault image urls in necessary environment file.

5] Configure multi-IP NFS

This section is only required when the multi-IP feature for NFS is required.

This feature allows to set the IP to access the NFS Volume per datamover instead of globally.

On Undercloud node, change directory

cd triliovault-cfg-scripts/common/

Edit file 'triliovault_nfs_map_input.yml' in the current directory and provide compute host and NFS share/ip map.

Get the compute hostnames from the following command. Check ‘Name' column. Use exact hostnames in 'triliovault_nfs_map_input.yml' file.

Run this command on undercloud by sourcing 'stackrc'.

(undercloud) [stack@ucqa161 ~]$ openstack server list
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+
| ID                                   | Name                          | Status | Networks             | Image          | Flavor  |
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+
| 8c3d04ae-fcdd-431c-afa6-9a50f3cb2c0d | overcloudtrain1-controller-2  | ACTIVE | ctlplane=172.30.5.18 | overcloud-full | control |
| 103dfd3e-d073-4123-9223-b8cf8c7398fe | overcloudtrain1-controller-0  | ACTIVE | ctlplane=172.30.5.11 | overcloud-full | control |
| a3541849-2e9b-4aa0-9fa9-91e7d24f0149 | overcloudtrain1-controller-1  | ACTIVE | ctlplane=172.30.5.25 | overcloud-full | control |
| 74a9f530-0c7b-49c4-9a1f-87e7eeda91c0 | overcloudtrain1-novacompute-0 | ACTIVE | ctlplane=172.30.5.30 | overcloud-full | compute |
| c1664ac3-7d9c-4a36-b375-0e4ee19e93e4 | overcloudtrain1-novacompute-1 | ACTIVE | ctlplane=172.30.5.15 | overcloud-full | compute |
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+

Edit input map file and fill all the details. Refer to the for details about the structure.

vi triliovault_nfs_map_input.yml

Update pyyaml on the undercloud node only

If pip isn't available please install pip on the undercloud.

## On Python3 env
sudo pip3 install PyYAML==5.1 3

## On Python2 env
sudo pip install PyYAML==5.1

Expand the map file to create a one-to-one mapping of the compute nodes and the nfs shares.

## On Python3 env
python3 ./generate_nfs_map.py

## On Python2 env
python ./generate_nfs_map.py

The result will be in file - 'triliovault_nfs_map_output.yml'

Validate output map file

Open file 'triliovault_nfs_map_output.yml

vi triliovault_nfs_map_output.yml

available in the current directory and validate that all compute nodes are covered with all the necessary NFS shares.

Append this output map file to 'triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_nfs_map.yaml'

grep ':.*:' triliovault_nfs_map_output.yml >> ../redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_nfs_map.yaml

Validate the changes in file 'triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_nfs_map.yaml'

Include the environment file (trilio_nfs_map.yaml) in overcloud deploy command with '-e' option as shown below.

-e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_nfs_map.yaml

Ensure that MultiIPNfsEnabled is set to true in trilio_env.yaml file and that nfs is used as backup target.

6] Provide environment details in trilio-env.yaml

The following information are required additionally:

Network for the datamover api
datamover password
Backup target type {nfs/s3}
In case of NFS
- list of NFS Shares
- NFS options
- MultiIPNfsEnabled

NFS options for Cohesity NFS : nolock,soft,timeo=600,intr,lookupcache=none,nfsvers=3,retrans=10

In case of S3
- S3 type {amazon_s3/ceph_s3}
- S3 Access key
- S3 Secret key
- S3 Region name
- S3 Bucket
- S3 Endpoint URL
- S3 Signature Version
- S3 Auth Version
- S3 SSL Enabled {true/false}
- S3 SSL Cert

Use ceph_s3 for any non-aws S3 backup targets.

resource_registry:
  OS::TripleO::Services::TrilioDatamover: ../services/trilio-datamover.yaml
  OS::TripleO::Services::TrilioDatamoverApi: ../services/trilio-datamover-api.yaml
  OS::TripleO::Services::TrilioHorizon: ../services/trilio-horizon.yaml

  # NOTE: If there are addition customizations to the endpoint map (e.g. for
  # other integratiosn), this will need to be regenerated.
  OS::TripleO::EndpointMap: endpoint_map.yaml

parameter_defaults:

   ## Enable Trilio's quota functionality on horizon
   ExtraConfig:
     horizon::customization_module: 'dashboards.overrides'

   ## Define network map for trilio datamover api service
   ServiceNetMap:
       TrilioDatamoverApiNetwork: internal_api

   ## Trilio Datamover Password for keystone and database
   TrilioDatamoverPassword: "test1234"

   ## Trilio container pull urls
   DockerTrilioDatamoverImage: devundercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1
   DockerTrilioDmApiImage: devundercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1

   ## If you do not want Trilio's horizon plugin to replace your horizon container, just comment following line.
   ContainerHorizonImage: devundercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1

   ## Backup target type nfs/s3, used to store snapshots taken by triliovault
   BackupTargetType: 'nfs'

   ## If backup target NFS share support multiple IPs and you want to use those IPs(more than one) then
   ## set this parameter to True. Otherwise keep it False.
   MultiIPNfsEnabled: False

   ## For backup target 'nfs'
   NfsShares: '192.168.122.101:/opt/tvault'
   NfsOptions: 'nolock,soft,timeo=180,intr,lookupcache=none'

   ## For backup target 's3'
   ## S3 type: amazon_s3/ceph_s3
   S3Type: 'amazon_s3'

   ## S3 access key
   S3AccessKey: ''

   ## S3 secret key
   S3SecretKey: ''

   ## S3 region, if your s3 does not have any region, just keep the parameter as it is
   S3RegionName: ''

   ## S3 bucket name
   S3Bucket: ''

   ## S3 endpoint url, not required for Amazon S3, keep it as it is
   S3EndpointUrl: ''

   ## S3 signature version
   S3SignatureVersion: 'default'

   ## S3 Auth version
   S3AuthVersion: 'DEFAULT'

   ## If S3 backend is not Amazon S3 and SSL is enabled on S3 endpoint url then change it to 'True', otherwise keep it as 'False'
   S3SslEnabled: False

   ## If S3 backend is not Amazon S3 and SSL is enabled on S3 endpoint URL and SSL certificates are self signed, then
   ## user need to set this parameter value to: '/etc/tvault-contego/s3-cert.pem', otherwise keep it's value as empty string.
   S3SslCert: ''

   ## Configure 'dmapi_workers' parameter of '/etc/dmapi/dmapi.conf' file
   ## This parameter value used to spawn the number of dmapi processes to handle the incoming api requests.
   ## If your dmapi node has ‘n' cpu cores, It is recommended, to set this parameter to '4*n’.
   ## If dmapi_workers field is not present in config file. The Default value will be equals to number of cores present on the node
   DmApiWorkers: 16

   ## Don't edit following parameter
   EnablePackageInstall: True


   ## Load 'rbd' kernel module on all compute nodes
   ComputeParameters:
     ExtraKernelModules:
       rbd: {}

7] Advanced Settings/Configuration

7.1] Haproxy customized configuration for Trilio dmapi service __

Following is the haproxy conf file location on haproxy nodes of the overcloud. Trilio datamover api service haproxy configuration gets added to this file.

/var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg

Trilio datamover haproxy default configuration from the above file looks as follows:

listen trilio_datamover_api
  bind 172.25.0.107:13784 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 172.25.0.107:8784 transparent
  balance roundrobin
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  http-request set-header X-Forwarded-Port %[dst_port]
  maxconn 50000
  option httpchk
  option httplog
  retries 5
  timeout check 10m
  timeout client 10m
  timeout connect 10m
  timeout http-request 10m
  timeout queue 10m
  timeout server 10m
  server overcloud-controller-0.internalapi.localdomain 172.25.0.106:8784 check fall 5 inter 2000 rise 2

The user can change the following configuration parameter values.

retries 5
timeout http-request 10m
timeout queue 10m
timeout connect 10m
timeout client 10m
timeout server 10m
timeout check 10m
balance roundrobin
maxconn 50000

For RHOSP13

/home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp13/services/trilio-datamover-api.yaml

For RHOSP16.1

/home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/services/trilio-datamover-api.yaml

For RHOSP16.2

/home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.2/services/trilio-datamover-api.yaml

For RHOSP17.0

/home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp17.0/services/trilio-datamover-api.yaml

ii) Search the following entries and edit as required

          tripleo::haproxy::trilio_datamover_api::options:
             'retries': '5'
             'maxconn': '50000'
             'balance': 'roundrobin'
             'timeout http-request': '10m'
             'timeout queue': '10m'
             'timeout connect': '10m'
             'timeout client': '10m'
             'timeout server': '10m'
             'timeout check': '10m'

iii) Save the changes.

7.2] Configure Custom Volume/Directory Mounts for the Trilio Datamover Service

If the user wants to add one or more extra volume/directory mounts to the Trilio Datamover Service container, they can use the following heat environment file. A variable named 'TrilioDatamoverOptVolumes' is available in this file.

triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE>/environments/trilio_datamover_opt_volumes.yaml

This variable 'TrilioDatamoverOptVolumes' accepts list of volume/bind mounts.
User needs to edit this file and add their volume mounts in below format.
For example:
In this volume mount "/mnt/dir2:/var/dir2", "/mnt/dir2" is a diretcory available on compute host and "/var/dir2" is the mount point inside datamover container.

parameter_defaults:
  TrilioDatamoverOptVolumes:
    - /opt/dir1:/opt/dir1
    - /mnt/dir2:/var/dir2

Next, User needs to pass this file to overcloud deploy command with '-e' option Like below.

openstack overcloud deploy --templates \
-e <> \
.
.
-e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/environments/trilio_datamover_opt_volumes.yaml

8] Deploy overcloud with trilio environment

Use the following heat environment file and roles data file in overcloud deploy command:

trilio_env.yaml
roles_data.yaml
Use correct Trilio endpoint map file as per available Keystone endpoint configuration
1. Instead of tls-endpoints-public-dns.yaml file, use environments/trilio_env_tls_endpoints_public_dns.yaml
2. Instead of tls-endpoints-public-ip.yaml file, useenvironments/trilio_env_tls_endpoints_public_ip.yaml
3. Instead of tls-everywhere-endpoints-dns.yaml file, useenvironments/trilio_env_tls_everywhere_dns.yaml
If activated use the correct trilio_nfs_map.yaml file

To include new environment files use '-e' option and for roles data file use '-r' option. An example overcloud deploy command is shown below:

openstack overcloud deploy --templates \
  -e /home/stack/templates/node-info.yaml \
  -e /home/stack/templates/overcloud_images.yaml \
  -e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/environments/trilio_env.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-tls.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/inject-trust-anchor.yaml \
  -e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/environments/trilio_env_tls_endpoints_public_dns.yaml \
  -e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/environments/trilio_nfs_map.yaml \
  --ntp-server 192.168.1.34 \
  --libvirt-type qemu \
  --log-file overcloud_deploy.log \
  -r /usr/share/openstack-tripleo-heat-templates/roles_data.yaml

Post deployment for multipath enabled environment, log into respective datamover container and add uxsock_timeout with value as 60000 (i.e. 60 sec) in /etc/multipath.conf. Restart datamover container

9] Verify deployment

If the containers are in restarting state or not listed by the following command then your deployment is not done correctly. Please recheck if you followed the complete documentation.

9.1] On Controller node

[root@overcloud-controller-0 heat-admin]# podman ps | grep trilio
26fcb9194566  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.2        kolla_start           5 days ago  Up 5 days ago         trilio_dmapi
094971d0f5a9  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.2       kolla_start           5 days ago  Up 5 days ago         horizon

Verify the haproxy configuration under:

/var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg

9.2] On Compute node

Make sure Trilio datamover container is in running state and no other Trilio container is deployed on compute nodes.

[root@overcloud-novacompute-0 heat-admin]# podman ps | grep trilio
b1840444cc59  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.2                    kolla_start           5 days ago  Up 5 days ago         trilio_datamover

9.3] On the node with Horizon service

[root@overcloud-controller-0 heat-admin]# podman ps | grep horizon
094971d0f5a9  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.2       kolla_start           5 days ago  Up 5 days ago         horizon

If Trilio Horizon container is in restarted state on RHOSP 16.1.8/RHSOP 16.2.4 then use below workaroud

## Either of the below workarounds should be performed on all the controller nodes where issue occurs for horizon pod.

option-1: Restart the memcached service on controller using systemctl (command: systemctl restart tripleo_memcached.service)

option-2: Restart the memcached pod (command: podman restart memcached)

10] Troubleshooting for overcloud deployment failures

Trilio components will be deployed using puppet scripts.

oIn case of the overcloud deployment failing do the following command provide the list of errors. The following document also provides valuable insights:

openstack stack failures list overcloud
heat stack-list --show-nested -f "status=FAILED"
heat resource-list --nested-depth 5 overcloud | grep FAILED

=> If trilio datamover api containers does not start well or in restarting state, use following logs to debug.


docker logs trilio_dmapi

tailf /var/log/containers/trilio-datamover-api/dmapi.log



=> If trilio datamover containers does not start well or in restarting state, use following logs to debug.


docker logs trilio_datamover

tailf /var/log/containers/trilio-datamover/tvault-contego.log