Trilio release 4.0 introduces new features and capabilities including

• Openstack Train support

• Trilio Openstack Quota integration

• Openstack CLI integration

• CLI restore of networks only

• CLI restore of security groups only

• Disk integrity check after Snapshots

• Performance Enhancements for big environments

• Enablement of rolling upgrades

Trilio 4.0.92 is the GA release of Trilio 4.0 Trilio 4.0.115 is the first patch release of Trilio 4.0.

Openstack Train support

Trilio 4.0 continues to enable Openstack versions and Distributions, allowing Trilio customers to stay up to date with Openstack releases.

Trilio 4.0 introduces full support for Openstack Train over the four supported Distributions. In addition, does Trilio 4.0 of course support longterm releases, giving Openstack users of those releases the possibility to use the latest Trilio functionalities.

Trilio Openstack Quota integration

Trilio for Openstack integrates natively with Openstack and we are happy to announce, that Trilio made another important step.

With the possibility to create Project quotas for Workloads, VMs and Storage usage can Openstack administrators now clearly control how Trilio is used by any tenant.

Combining the Quota feature with the already available policies enables Openstack administrators to define exactly what an Openstack Project can do or can't do with Trilio.

Openstack CLI integration

Trilio has been using its very own Python based CLI client, the workloadmgr client.

This client has followed the Openstack standards for CLI clients and is now again following the established standards, which integrated all Python Openstack clients into a single Openstack Python CLI client.

Trilio 4.0 enables the usage of the Openstack client for the most common commands required during the daily work with Trilio.

CLI restore of networks and security groups only

Trilio restore possibilities allowed to restore any VM and its attached resources as required. Still it was always necessary to restore the complete VM to get the networks or security groups tied to that VM or backup.

Trilio 4.0 introduces the possibility to restore only the networks or security groups directly from the CLI.

Disk integrity check after a backup

Trilio always protected the integrity of any Snapshots. However, when Trilio identified a failure in the integrity did the backup job fail and not only was the Snapshot with a failed integrity lost but the following backups as well.

To overcome this limitation did Trilio enhance the already available disk integrity check and implemented it at the end of each Snapshot. When it is detected that a Snapshot disk integrity is not provided, will the next Snapshot automatically be a full Snapshot.

That way ensuring that future backups are again having a reliable disk integrity.

Performance enhancements for big environments

With the success of Trilio around the world was it only a question of time until Trilio would be tested against a real big environment with thousands of API calls per second.

We are happy to say that our solution did keep up to the task, but it still showed potential to improve even more. And that improvement comes with a small change on the Trilio Appliance, which allows the load-balancing of incoming API calls against a complete Trilio cluster, instead of a master node.

Enablement of rolling upgrades and updates

Trilio has been part of some Openstack environments for years, going through the process of updating and upgrading together with the environment and stand alone.

This upgrade process so far was always a complete uninstallation and reinstallation of the Trilio solution, which was very time consuming and frustrating for Openstack Administrators.

Trilio 4.0 changed the way the Trilio Appliance is build to allow to update and upgrade of the complete Trilio solution.

The exact process will be announced together with the first patch release of Trilio 4.0.

Known Issues

This release contains the following known issues which are tracked for a future update.

Workloadmgr Quota feature does not support RHOSP13 & Canonical Queens Horizon

The workloadmgr Quota feature is still fully supported through CLI.

[Openstack Ansible] snapshot mount failed with error 'Permission denied'

Observation:

• It has been observed that on non-kolla, non-rhosp setups, such as openstack ansible, nova user id is not same as we consider as default(162).

• Another observation is that, the id which is assigned to nova, was conflicting with id of system user in TVM, this created a situation where we had to redeploy Openstack.

Workaround:

• Update permissions of /var/triliovault-mounts to 755

Snapshots are getting stuck when wlm-workloads service stops during a backup process

Observation:

• When wlm-workloads service gets stopped on the node with an active snapshot process does the process get stuck without erroring.

Workaround:

• Restart wlm-workloads service on Trilio cluster, snapshot will throw error.

[Volume Backup Exclude] Excluded Ceph Volume after restore not mountable or formatable

Observation:

• VM Volumes stored on Ceph are successfully excluded from backup if desired

• Restore does create empty Ceph Volume

• created empty Ceph Volume is not attachable or formatable

Restored VMs have blank metadata config_drive attached

Observation:

• For every restore will the metadata config_drive be set as blank value

• No impact on restored VMs known

Workaround

• delete metadata config_drive

• or set desired value

Scheduler trust not getting imported together with workload

Observation:

• Workload gets imported successfully

• original User and Project still exist (ownership stays same)

• All schedulers trusts are defined as disabled

Workaround:

• Edit any (just 1) workload, modify the scheduler policy and submit

• trust will be recreated

TVM reconfig fails when adding new Trilio VM to the cluster

Observation:

• TVault re-configuration while adding nodes to existing TVM cluster fails at "Configuring Trilio Cluster"

• Reason is that the prev mysql password was not working and mysql root access has be reset.

Workaround:

• remove /root/.my.cnf file on already configured TVM and reconfigure it

Database does not sync after Trilio cluster gets new nodes

Observation:

• After TVault re-configuration post addition of 2 more nodes to existing TVM cluster ("import workloads" was not seleted), the databases do not sync against already existing TVM.

• It is expected that while adding the 2 new nodes, the db on node1 should get synced up with 2 new nodes and the existing workloads should be available post the reconfig on the new 3 node TVM cluster.

Workaround:

• Run workload import from CLI

Workload scheduler gets disabled after primary node goes down

Observation:

• Deactivating the primary node of a TVM cluster deactivates scheduler of associated workloads.

Workaround:

• restart wlm-cron service on new primary node

[exclude_boot_disk] Data on boot disk gets backed up despite exclusion

Observation:

• VM was set with metadata exclude_boot_disk_from_backup set to true

• Restored instance showed, that data was backed up and restored

Post workload reassignment, scheduler trust gets disabled even if Established before reassignment

Observation:

• Workload with scheduler trust established gets reassigned to new project

• scheduler trust is getting disabled

Workaround:

• Edit the workload to enable scheduler

Additional parameters required in rc file to run wlm commands from openstack cli

Observation:

• using workloadmgr commands with Openstack CLI still requires OS_TENANT_ID and OS_USER_DOMAIN_ID

Workaround:

• edit rc file and add required fields

It is recommended to think about the following elements prior to the installation of Trilio for Openstack.

Tenant Quotas

Trilio uses Cinder snapshots for calculating full and incremental backups. For full backups, Trilio creates Cinder snapshots for all the volumes in the backup job. It then leaves these Cinder snapshots behind for calculating the incremental backup image during next backup. During an incremental backup operation it creates new Cinder snapshots, calculates the changed blocks between the new snapshots and the old snapshots that were left behind during full/previous backups. It then deletes the old snapshots but leaves the newly created snapshots behind. So, it is important that each tenant who is availing Trilio backup functionality has sufficient Cinder snapshot quotas to accommodate these additional snapshots. The guideline is to add 2 snapshots for every volume that is added to backups to volume snapshot quotas for that tenant. You may also increase the volume quotas for the tenant by the same amount because Trilio briefly creates a volume from snapshot to read data from the snapshot for backup purposes. During a restore process, Trilio creates additional instances and Cinder volumes. To accommodate restore operations, a tenant should have sufficient quota for Nova instances and Cinder volumes. Otherwise restore operations will result in failures.

AWS S3 eventual consistency¶

AWS S3 object consistency model includes:

1. Read-after-write

2. Read-after-update

3. Read-after-delete

Each of them describes how an object will reach its consistent state after an object is created/updated or deleted. None of them provides strong consistency and there is a lag time for an object to reach the consistent state. Though Trilio employed mechanisms to work around the limitations of eventual consistency of AWS S3, when an object reach its consistency state is not deterministic. There is no official statement from AWS on how long it takes for an object to reach consistent state. However read-after-write has a shorter time to reach consistency compared to other IO patterns. Our solution is designed to maximize read-after-write IO pattern. The time in which an object reaches eventual consistency also depends on the AWS region. For example, aws-standard region does not have strong consistency model compared to us-east or us-west. We suggest to use these regions when creating s3 buckets for Trilio. Though read-after-update IO pattern is hard to avoid completely, we employed ample delays in accessing objects to accommodate larger durations for objects to get into consistent state. However in rare occasions, backups may still fail and need to restarted.

Trilio Cluster¶

Trilio can be deployed as a single node or a three node cluster. It is highly recommended that Trilio is deployed as three node cluster for fault tolerance and load balancing. Starting with 3.0 release, Trilio requires additional IP for cluster and is required for both single node and three node deployments. Cluster ip a.k.a virtual ip is used for managing cluster and is used to register Trilio service endpoint in the keystone sevice catalog.

Trilio integrates natively with Openstack. This includes that Trilio communicates completely through APIs using the Openstack Endpoints. Trilio is also generating its own Openstack endpoints. In addition, is the Trilio appliance and the compute nodes writing to and reading from the backup target. These points affect the network planning for the Trilio installation.

Existing endpoints in Openstack

Openstack knows 3 types of endpoints:

• Public Endpoints

• Internal Endpoints

• Admin Endpoints

Each of these endpoint types is designed for a specific purpose. Public endpoints are meant to be used by the Openstack end-users to work with Openstack. Internal endpoints are meant to be used by the Openstack services to communicate with each other. Admin endpoints are meant to be used by Openstack administrators.

Out of those 3 endpoint types does only the admin endpoint sometimes contain APIs which are not available on any other endpoint type.

To learn more about Openstack endpoints please visit the official Openstack documentation.

Openstack endpoints required by Trilio

Trilio is communicating with all services of Openstack on a defined endpoint type. Which endpoint type Trilio is using to communicate with Openstack is decided during the configuration of the Trilio appliance.

The following network requirement can be identified this way:

• Trilio appliance needs access to the Keystone admin endpoint on the admin endpoint network

• Trilio appliance needs access to all endpoints of one type

Recommendation: Provide access to all Openstack Endpoint types

Trilio is recommending providing full access to all Openstack endpoints to the Trilio appliance to follow the Openstack standards and best practices.

Trilio is generating its own endpoints as well. These endpoints are pointing towards the Trilio Appliance directly. This means that using those endpoints will not send the API calls towards the Openstack Controller nodes first, but directly to the Trilio VM.

Following the Openstack standards and best practices, it is therefore recommended to put the Trilio endpoints on the same networks as the already existing Openstack endpoints. This allows to extend the purpose of each endpoint type to the Trilio service:

• The public endpoint to be used by Openstack users when using Trilio CLI or API

• The internal endpoint to communicate with the Openstack services

• The admin endpoint to use the required admin only APIs of Keystone

Backup target access required by Trilio

The Trilio solution is using backup target storage to securely place the backup data. Trilio is dividing its backup data into two parts:

1. Metadata

2. Volume Disk Data

The first type of data is generated by the Trilio appliance through communicating with the Openstack Endpoints. All metadata that is stored together with a backup is written by the Trilio Appliance to the backup target in the json format.

The second type of data is generated by the Trilio Datamover service running on the compute nodes. The Datamover service is reading the Volume Data from the Cinder or Nova storage and transferring this data as qcow2 image to the backup target. Each Datamover service is hereby responsible for the VMs running on its compute node.

The network requirements are therefor:

• The Trilio appliance needs access to the backup target

• Every compute node needs access to the backup target

Example of a typical Trilio network integration

Most Trilio customers are following the Openstack standards and best practices to have the public, internal, and admin endpoints on separate networks. They also typically don't have any network yet, which can access the desired backup target.

The starting network configuration typically looks like this:

Following the Openstack standards and Trilio's recommendation will the Trilio Appliance be placed on all those 3 networks. Further is the access to the backup target required by Trilio Appliance and Compute nodes. Here done by adding a 4th network.

The resulting network configuration would look like this:

It is of course possible to combine networks as necessary. As long as the required network access is available will Trilio work.

Other examples of Trilio network integrations

Each Openstack installation is different and so is the network configuration. There are endless possibilities of how to configure the Openstack network and how to implement the Trilio appliance into this network. The following three examples have been seen in production:

The first example is from a manufacturing company, which wanted to split the networks by function and decided to put the Trilio backup target on the internal network as the backup and recovery function was identified as an Openstack internal solution. This example looks complex but integrates Trilio just as recommended.

The second example is from a financial institute that wanted to be sure that the Openstack Users have no direct uncontrolled network access to the Openstack infrastructure. Following this example requires additional work as the internal HA-Proxy needs to be configured to correctly translates the API calls towards the Trilio appliance.

The third example is from a service company that was forced to treat Trilio as an external 3rd party solution, as we require a virtual machine running outside of Openstack. This kind of network configuration requires good planning on the Trilio endpoints and firewall rules.

Trilio, by TrilioData, is a native OpenStack service that provides policy-based comprehensive backup and recovery for OpenStack workloads. The solution captures point-in-time workloads (Application, OS, Compute, Network, Configurations, Data and Metadata of an environment) as full or incremental snapshots. These snapshots can be held in a variety of storage environments including NFS AWS S3 compatible storage. With Trilio and its single click recovery, organizations can improve Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO). With Trilio, IT departments are enabled to fully deploy OpenStack solutions and provide business assurance through enhanced data retention, protection and integrity.

With the use of Trilio’s VAST (Virtual Snapshot Technology), Enterprise IT and Cloud Service Providers can now deploy backup and disaster recovery as a service to prevent data loss or data corruption through point-in-time snapshots and seamless one-click recovery. Trilio takes point-in-time backup of the entire workload consisting of compute resources, network configurations and storage data as one unit. It also takes incremental backups that only captures the changes that were made since last backup. Incremental snapshots save time and storage space as the backup only includes changes since the last backup. The benefits of using VAST for backup and restore could be summarized as below:

1. Efficient capture and storage of snapshots. Since our full backups only include data that is committed to storage volume and the incremental backups only include changed blocks of data since last backup, our backup processes are efficient and storages backup images efficiently on the backup media

2. Faster and reliable recovery. When your applications become complex that snap multiple VMs and storage volumes, our efficient recovery process will bring your application from zero to operational with just click of button

3. Easy migration of workloads between clouds. Trilio captures all the details of your application and hence our migration includes your entire application stack without leaving any thing for guess work.

4. Through policy and automation lower the Total Cost of Ownership. Our tenant driven backup process and automation eliminates the need for dedicated backup administrators, there by improves your total cost of ownership.

Trilio 4.0 GA

Trilio 4.0 SP1

Trilio has four main software components:

1. Trilio ships as a QCOW2 image. User can instantiate one or more VMs from the QCOW2 image on a standalone KVM boxes.

2. Trilio API is a python module that is an extension to nova api service. This module is installed on all OpenStack controller nodes

3. Trilio Datamover is a python module that is installed on every OpenStack compute nodes

4. Trilio horizon plugin is installed as an add on to horizon servers. This module is installed on every server that runs horizon service.

System requirements Trilio Appliance

The Trilio Appliance gets delivered as a qcow2 image, which gets attached to a virtual machine.

Trilio supports KVM-based hypervisors on x86 architectures, with the following properties:

The recommended size of the VM for the Trilio Appliance is:

The qcow2 image itself defines the 40GB disk size of the VM.

In addition to the Trilio Appliance does Trilio contain components, which are installed directly into the Openstack itself.

Additional it is necessary to have the nfs-common packages installed on the compute nodes in case of using the NFS protocol for the backup target.

The Trilio Appliance is delivered as qcow2 image and runs as VM on top of a KVM Hypervisor.

Creating the cloud-init image

The Trilio appliance is utilizing cloud-init to provide the initial network and user configuration.

Cloud-init is reading it's information either from a metadata server or from a provided cd image. Trilio is utilizing the cd image.

Needed tools

To create the cloud-init image it is required to have genisoimage available.

Providing the Metadata

Cloud-init is using two files for it's metadata.

The first file is called meta-data and contains the information about the network configuration. Below is an example of this file.

The second file is called user-data and contains little scripts and information to set up for example the user passwords. Below is an example of this file.

creating the image file

The image is getting created using genisoimage follwing this general command:

genisoimage -output <name>.iso -volid cidata -joliet -rock </path/user-data> </path/meta-data>

An example of this command is shown below.

Spining up the Trilio appliance

After the cloud-init image has been created the TriloVault appliance can be spun up on the desired KVM server.

Extract the Trilio QCOW2 tar file using the following command :

See below an example command, how to spin up the Trilio appliance using virsh and the created iso image.

Uninstalling cloud-init after first boot

Once the Trilio appliance is up and running with it's initial configuration is it recommended to uninstall cloud-init.

To uninstall cloud-init, follow the example below.

The Red Hat Openstack Platform Director is the supported and recommended method to deploy and maintain any RHOSP installation.

Trilio is integrating natively into the RHOSP Director. Manual deployment methods are not supported for RHOSP.

Prepare for deployment

Depending whether the RHOSP environment is already installed or is getting installed for the first time different steps are done to be able to deploy Trilio.

The following commands upload the Trilio puppet module to the overcloud. The actual upload happens upon the next deployment.

cd /home/stack
git clone -b <branch> https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/redhat-director-scripts/
# ./upload_puppet_module_puppet5.sh
Creating tarball...
Tarball created.
Creating heat environment file: /home/stack/.tripleo/environments/puppet-modules-url.yaml
Uploading file to swift: /tmp/puppet-modules-8Qjya2X/puppet-modules.tar.gz
+-----------------------+---------------------+----------------------------------+
| object                | container           | etag                             |
+-----------------------+---------------------+----------------------------------+
| puppet-modules.tar.gz | overcloud-artifacts | 368951f6a4d39cfe53b5781797b133ad |
+-----------------------+---------------------+----------------------------------+

## Make sure that
ls -ll /home/stack/.tripleo/environments/puppet-modules-url.yaml

Update overcloud roles data file to include Trilio services

Trilio contains of multiple services. Add these services to your roles_data.yaml.

Add the following services to the roles_data.yaml

Trilio Datamover Api Service

This service needs to share the same role as the nova-api service. In case of the pre-defined roles will the nova-api service run on the role Controller. In case of custom defined roles, it is necessary to use the role the nova-api service is using.

Add the following line to the identified role:

'OS::TripleO::Services::TrilioDatamoverApi'

Trilio Datamover Service

This service needs to share the same role as the nova-compute service. In case of the pre-defined roles will the nova-compute service run on the role Compute. In case of custom defined roles, it is necessary to use the role the nova-compute service is using.

Add the following line to the identified role:

'OS::TripleO::Services::TrilioDatamover' 

Prepare Trilio container images

Trilio containers are pushed to 'RedHat Container Registry'. Registry URL: 'registry.connect.redhat.com'. Container pull urls are given below.

Trilio Datamover container: registry.connect.redhat.com/trilio/trilio-datamover:<container-tag> Trilio Datamover Api Container: registry.connect.redhat.com/trilio/trilio-datamover-api:<container-tag> Trilio horizon plugin: registry.connect.redhat.com/trilio/trilio-horizon-plugin:<container-tag>

There are three registry methods available in RedHat Openstack Platform.

1. Remote Registry

2. Local Registry

3. Satellite Server

Remote Registry

Follow this section when 'Remote Registry' is used.

For this method it is not necessary to pull the containers in advance. It is only necessary to populate the trilio_env.yaml file with the Trilio container URLs from Redhat registry.

Populate the trilio_env_osp16.yaml with container urls for:

• Trilio Datamover container

• Trilio Datamover api container

• Trilio Horizon Plugin

(undercloud) [stack@undercloud redhat-director-scripts]$ grep 'Image' trilio_env_osp16.yaml
   DockerTrilioDatamoverImage: registry.connect.redhat.com/trilio/trilio-datamover:<container-tag>
   DockerTrilioDmApiImage: registry.connect.redhat.com/trilio/trilio-datamover-api:<container-tag>
   ContainerHorizonImage: registry.connect.redhat.com/trilio/trilio-horizon-plugin:<container-tag>

Local Registry

Follow this section when 'local registry' is used on the undercloud.

In this case it is necessary to pull the Trilio containers to the undercloud registry. Trilio provides shell scripts which will pull the containers from 'registry.connect.redhat.com' to the undercloud and updates the trilio_env_osp16.yaml with the values for the datamover and datamover api containers.

## Command to prepare, push trilio container images to undercloud registry: 
sudo ./podman_prepare_images.sh <UNDERCLOUD_REGISTRY_HOSTNAME> <CONTAINER_TAG> 


## Run following command to find 'UNDERCLOUD_REGISTRY_HOSTNAME'. Hostname is higlighted in bold format in below example.
$ openstack tripleo container image list | grep keystone
| docker://trilio-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-keystone:16.0-82                       |
| docker://trilio-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-barbican-keystone-listener:16.0-84   

## 'CONTAINER_TAG' format: <TRILIOVAULT_VERSION>-rhosp16
## For example. If Trilio version='4.0.92',  then 'CONTAINER_TAG'=4.0.92-rhosp16

## Example:
sudo ./podman_prepare_images.sh trilio-undercloud.ctlplane.localdomain 4.0.92-rhosp16

The changes can be verified using the following commands.

(undercloud) [stack@undercloud redhat-director-scripts]$ openstack tripleo container image list | grep trilio
| docker://undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:4.0.92-rhosp16                       |                        |
| docker://undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:4.0.92-rhosp16                   |                   |
| docker://undercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:4.0.92-rhosp16                  |

-----------------------------------------------------------------------------------------------------

(undercloud) [stack@undercloud redhat-director-scripts]$ grep 'Image' trilio_env_osp16.yaml
   DockerTrilioDatamoverImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:4.0.92-rhosp16
   DockerTrilioDmApiImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:4.0.92-rhosp16
   ContainerHorizonImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:4.0.92-rhosp16

Satellite Server

Follow this section when a Satellite Server is used for the container registry.

Pull the Trilio containers on the Red Hat Satellite using the given Red Hat registry URLs.

Populate the trilio_env_osp16.yaml with container urls for:

• Trilio Datamover container

• Trilio Datamover api container

(undercloud) [stack@undercloud redhat-director-scripts]$ grep 'Image' trilio_env_osp16.yaml
   DockerTrilioDatamoverImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:4.0.92-rhosp16
   DockerTrilioDmApiImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:4.0.92-rhosp16
   ContainerHorizonImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:4.0.92-rhosp16

Provide environment details in trilio-env_osp16.yaml

Provide backup target details and other necessary details in the provided environment file. This environment file will be used in the overcloud deployment to configure Trilio components. Container image names have already been populated in the preparation of the container images. Still it is recommended to verify the container URLs.

The following information are required additionally:

• Network for the datamover api

• Backup target type {nfs/s3}

• In case of NFS

  • list of NFS Shares

  • NFS options

• In case of S3

  • s3 type {amazon_s3/ceph_s3}

  • Access key

  • Secret key

  • S3 Region name

  • S3 Bucket name

  • S3 Endpoint URL

  • S3 SSL Enabled {true/false}

resource_registry:
  OS::TripleO::Services::TrilioDatamover: docker/services/trilio-datamover-osp16.yaml
  OS::TripleO::Services::TrilioDatamoverApi: docker/services/trilio-datamover-api-osp16.yaml

parameter_defaults:

   ## Enable Trilio's quota functionality on horizon
   ExtraConfig:
     horizon::customization_module: 'dashboards.overrides'

   ## Define network map for trilio datamover api service
   ServiceNetMap:
       TrilioDatamoverApiNetwork: internal_api

   ## Trilio container pull urls
   DockerTrilioDatamoverImage: devundercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:4.0.92-rhosp16
   DockerTrilioDmApiImage: devundercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:4.0.92-rhosp16

   ## If you do not want Trilio's horizon plugin to replace your horizon container, just comment following line.
   ContainerHorizonImage: devundercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:4.0.92-rhosp16


   ## Backup target type nfs/s3, used to store snapshots taken by triliovault
   BackupTargetType: 'nfs'

   ## For backup target 'nfs'
   NfsShares: '192.168.122.101:/opt/tvault'
   NfsOptions: 'nolock,soft,timeo=180,intr,lookupcache=none'

   ## For backup target 's3'
   ## S3 type: amazon_s3/ceph_s3
   S3Type: 'amazon_s3'

   ## S3 access key
   S3AccessKey: ''
  
   ## S3 secret key
   S3SecretKey: ''

   ## S3 region, if your s3 does not have any region, just keep the parameter as it is
   S3RegionName: ''

   ## S3 bucket name
   S3Bucket: ''

   ## S3 endpoint url, not required for Amazon S3, keep it as it is
   S3EndpointUrl: ''

   ## If SSL enabled on S3 url, not required for Amazon S3, just keep it as it is
   S3SslEnabled: false

   ## Don't edit following parameter
   EnablePackageInstall: True

Deploy overcloud with trilio environment

Use the following heat environment file and roles data file in overcloud deploy command:

1. trilio_env_osp16.yaml

2. roles_data.yaml

To include new environment files use '-e' option and for roles data file use '-r' option. An example overcloud deploy command is shown below:

openstack overcloud deploy --templates \
--libvirt-type qemu \
--ntp-server 192.168.1.34 \
-e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/trilio_env_osp16.yaml \
-e /home/stack/templates/node-info.yaml \
-e /home/stack/containers-prepare-parameter.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-tls.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/inject-trust-anchor.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \
-r /usr/share/openstack-tripleo-heat-templates/roles_data.yaml

Verify deployment

On Controller node

Make sure Trilio dmapi and horizon containers are in a running state and no other Trilio container is deployed on controller nodes.

[root@overcloud-controller-0 heat-admin]# podman ps | grep trilio
26fcb9194566  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-datamover-api:4.0.92-rhosp16  kolla_start  5 days ago  Up 5 days ago  trilio_dmapi
094971d0f5a9  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:4.0.92-rhosp16  kolla_start  5 days ago  Up 5 days ago  horizon

On Compute node

Make sure Trilio datamover container is in running state and no other Trilio container is deployed on compute nodes.

[root@overcloud-novacompute-0 heat-admin]# podman ps | grep trilio
b1840444cc59  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-datamover:4.0.92-rhosp16  kolla_start  5 days ago  Up 5 days ago  tilio_datamover

On the node with Horizon service

Make sure horizon container is in running state. Please note that 'Horizon' container is replaced with Trilio Horizon container. This container will have latest OpenStack horizon + Trilio's horizon plugin.

[root@overcloud-controller-0 heat-admin]# podman ps | grep horizon
094971d0f5a9  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:4.0.92-rhosp16  kolla_start  5 days ago  Up 5 days ago  horizon

Configure Trilio Appliance

Once RHOSP16.0 Installation steps have completed successfully, follow the instructions below to now configure the Trilio Appliance

Change the nova user id on the Trilio Nodes

In RHOSP, 'nova' user id on nova-compute docker container is set to '42436'. The 'nova' user id on the Trilio nodes need to be set the same. Do the following steps on all Trilio nodes:

1. Download the shell script that will change the user id

2. Assign executable permissions

3. Execute the script

4. Verify that 'nova' user and group id has changed to '42436'

# curl -O https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/master/common/nova_userid.sh
# chmod +x nova_userid.sh
# ./nova_userid.sh
# id nova
  uid=42436(nova) gid=42436(nova) groups=42436(nova),990(libvirt),36(kvm)

Run the Trilio configurator

Follow this documentation to configure Trilio Appliance.

Change the workloadmgr.conf to use the right mountpoint

RHOSP16 is using a different mount point in its datamover containers than other Openstack distribution. It is necessary to adjust the mountpoint of the Trilio Nodes to match this. If the mountpoints are not getting aligned, will files created by the datamover and read by the Trilio appliance not match in their paths and backup and restore processes will fail.

Please follow these steps to align the mountpoints:

1. Edit /etc/workloadmgr/workloadmgr.conf file

2. Set parameter 'vault_data_directory' to '/var/lib/nova/triliovault-mounts'

3. create the directory for the mountpoint

4. assign the created directory to nova:nova

5. unmount the old mountpoint

6. Update the Trilio configurator

7. Restart the Trilio services

8. Verify the mountpoint has been configured correctly

# cat /etc/workloadmgr/workloadmgr.conf | grep vault_data_directory
vault_data_directory = /var/lib/nova/triliovault-mounts
vault_data_directory_old = /var/triliovault

# mkdir -p /var/lib/nova/triliovault-mounts

# chown nova:nova /var/lib/nova/triliovault-mounts

# umount /var/triliovault-mounts

# cat /home/stack/myansible/lib/python3.6/site-packages/workloadmgr/tvault_configurator/ansible-play/roles/ansible-workloadmgr/templates/workloadmgr.conf.j2 | grep vault_data_directory
vault_data_directory = /var/lib/nova/triliovault-mounts

#pcs resource restart wlm-cron 

#pcs resource restart wlm-scheduler 

#systemctl restart wlm-api 

#systemctl restart wlm-workloads

#df-h
Filesystem             Size  Used Avail Use% Mounted on
devtmpfs               3.8G     0  3.8G   0% /dev
tmpfs                  3.8G   38M  3.8G   1% /dev/shm
tmpfs                  3.8G  427M  3.4G  12% /run
tmpfs                  3.8G     0  3.8G   0% /sys/fs/cgroup
/dev/vda1               40G  8.8G   32G  22% /
tmpfs                  773M     0  773M   0% /run/user/996
tmpfs                  773M     0  773M   0% /run/user/0
10.10.2.20:/upstream  1008G  704G  254G  74% /var/lib/nova/triliovault-mounts/MTAuMTAuMi4yMDovdXBzdHJlYW0= 

Troubleshooting for overcloud deployment failures

Trilio components will be deployed using puppet scripts.

In case of the overcloud deployment failing does the following command provide the list of errors:

openstack stack failures list overcloud

Further commands that can help identifying any errors.

heat stack-list --show-nested -f "status=FAILED"
heat resource-list --nested-depth 5 overcloud | grep FAILED

In case of the trilio datamover api container is failing to start or stuck in restart, check these logs:

sudo podman logs trilio_dmapi
tailf /var/log/containers/trilio-datamover-api/dmapi.log

In case of Trilio datamover container failing to start or being stuck in restart, check these logs:

sudo podman logs trilio_datamover
tailf /var/log/containers/trilio-datamover/tvault-contego.log

Cinder backend is Ceph - additional steps

Add Ceph details to configuration file

If Cinder backend is Ceph it is necessary to manually add the ceph details to tvault-contego.conf on all compute nodes.

The file can be found here: /var/lib/config-data/puppet-generated/triliodm/etc/tvault-contego/tvault-contego.conf

Add the following information:

[libvirt]
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = <rbd_user>

[ceph] 
keyring_ext = .<rbd_user>.keyring

Restart the datamover container

sudo podman restart trilio_datamover

Once the Trilio VM or the Cluster of Trilio VMs has been spun, the actual installation process can begin. This process contains of the following steps:

1. Install the Trilio dm-api service on the control plane

2. Install the Trilio datamover service on the compute plane

3. Install the Trilio Horizon plugin into the Horizon service

How these steps look in detail is depending on the Openstack distribution Trilio is installed in. Each supported Openstack distribution has its own deployment tools. Trilio is integrating into these deployment tools to provide a native integration from the beginning to the end.

Change the nova user id on the Trilio Nodes

In Kolla, 'nova' user id on nova-compute docker container is set to '42436'. The 'nova' user id on the Trilio nodes need to be set the same. Do the following steps on all compute nodes:

1. Download the shell script that will change the user id

2. Assign executable permissions

3. Execute the script

4. Verify that 'nova' user and group id has changed to '42436'

curl -O https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/master/common/nova_userid.sh 
chmod +x nova_userid.sh
./nova_userid.sh
id nova
# uid=42436(nova) gid=42436(nova) groups=42436(nova),990(libvirt),36(kvm)

Deploy Trilio Datamover API on all Openstack controller nodes

Trilio Datamover Api container should be deployed on all nodes where nova_api container is running. In standard deployment, we can call these nodes as OpenStack controller nodes.

Pull TrilioVaullt Datamover API container image

The very first step is to pull the container image from docker.io.

Login to docker and pull the Trilio Datamover API container.

docker login docker.io

## Pull Trilio Datamover Api container using following command.
## You need to edit OS_PLATFORM, TVAULT_VERSION, OPENSTACK_RELEASE_NAME.
## <OS_PLATFORM> ubuntu or centos.
## <OPENSTACK_RELEASE_NAME> train
## <TVAULT_VERSION> 4.0.92 or 4.0.115

docker pull docker.io/trilio/<OS_PLATFORM>-source-trilio-datamover-api:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME>;

Create the Trilio Datamover API configuration

In this part of the process is the configuration file for the Trilio Datamover API created.

The following steps need to be done:

1. Create config directory

2. get default config file dmapi.conf

3. edit dmapi.conf

4. copy nova.conf to config directory

Create config directory for trilio-datamover-api service

mkdir -p /etc/kolla/trilio-datamover-api

Get the trilio-datamover-api default config file

git clone https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/
git checkout stable/4.0 

cp kolla-ansible/trilio-datamover-api/dmapi.conf.sample /etc/kolla/trilio-datamover-api/dmapi.conf

Edit dmapi.conf

The dmapi.conf located in /etc/kolla/trilio-datamover-api/ needs to be edited to adjust to the Openstack environment.

An example dmapi.conf can be seen here:

[DEFAULT]
dmapi_workers = 2
# Get this parameter value from nova-api.conf
transport_url = rabbit://openstack:aklGvwxJ2GE9nnp1ivEOBPTCfwXev6TNGj2RIUHc@11.11.11.95:5672//

##In the following paramater, edit IP address only. Use dmapi/nova-api nodes fixed ip.
dmapi_link_prefix = http://11.11.11.95:8784
dmapi_enabled_ssl_apis =
dmapi_listen_port = 8784
dmapi_enabled_apis = dmapi
bindir = /usr/bin
instance_name_template = instance-%08x
## dmapi_listen should be assigned with nova-api/dmapi node's fixed ip
dmapi_listen = 11.11.11.95
my_ip = 11.11.11.95
rootwrap_config = /etc/dmapi/rootwrap.conf
debug = True
log_file = /var/log/kolla/trilio-datamover-api/dmapi.log
log_dir = /var/log/kolla/trilio-datamover-api

[wsgi]
ssl_cert_file =
ssl_key_file =
api_paste_config = /etc/dmapi/api-paste.ini

[database]
# Get this parameter value from nova-api.conf
connection = mysql+pymysql://nova:IJP6ryxvAvQDZdNxosJr2OKDRqEofTN4Zbet638U@11.11.11.96:3306/nova

[api_database]
# Get this parameter value from nova-api.conf
connection = mysql+pymysql://nova_api:YoHe1umK1pso0xINCmcmvmK7IkTLPT9QBn3aMduF@11.11.11.96:3306/nova_api

[keystone_authtoken]
signing_dir = /var/cache/dmapi
# Get these parameter values from nova-api.conf
cafile =
user_domain_name = Default
insecure = True
auth_uri = http://11.11.11.96:5000
auth_url = http://11.11.11.96:35357
auth_type = password
project_domain_id = default
user_domain_id = default
project_name = service
username = nova
password = XZ2Do9MYEWivxZoiuKKHAc6wOJOPVGHzgXyYu9ic
memcache_security_strategy = ENCRYPT
memcache_secret_key = W8NudaXmPejmCcefr0I0WEcZExFnGrbd6I6ZOkOv
memcached_servers = 11.11.11.95:11211

[oslo_messaging_notifications]
# Get this parameter value from nova-api.conf
transport_url = rabbit://openstack:aklGvwxJ2GE9nnp1ivEOBPTCfwXev6TNGj2RIUHc@11.11.11.95:5672//
driver = noop

[oslo_middleware]
enable_proxy_headers_parsing = true

Copy nova.conf of nova-api service to trilio-datamover-api directory

cp /etc/kolla/nova-api/nova.conf /etc/kolla/trilio-datamover-api/
chmod -R 744 /etc/kolla/trilio-datamover-api/

Create trilio-datamover-api log directory

mkdir -p /var/log/kolla/trilio-datamover-api

1. If openstack is based on CentOS :
##Change ownership of log directory to 42436:42436 - nova user and group id on container
chmod 755 /var/log/kolla/trilio-datamover-api
chown 42436:42436 /var/log/kolla/trilio-datamover-api

2. If openstack is based on Ubuntu :
##Change ownership of log directory to 42486:42487- dmapi user and group id on container 
chmod 755 /var/log/kolla/trilio-datamover-api 
chown 42486:42487 /var/log/kolla/trilio-datamover-api

Add trilio-datamover-api to haproxy.cfg

## Edit haproxy.cfg 
vi /etc/kolla/haproxy/haproxy.cfg

## If SSL enabled on public interface of dmapi
listen trilio_datamover_api
  bind <Keystone_virtual_ip>:8784  ssl crt /etc/haproxy/haproxy.pem
  server <controller_hostname_1> <controller_IP1>:8784 check inter 2000 rise 2 fall 5
  server <controller_hostname_2> <controller_IP2>:8784 check inter 2000 rise 2 fall 5
  server <controller_hostname_3> <controller_IP3>:8784 check inter 2000 rise 2 fall 5

## If SSL is not enabled on any interface
listen trilio_datamover_api
  bind <Keystone_virtual_ip>:8784
  server <controller_hostname_1> <controller_IP1>:8784 check inter 2000 rise 2 fall 5
  server <controller_hostname_2> <controller_IP2>:8784 check inter 2000 rise 2 fall 5 
  server <controller_hostname_3> <controller_IP3>:8784 check inter 2000 rise 2 fall 5

## Restart haproxy container
docker restart haproxy

Run trilio-datamover-api container

Now the trilio-datamover-api container can be deployed and started.

## You need to edit OS_PLATFORM, TVAULT_VERSION, OPENSTACK_RELEASE_NAME. 
## <OS_PLATFORM> ubuntu or centos.
## <OPENSTACK_RELEASE_NAME> train
## <TVAULT_VERSION> : 4.0.92

docker run --network host --name trilio_datamover_api -d --restart always -v /etc/kolla/trilio-datamover-api/nova.conf:/etc/nova/nova.conf \
-v /etc/kolla/trilio-datamover-api/dmapi.conf:/etc/dmapi/dmapi.conf \
-v /var/log/kolla/:/var/log/kolla/ \
trilio/<OS_PLATFORM>-source-trilio-datamover-api:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME> /var/lib/kolla/venv/bin/python /usr/bin/dmapi-api

Verify deployment of trilio-datamover-api container

To verify the deployment was successful check the container status using docker ps.

root@trilio-Standard-PC-i440FX-PIIX-1996:~# docker ps | grep trilio_datamover_api
736e4ac462c9  trilio/ubuntu-source-trilio-datamover-api:4.0.92  "/var/lib/kolla/venv/"  3 days ago  Up 23 hours  trilio_datamover_api

Deploy Trilio Datamover container on all Compute nodes

Trilio Datamover container should be deployed on all nodes where nova_compute container is running. In standard deployment, we can call these nodes as openstack compute nodes.

Pull Trilio Datamover container

The very first step is to pull the container image from docker.io.

Login to docker and pull the Trilio Datamover API container.

docker login docker.io

## Pull Trilio Datamover container using following command.
## You need to edit OS_PLATFORM, TVAULT_VERSION, OPENSTACK_RELEASE_NAME.
## <OS_PLATFORM> ubuntu or centos.
## <OPENSTACK_RELEASE_NAME> train
## <TVAULT_VERSION> 4.0.92 or 4.0.115

docker pull docker.io/trilio/<OS_PLATFORM>-source-trilio-datamover:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME>;

Create the Trilio Datamover configuration

In this part of the process is the configuration file for the Trilio Datamover created.

The following steps need to be done:

1. Create config directory

2. copy nova.conf to config directory

3. get default config file tvault-contego.conf

4. edit tvault.conf

Create service config directory for trilio-datamover service

mkdir -p /etc/kolla/trilio-datamover

Copy nova.conf of nova-compute service to trilio-datamover config directory

cp /etc/kolla/nova-compute/nova.conf /etc/kolla/trilio-datamover/

Get the trilio-datamover default config file

## Clone triliovault devops repository. It's public github repository.
git clone https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/ 
git checkout <GITHUB_BRANCH>          // Check section: 1.i) plan for deployment for correct github branch name

## If backup storage type you want to use is 'NFS'
cp kolla-ansible/trilio-datamover/tvault-contego.conf.nfs.sample /etc/kolla/trilio-datamover/tvault-contego.conf

## If backup storage type you want to use is 'S3'
cp kolla-ansible/trilio-datamover/tvault-contego.conf.s3.sample /etc/kolla/trilio-datamover/tvault-contego.conf

## Set correct permissions
chmod -R 744 /etc/kolla/trilio-datamover/

Edit tvault-contego.conf

Edit /etc/kolla/trilio-datamover/tvault-contego.conf config file to provide NFS/S3 details as per backup storage selected.

[DEFAULT]
vault_storage_type = nfs
vault_storage_nfs_export = 192.168.1.34:/mnt/tvault/tvm
vault_data_directory_old = /var/triliovault
vault_data_directory = /var/triliovault-mounts
log_file = /var/log/kolla/trilio-datamover/tvault-contego.log
debug = False
verbose = True
max_uploads_pending = 3
max_commit_pending = 3
vault_s3_auth_version = DEFAULT
vault_s3_access_key_id =
vault_s3_secret_access_key =
vault_s3_region_name = us-east-2
vault_s3_bucket =
qemu_agent_ping_timeout = 900

[contego_sys_admin]
helper_command = sudo /var/lib/kolla/venv/bin/privsep-helper

[conductor]
use_local = True

Create trilio-datamover log directory

mkdir -p /var/log/kolla/trilio-datamover

##Change ownership of log directory to 42436:42436 - nova user and group id on container
chmod 755 /var/log/kolla/trilio-datamover
chown 42436:42436 /var/log/kolla/trilio-datamover

If Ceph is used for Nova/Cinder Storage

If ceph is getting used for cinder/nova, the correct permissions for ceph.conf and keyrings files need to be assigned. The trilio_datamover container will be using ceph.conf and keyring files with the 'nova' user.

chmod o+rx /etc/ceph
chmod o+r /etc/ceph/*

If nova/cinder backend is ceph, you need to add ceph user and keyring details to /etc/kolla/trilio-datamover/tvault-contego.conf file. Add the following sections to the tvault-contego.conf file. In the provided example is ceph's 'cinder' user configured to use for trilio read/write operations.

[libvirt]
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = cinder

[ceph]
keyring_ext = .cinder.keyring

Run trilio-datamover container

## In following docker run command you need to edit OS_PLATFORM, TVAULT_VERSION,vOPENSTACK_RELEASE_NAME.

## <OS_PLATFORM> ubuntu or centos.
## <OPENSTACK_RELEASE_NAME> train
## <TVAULT_VERSION> : 4.0.92 or 4.0.115

##If backup storage type is 'NFS'
docker run --privileged --network host --name trilio_datamover -d \
--restart always \
-v /etc/kolla/trilio-datamover/tvault-contego.conf:/etc/tvault-contego/tvault-contego.conf \
-v /etc/kolla/trilio-datamover/nova.conf:/etc/nova/nova.conf \
-v /dev:/dev:rw \
-v /etc/ceph:/etc/ceph:ro \
-v nova_compute:/var/lib/nova/:rw \
-v /var/log/kolla/:/var/log/kolla/ \
-v iscsi_info:/etc/iscsi:rw -v /var/run/libvirt \
trilio/<OS_PLATFORM>-source-trilio-datamover:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME> /opt/tvault/start_datamover_nfs

##If backup storage type is 'S3'
docker run --privileged --network host --name trilio_datamover -d \
--restart always \
-v /etc/kolla/trilio-datamover/tvault-contego.conf:/etc/tvault-contego/tvault-contego.conf \
-v /etc/kolla/trilio-datamover/nova.conf:/etc/nova/nova.conf \
-v /dev:/dev:rw \
-v /etc/ceph:/etc/ceph:ro \
-v nova_compute:/var/lib/nova/:rw \
-v /var/log/kolla/:/var/log/kolla/ \
-v iscsi_info:/etc/iscsi:rw -v /var/run/libvirt \
trilio/<OS_PLATFORM>-source-trilio-datamover:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME> /opt/tvault/start_datamover_s3

Verify deployment of trilio-datamover container

To verify the deployment was successful check the container status using docker ps.

root@trilio-Standard-PC-i440FX-PIIX-1996:~# docker ps | grep trilio_datamover
9d572974c75d  trilio/centos-source-trilio-datamover:4.0.92  "/opt/tvault/start_da"  2 days ago  Up 23 hours  trilio_datamover

Installing Trilio Horizon plugin

Trilio Horizon plugin needs to be installed inside the OpenStack horizon container. Once installed, the Trilio dashboard will be visible in OpenStack Horizon.

The following steps need to be done:

1. Download installation shell script

2. Run the shell script

3. Edit Horizon settings

4. Restart the Horizon container

Download the installation shell script

To download the shell script directly into the Horizon container do:

docker exec -it horizon /bin/bash
curl -kO https://<TVAULT_VM_IP>/tvault-horizon-plugin-install.sh
chmod +x tvault-horizon-plugin-install.sh

Run the shell script and restart Horizon container

Run the shell script and restart horizon container. This will restart apache service, which may enforce a log out of the container.

# Login to horizon container if not already.
docker exec -itu root horizon /bin/bash

## Run the trilio horizon plugin shell script 
- Script will ask for python2/python3, you need to select the option as per your enviornment
- Trilio horizon install script will ask for horizon's openstack_dashboard directory path if it's not at the default
 location - '/usr/shar/openstack-dashboard' For train ubuntu bionic, it's : '/var/lib/kolla/venv/lib/python2.7/site-packages'

./tvault-horizon-plugin-install.sh

## Exit the horizon container
Ctrl + D

Edit Horizon settings

The following line needs to be aded in 'local_settings' of Openstack's Horizon file to enable workloadmanager quota feature in the Horizon dashboard.

## vi /etc/kolla/horizon/local_settings

Add this line : HORIZON_CONFIG['customization_module'] = 'dashboards.overrides'

Restart the Horizon container

To enable the done changes restart the Horizon container

## Restart the horizon container
docker restart horizon

Known issues for Horizon plugin installation

If OpenStack is based on 'Centos' platform

Grafana yum repository has an issue on the latest horizon containers of OpenStack (not Trilio). To confirm the issue, you can just run yum repolist, it will fail. Use the following command to disable the grafana repository.

yum --disablerepo=grafana

If Openstack is based on Ubuntu platform

Trilio horizon install script will ask for horizon's openstack_dashboard directory path if it's not at the default location - /usr/shar/openstack-dashboard

For train ubuntu bionic, it's : /var/lib/kolla/venv/lib/python2.7/site-packages

If Trilio Horizon tabs are not accessible but Openstack Horizon works

If Trilio Horizon tabs are not accessible but OpenStack Horizon is working fine, make sure that endpoints for service 'TrilioVaultWLM' are created correctly. The root cause of this issue is typically, that SSL is enabled on all three endpoint types of 'TrilioVaultWLM' service.

If SSL is enable only on public 'keystone' service endpoints, then create 'TrilioVaultWLM' service endpoints in the same fashion. Endpoints for service 'TrilioVaultWLM' get created during Trilio configuration step. If these endpoints need to be edited reconfigure the Trilio.

Enabling Snapshout Mount

To make 'snapshot mount' functionality work, the cloud administrator needs to complete the following steps.

1. Identify backup target mount point on Trilio VM

2. install nfs-common on nova_compute and nova_libvirt containers

3. Mount backup target nfs share on nova_compute and nova_libvirt containers

Identify backup target mount point in Trilio VM

The following command will provide the active mountpoint on the Trilio VM

[stack@tvm ~]$ mount | grep triliovault
192.168.1.33:/mnt/tvault on /var/triliovault-mounts/MTkyLjE2OC4xLjMzOi9tbnQvdHZhdWx0 type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=180,retrans=2,sec=sys,clientaddr=192.168.10.10,lookupcache=none,local_lock=none,addr=192.168.1.33)

This example gives the following information:

Backup target is NFS share: 192.168.1.33:/mnt/tvault Mountpoint is: /var/triliovault-mounts/MTkyLjE2OC4xLjMzOi9tbnQvdHZhdWx0

Install nfs-common on nova_compute and nova_libvirt containers

It is necessary to install nfs-common package on both nova_compute and nova_libvirt containers.

# Login to nova_compute container with root user
docker exec -itu root nova_compute /bin/bash

# Install nfs-utils/nfs-common package using
=> If it's centos
yum install nfs-utils
=> If it's ubuntu
apt-get install nfs-common

# Login to nova_libvirt container with root user
docker exec -itu root nova_libvirt /bin/bash

# Install nfs-utils/nfs-common package using yum
=> If it's centos 
yum install nfs-utils 
=> If it's ubuntu
apt-get install nfs-common

Mount backup target nfs share on nova_compute and nova_libvirt containers

Mount the backup target nfs share on 'nova_compute' and 'nova_libvirt' container at exactly same mount point as done on triliovault VM.

mount -t nfs 192.168.1.33:/mnt/tvault /var/triliovault-mounts/MTkyLjE2OC4xLjMzOi9tbnQvdHZhdWx0

Troubleshoot installation

If any triliovault container is stuck in restarting state the following logs can be checked.

# Get docker run logs for datamover container
docker logs trilio_datamover

## Get docker logs for datamover api container
docker logs trilio_datamover_api

Possible issues for trilio-datamover container failure are for example NFS mount issues or S3 credentials might be wrong. If it's Amazon S3, then network connectivity between compute node and AWS s3 is needed. The docker logs will clearly tell the exact error.

If the above logs do not help OR If containers running well but, backups fail, following service logs will help:

/var/log/kolla/trilio-datamover/tvault-contego.log
/var/log/kolla/trilio-datamover-api/dmapi.log

If the Trilio Horizon tabs are not visible on Openstack, verify the following:

• Make sure trilio horizon plugin is installed on OpenStack horizon container

• Trilio configuration step needs to be completed to see the triliovault dashboard on OpenStack

• Make sure correct openstack_dashboard directory got provided and the triliovault horizon plugin files got successfully copied there.

Backup-as-a-Service

Trilio is an add on service to OpenStack cloud infrastructure and provides backup and disaster recovery functions for tenant workloads. Trilio is very similar to other openstack services including nova, cinder, glance, etc and adheres to all tenets of OpenStack. It is a stateless service that scales with your cloud.

Main Components

Trilio has four main software components:

1. Trilio ships as a QCOW2 image. User can instantiate one or more VMs from the QCOW2 image on a standalone KVM boxes.

2. Trilio API is a python module that is installed on all OpenStack controller nodes where the nova-api service is running.

3. Trilio Datamover is a python module that is installed on every OpenStack compute nodes

4. Trilio horizon plugin is installed as an add on to horizon servers. This module is installed on every server that runs horizon service.

Service Endpoints

Trilio is both a provider and consumer into OpenStack ecosystem. It uses other OpenStack services such as nova, cinder, glance, neutron, and keystone and provides its own service to OpenStack tenants. To accomodate all possible OpenStack deployments, Trilio can be configured to use either public or internal URLs of services. Likewise Trilio provides its own public, internal and admin URLs.

Network Topology

This figure represents a typical network topology. Trilio exposes its public URL endpoint on public network and Trilio virtual appliances and data movers typically use either internal network or dedicated backup network for storing and retrieving backup images from backup store.

Trilio and Canonical have started a partnership to ensure a native deployment of Trilio using JuJu Charms.

Those JuJu Charms are publicly available as Open Source Charms.

The following charms exist:

• Installs and manages Trilio Controller services.

• Installs and manages the Trilio Datamover API service.

• Installs and manages the Trilio Datamover service.

• Installs and manages the Trilio Horizon Plugin.

The documentation of the charms can be found here:

Installing Trilio Datamover-API container

The installation of the Datamover-API, short dmapi, requires to create a new container, in which all necessary packages and the Trilio dmapi code are loaded.

Create the dmapi container on Controller nodes

Create lxc container for hosting dmapi service on controller nodes with below commands.

Prepare the dmapi container for installation

Add nova user and required directory on container controller_dmapi.

Add required packages on container controller_dmapi.

Copy nova.conf file from nova-api container to /etc/nova directory in dmapi container. Run the below command on controller nodes:

Create a new interface with specific ip for dmapi container.

Edit /var/lib/lxc/controller_dmapi/config and add below section as per network bridge available on the controller node.

Restart the container with the below commands.

Install the dmapi service

Download and run the tvault-installation script inside the container.

The script to be executed inside dmapi container, after the following changes have been done: Comment the 2 lines below and add a line below NOVA_VERSION = 20, as nova-manage doesn't work in Ansible Openstack.

Run the script

Verify the installation and set necessary configuration

Verify in dmapi.conf domain name for the nova service user under keystone section. Check field values for project_domain_nameand user_domain_name and update those if not in keystone section

If SSL is enabled then add the following section in dmapi.conf:

Verify below entries are there in keystone policy.json file

Once verified above checks, start the dmapi service.

Install Trilio Datamover on Compute nodes

Prepare the installation

Activate the virtual environment on the compute node.

After activating the virtual environment, find out the location of compute.filters file.

Download the installation script.

Modify install script to use the same location for creating trilio filters.

Also comment the 2 lines below and add a line below NOVA_VERSION = 20, as nova-manage doesn't work in Ansible Openstack.

Install the Trilio datamover service

Run install script and if you get prompt while installing, choose the default selection.

Make sure ExecStart value look like below in /etc/systemd/system/tvault-contego.service file.

Use below commands to restart and verify the service.

Use below command and check if nfs/s3 storage is mounted or not.

Install Horizon Plugin into Horizon service

List running containers on controller nodes and login to horizon container using the below command.

Install curl package on the Horizon container if not present.

Activate virtual environment on horizon container

Download script to install horizon plugin on horizon container and run install script

Install script will ask for the dashboard folder, provide below path

Verify installation using below commands

Haproxy configuration on controller nodes

Refer to the keystone haproxy settings for dmapi haproxy.

A sample configuration is shown below.

Check the syntax of the file and restart the service.

The is the supported and recommended method to deploy and maintain any RHOSP installation.

Trilio is integrating natively into the RHOSP Director. Manual deployment methods are not supported for RHOSP.

Prepare for deployment

Depending whether the RHOSP environment is already installed or is getting installed for the first time different steps are done to be able to deploy Trilio.

The following command clones the Trilio configscripts github.

If your backup target is ceph S3 with SSL and SSL certificates are self signed or authorized by private CA, then user needs to provide CA chain certificate to validate the SSL requests. For that, user needs to rename his ca chain cert file to 's3-cert.pem' and copy it into directory - 'triliovault-cfg-scripts/redhat-director-scripts/redhat-director-scripts/puppet/trilio/files'

The following commands upload the Trilio puppet module to the overcloud registry. The actual upload happens upon the next deployment.

Update overcloud roles data file to include Trilio services

Trilio contains of multiple services. Add these services to your roles_data.yaml.

Add the following services to the roles_data.yaml

Trilio Datamover Api Service

This service needs to share the same role as the nova-api service. In case of the pre-defined roles will the nova-api service run on the role Controller. In case of custom defined roles, it is necessary to use the role the nova-api service is using.

Add the following line to the identified role:

Trilio Datamover Service

This service needs to share the same role as the nova-compute service. In case of the pre-defined roles will the nova-compute service run on the role Compute. In case of custom defined roles, it is necessary to use the role the nova-compute service is using.

Add the following line to the identified role:

Prepare Trilio container images

Trilio containers are pushed to 'RedHat Container Registry'. Registry URL: 'registry.connect.redhat.com'. Container pull urls are given below.

Trilio Datamover container: registry.connect.redhat.com/trilio/trilio-datamover:4.0.116-rhosp16.1 Trilio Datamover Api Container: registry.connect.redhat.com/trilio/trilio-datamover-api:4.0.116-rhosp16.1 Trilio horizon plugin: registry.connect.redhat.com/trilio/trilio-horizon-plugin:4.0.116-rhosp16.1

There are three registry methods available in RedHat Openstack Platform.

1. Remote Registry

2. Local Registry

3. Satellite Server

Remote Registry

Follow this section when 'Remote Registry' is used.

For this method it is not necessary to pull the containers in advance. It is only necessary to populate the trilio_env.yaml file with the Trilio container URLs from Redhat registry.

Populate the trilio_env_osp16.yaml with container urls for:

• Trilio Datamover container

• Trilio Datamover api container

• Trilio Horizon Plugin

Local Registry

Follow this section when 'local registry' is used on the undercloud.

In this case it is necessary to pull the Trilio containers to the undercloud registry. Trilio provides shell scripts which will pull the containers from 'registry.connect.redhat.com' to the undercloud and updates the trilio_env_osp16.yaml with the values for the datamover and datamover api containers.

The changes can be verified using the following commands.

Satellite Server

Follow this section when a Satellite Server is used for the container registry.

Pull the Trilio containers on the Red Hat Satellite using the given Red Hat registry URLs.

Populate the trilio_env_osp16.yaml with container urls for:

• Trilio Datamover container

• Trilio Datamover api container

Provide environment details in trilio-env_osp16.yaml

Provide backup target details and other necessary details in the provided environment file. This environment file will be used in the overcloud deployment to configure Trilio components. Container image names have already been populated in the preparation of the container images. Still it is recommended to verify the container URLs.

The following information are required additionally:

• Network for the datamover api

• Backup target type {nfs/s3}

• In case of NFS

  • list of NFS Shares

  • NFS options

• In case of S3

  • s3 type {amazon_s3/ceph_s3}

  • Access key

  • Secret key

  • S3 Region name

  • S3 Bucket name

  • S3 Endpoint URL

  • S3 SSL Enabled {true/false}

  • S3 SSL Cert

Deploy overcloud with trilio environment

Use the following heat environment file and roles data file in overcloud deploy command:

1. trilio_env_osp16.yaml

2. roles_data.yaml

To include new environment files use '-e' option and for roles data file use '-r' option. An example overcloud deploy command is shown below:

Verify deployment

On Controller node

Make sure Trilio dmapi and horizon containers are in a running state and no other Trilio container is deployed on controller nodes.

On Compute node

Make sure Trilio datamover container is in running state and no other Trilio container is deployed on compute nodes.

On the node with Horizon service

Make sure horizon container is in running state. Please note that 'Horizon' container is replaced with Trilio Horizon container. This container will have latest OpenStack horizon + Trilio's horizon plugin.

Configure Trilio Appliance

Once RHOSP16.1 Installation steps have completed successfully, follow the instructions below to now configure the Trilio Appliance.

Change the nova user id on the Trilio Nodes

In RHOSP, 'nova' user id on nova-compute docker container is set to '42436'. The 'nova' user id on the Trilio nodes need to be set the same. Do the following steps on all Trilio nodes:

1. Download the shell script that will change the user id

2. Assign executable permissions

3. Execute the script

4. Verify that 'nova' user and group id has changed to '42436'

Run the Trilio configurator

Change the workloadmgr.conf to use the right mountpoint

RHOSP16 is using a different mount point in its datamover containers than other Openstack distribution. It is necessary to adjust the mountpoint of the Trilio Nodes to match this. If the mountpoints are not getting aligned, will files created by the datamover and read by the Trilio appliance not match in their paths and backup and restore processes will fail.

Please follow these steps to align the mountpoints:

1. Edit /etc/workloadmgr/workloadmgr.conf file

2. Set parameter 'vault_data_directory' to '/var/lib/nova/triliovault-mounts'

3. create the directory for the mountpoint

4. assign the created directory to nova:nova

5. unmount the old mountpoint

6. Update the Trilio configurator

7. Restart the Trilio services

8. Verify the mountpoint has been configured correctly

Troubleshooting for overcloud deployment failures

Trilio components will be deployed using puppet scripts.

In case of the overcloud deployment failing does the following command provide the list of errors:

Further commands that can help identifying any errors.

In case of the trilio datamover api container is failing to start or stuck in restart, check these logs:

In case of Trilio datamover container failing to start or being stuck in restart, check these logs:

Cinder backend is Ceph - additional steps

Add Ceph details to configuration file

If Cinder backend is Ceph it is necessary to manually add the ceph details to tvault-contego.conf on all compute nodes.

The file can be found here: /var/lib/config-data/puppet-generated/triliodm/etc/tvault-contego/tvault-contego.conf

Add the following information:

Restart the datamover container

Trilio configuration process is using ansible scripts. Ansible, in the last few years, has grown in popularity as a preferred configuration management tool and Trilio uses ansible play books extensively to configure the Trilio cluster. To troubleshoot Trilio configuration issues, user should have basic understanding of ansible playbook output.

Ansible modules are inherently idempotent and hence Trilio configuration can run any number of times to change or reconfigure Trilio cluster.

Once the VM is booted, point your browser (Chrome or Firefox) to Trilio node IP address.

This will bring you to the Trilio Dashboard, which contains the Trilio configurator.

The user is: admin The default password is: password

Unlike previous versios of Trilio, the current version only requires you to configure the cluster once and the Trilio dashboard provides cluster wide management capability.

Uploading the OpenStack certificate bundle

OpenStack endpoints can be configured to use TLS. In such a configuration the Trilio appliance needs to trust the certificates provided by the OpenStack endpoints.

To achieve this trust it is required to upload the OpenStack certificate bundle through the OS API certificate tab of the Trilio appliance Dashboard.

The certificate bundle is located on the controller nodes of the OpenStack installation.

The default paths for each distribution are as follows:

The uploaded certificates can be verified on the Trilio appliance at the following location.

Details needed for the Trilio Appliance

Upon login into an unconfigured Trilio Appliance, the shown page is the configurator. The configurator requires some information about the Trilio Appliance, Openstack and Backup Storage.

Trilio Cluster information

The Trilio Cluster needs to be integrated into an existing environment to be able to operate correctly. This block asks for the information about the Trilio Cluster operating details.

• Controller Nodes

  • This is the list of Trilio virtual appliance IP addresses along with their hostnames.

  • Format: comma separated list with pairs combined through '='

  • Example: 172.20.4.151=tvault-104-1,172.20.4.152=tvault-104-2,172.20.4.153=tvault-104-3’

• Virtual IP Address

  • This is the Trilio cluster IP address which is mandatory

  • Format: IP/Subnet

  • Example: 172.20.4.150/24

• Name Server

  • List of nameservers, primarily used to resolve OpenStack service endpoints.

  • Format: comma separated list

  • example: 10.10.10.1,172.20.4.1

• Domain Search Order

  • The domain the Trilio Cluster will use.

  • Format: comma separated list

  • example: trilio.io,trilio.demo

• NTP Servers

  • NTP servers the Trilio Cluster will use

  • format: comma separated list

  • example: 0.pool.ntp.org,10.10.10.10

• Timezone

  • Timezone the Trilio Cluster will use internally

  • format: pre-populated list

  • example: UTC

Openstack Credentials informations

The Trilio appliance integrates with one RHV environment. This block asks for the information required to access and connect with the RHV Cluster.

• Keystone Admin URL

  • The Keystone admin endpoint mainly used during configuration

  • format: URL

  • example: https://keystone.trilio.io:35357/v3

• Keystone Public/Internal URL

  • The URL type defines which endpoint type will communicate with the Openstack endpoints

  • format: URL

  • example: https://internal.trilio.io:5000/v3

• Administrator

  • Username of an account with the domain admin role

  • format: String

  • example: admin

• Password

  • password for the user provided before

  • format: String

  • example: password

• Admin Tenant

  • The tenant to be used together with the provided user

• Region

  • Openstack Region the user and tenant are located in

  • format: String

  • example: RegionOne

• Domain ID

  • domain the provided user and tenant are located in

  • format: ID

  • exmaple: default

• Trustee Role

  • The Openstack role required to be able to use Trilio functionalities

Backup Storage Configuration information

This block is requesting the necessary information about the backup target that the Trilio installation will be used to store and read backups.

The very first field in this block decides the protocol used to connect with Backup Storage, NFS or S3.

Using the NFS protocol

• NFS Export

  • Path under which the NFS Volumes to be used can be found

  • format: comma separated list of NFS Volumes paths

  • example: 10.10.2.20:/upstream,10.10.5.100:/nfs2

• NFS Options

  • NFS options used by the Trilio Cluster when mounting the NFS Exports

  • format: NFS options

  • example: nolock,soft,timeo=180,intr,lookupcache=none

Using the S3 protocol

• S3 Compatible

  • Switch between Amazon and Ceph

  • format: predefined list

  • example: Amazon S3

• Access Key

  • Access Key necessary to login into the S3 storage

  • format: access key

  • example: SFHSAFHPFFSVVBSVBSZRF

• Secret Key

  • Secret Key necessary to login into the S3 storage

  • format: secret key

  • example: bfAEURFGHsnvd3435BdfeF

• Region

  • Configured Region for the S3 Bucket

  • format: String

  • example: us-east

• Bucket Name

  • Name of the bucket to be used as Backup target

  • format: string

  • example: Trilio-backup

• (CEPH S3 ONLY) Endpoint URL

  • URL to be used to reach and access the provided S3 compatible storage

  • format: URL

  • example: objects.trilio.io

Workload Import

If you are upgrading either from older versions of Trilio or reinstalling the appliance for maintenance reasons, please check this box during the configuration. Trilio is a stateless appliance and all the state is securely saved on the NFS/S3 storage. So, during the upgrade process, the user will need to import all backup job records back from the NFS/S3 storage to the appliance MySQL database. By checking this box, the configuration automatically imports all backup records.

Advanced settings

At the end of the configuratorformulars is the option to activate the advanced settings. Activating this option does provide the possibility to configure the Keystone endpoints used for the Datamover API and Trilio.

Starting the configurator

Once all entries have been set and all validations are error free the configurator can be started.

• Click Finish

• Reconfirm in the pop-up that you want to start the configuration

• Wait for the configurator to finish

The configurator is using Ansible and a few Trilio internal API calls. After each configuration block or after the configurator finished it is possible to visit the ansible output.

At the end of a successful configuration does the configurator forward to the set VIP.

After the Trilio VM has been configured and all components are installed can the license be applied.

The license can be applied either through the admin-tab in Horizon or the CLI

Apply license through Horizon

To apply the license through Horizon follow these steps:

1. Login to Horizon using admin user.

2. Click on Admin Tab.

3. Navigate to Backups-Admin

4. Navigate to Trilio

5. Navigate to License

6. Click "Update License"

7. Click "Choose File"

8. choose license-file on client system

9. click "Apply"

Apply license through CLI

Generic Pre-requisites

1. Please ensure the following points before starting the upgrade process:

   1. No snapshot OR restore to be running.

   2. Global job scheduler should be disabled.

2. The mentioned gemfury repository should be accessible from TVault VM.

Rolling upgrade on Openstack Ansible

Repo Updates

Add gemfury repo on each dmapi, horizon containers & compute node(s) to get updated packages.

Create a file /etc/apt/sources.list.d/fury.list and add below line in it.

deb [trusted=yes] https://apt.fury.io/triliodata-4-0/ /

Use the below commands to get a list of updated packages available on the configured repositories.

apt-get update
apt list --upgradable 

Upgrade tvault-datamover-api package

• Login to dmapi container from the controller node using the below command.

lxc-attach -n controller_dmapi

• Take a backup of the following file on each dmapi container(s).

tar -czvf dmapi_config.tar.gz /etc/dmapi

• Add the gemfury repo and upgrade the dmapi package using the below command. Select the appropriate package depending on the python version used.

apt install python3-dmapi --upgrade

• Restore the backed-up config files

tar -xzvf dmapi_config.tar.gz -C /

• Now restart and check the service tvault-datamover-api on controller.

systemctl restart tvault-datamover-api
systemctl status tvault-datamover-api

Upgrade Horizon plugin

• For the Horizon plugin upgrade, we need to upgrade below two packages.

python3-tvault-horizon-plugin python3-workloadmgrclient

• Login to the horizon container from the controller node using the below command.

  Copy

  lxc-attach -n controller_horizon_container-ead7cc60

• Add the gemfury repo and upgrade the tvault-horizon-plugin & workloadmgrclient packages using the below command. Select the appropriate package depending on the python version used.

apt install python3-tvault-horizon-plugin --upgrade
apt install python3-workloadmgrclient --upgrade

• Restart the apache2 service and verify the workloadmgrclient version using the below commands.

systemctl restart apache2
workloadmgr --version

Upgrade the tvault-contego package

NFS Storage Backend

• Take a backup of following file on each compute node(s) for nfs storage backend.

tar -czvf  contego_config.tar.gz /etc/tvault-contego/ 

• Upgrade the tvault-contego package using below command.Select the appropriate package depending on python version used.

apt install python3-tvault-contego --upgrade

• Restore the backed-up config files

 tar -xzvf  contego_config.tar.gz -C /

• Now restart and check the service tvault-contego on compute node(s).

systemctl restart tvault-contego
systemctl status tvault-contego

Please make sure all configuration files are unchanged.

S3 Storage Backend

• Take a backup of following file on each compute node(s) for s3 storage backend.

tar -czvf  contego_config.tar.gz /etc/tvault-contego/ /etc/tvault-object-store/

• Upgrade the tvault-contego package using below command. Select the appropriate package depending on the python version used.

apt install python3-tvault-contego python3-s3-fuse-plugin --upgrade

Note*:- If you get prompt while installing, choose the default selection.

• Restore the backed-up config files

 tar -xzvf  contego_config.tar.gz -C /

• Now restart and check the service tvault-contego on compute node(s).

systemctl restart tvault-object-store
systemctl restart tvault-contego 
systemctl status tvault-object-store
systemctl status tvault-contego

Please make sure all configuration files are unchanged.

Rolling upgrade on Kolla Openstack

Upgrade tvault-datamover-api package (trilio_datamover_api Container)

Container trilio_datamover_api needs to be redeployed. Follow below steps on all controller nodes :

• Backup Existing conf files/folders

Copy folders /etc/kolla/nova-api & /etc/kolla/trilio-datamover-api/ to some local location.
Eg. cp -r /etc/kolla/nova-api $HOME/4.0-BK/

• Stop and Remove existing container trilio_datamover_api

docker stop trilio_datamover_api
docker rm trilio_datamover_api

• Pull trilio_datamover_api container image:

## Login to docker using credentials: triliodocker/triliopassword
docker login docker.io

## Pull Trilio Datamover Api container using following command. You need to edit OS_PLATFORM, TVAULT_VERSION,
 OPENSTACK_RELEASE_NAME.

<OS_PLATFORM> could be ubuntu/centos.
<OPENSTACK_RELEASE_NAME> could would be train
<TVAULT_VERSION>    : For exact GA build version check section 1. i) Plan for deployment

docker pull docker.io/trilio/<OS_PLATFORM>-source-trilio-datamover-api:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME>

Example command for train openstack on ubuntu platform with triliovault 4.0 release:
docker pull docker.io/trilio/ubuntu-source-trilio-datamover-api:4.0.106-train

• Run datamover api container.

You need to edit OS_PLATFORM, TVAULT_VERSION, OPENSTACK_RELEASE_NAME. 
<OS_PLATFORM> could be ubuntu/centos.
<OPENSTACK_RELEASE_NAME> would be train
<TVAULT_VERSION> : For exact TVAULT build version check section 1. i) Plan for deployment

docker run --network host --name trilio_datamover_api -d --restart always -v /etc/kolla/trilio-datamover-api/nova.conf:/etc/nova/nova.conf \
-v /etc/kolla/trilio-datamover-api/dmapi.conf:/etc/dmapi/dmapi.conf \
-v /var/log/kolla/:/var/log/kolla/ \
trilio/<OS_PLATFORM>-source-trilio-datamover-api:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME> /var/lib/kolla/venv/bin/python /usr/bin/dmapi-api 

• Verify deployment

Make sure 'trilio_datamover_api' named container running well and not in 'restarting' state on all controller nodes. 
'docker ps' command can be used for the same.

# docker ps | grep trilio_datamover_api
e603ea78094f  trilio/ubuntu-source-trilio-datamover-api:4.0.106-train   "dumb-init --single-…"   5 minutes ago Up 5 minutes
trilio_datamover_api

Upgrade Horizon Plugin Package

• Login to horizon container (on controller node)

docker exec -it horizon /bin/bash

• Add gemfury repo on horizon to get updated packages.

  Create a file /etc/apt/sources.list.d/fury.list and add below line in it.

  Copy

  deb [trusted=yes] https://apt.fury.io/triliodata-4-0/ /

  Use below commands to get list of updated packages available on repo's.

  Copy

  apt-get update
  apt list --upgradable

• For Horizon plugin upgrade, following packages need to be upgraded.

python3-tvault-horizon-plugin python3-workloadmgrclient

• Run following commands to upgrade the tvault-horizon-plugin & workloadmgrclient packages. Select the appropriate package depending on python version used.

dpkg -l | grep -i tvault
  ii  python3-tvault-horizon-plugin  4.0.92    all   tVault plugin for  OpenStack Horizon Dashboard
  ii  python3-workloadmgrclient      4.0.92   all  tVault Workload Manager Client

apt install python3-tvault-horizon-plugin --upgrade
apt install python3-workloadmgrclient --upgrade

dpkg -l | grep -i tvault
   ii  python3-tvault-horizon-plugin  4.0.115                             all          tVault plugin for  OpenStack Horizon Dashboard
   ii  python3-workloadmgrclient      4.0.115                             all          tVault Workload Manager Client

• Restart docker container (from controller node) and verify the workloadmgrclient version (inside horizon container).

docker restart horizon
docker exec -it horizon /bin/bash
workloadmgr --version

Upgrade Tvault-Contego (trilio_datamover Container)

Container trilio_datamover needs to be redeployed. Follow below steps on all Compute nodes :

• Backup Existing conf files/folders

Copy folder /etc/kolla/trilio-datamover to some local location.
Eg. cp -r /etc/kolla/trilio-datamover $HOME/4.0-BK/

• Stop & remove existing trilio_datamover container

docker stop trilio_datamover
docker rm trilio_datamover

• Pull Trilio Datamover container image using the following command:

## Login to docker using credentials: triliodocker/triliopassword
docker login docker.io

## Pull Trilio Datamover Api container using following command. You need to edit OS_PLATFORM, TVAULT_VERSION,
OPENSTACK_RELEASE_NAME.

<OS_PLATFORM> could be ubuntu/centos.
<OPENSTACK_RELEASE_NAME> would be train
<TVAULT_VERSION> : For exact TVAULT build version check section 1. i) Plan for deployment

docker pull docker.io/trilio/<OS_PLATFORM>-source-trilio-datamover:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME>

Example command for train openstack on ubuntu platform with Trilio 4.0 release:
docker pull docker.io/trilio/ubuntu-source-trilio-datamover:4.0.106-train

• Run datamover container.

-- In following docker run command you need to edit OS_PLATFORM, TVAULT_VERSION,
OPENSTACK_RELEASE_NAME.

<OS_PLATFORM> could be ubuntu/centos.
<OPENSTACK_RELEASE_NAME> would be train
<TVAULT_VERSION> : For exact TVAULT build version check section 1. i) Plan for deployment

Note: If your cloud does not use 'ceph' storage for nova/cinder, remove '/etc/ceph' volume mount option 
from below commands.

##If backup storage type is 'NFS'
docker run --privileged --network host --name trilio_datamover -d \
--restart always \
-v /etc/kolla/trilio-datamover/tvault-contego.conf:/etc/tvault-contego/tvault-contego.conf \
-v /etc/kolla/trilio-datamover/nova.conf:/etc/nova/nova.conf \
-v /dev:/dev:rw \
-v /etc/ceph:/etc/ceph:ro \
-v nova_compute:/var/lib/nova/:rw \
-v /var/log/kolla/:/var/log/kolla/ \
-v iscsi_info:/etc/iscsi:rw -v /var/run/libvirt \
trilio/<OS_PLATFORM>-source-trilio-datamover:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME> /opt/tvault/start_datamover_nfs

• Verify Deployment of trilio_datamover

Make sure 'trilio_datamover' named container running well and not in 'restarting' state on all compute nodes.
   'docker ps' command can be used for the same.
# docker ps | grep trilio_datamover
3e62813e5ade        trilio/ubuntu-source-trilio-datamover:4.0.115-train   "dumb-init --single-…"   7 hours ago         Up 7 hours                              trilio_datamover

Generic Pre-requisites

1. Please ensure following points before starting the upgrade process:

   1. No snapshot OR restore to be running.

   2. Global job scheduler should be disabled.

2. The mentioned gemfury repository should be accessible from TVault VM.

Rolling upgrade on Openstack Ansible

Repo Updates

Add trilio repo on each dmapi, horizon containers & compute node(s) to get updated packages.

Modify the file /etc/yum.repos.d/trilio.repo and add below line in it.

[trilio]
name=Trilio Repository
baseurl=http://trilio:XpmkpMFviqSe@repos.trilio.io:8283/triliovault-4.0/yum/
enabled=1
gpgcheck=0

Use below commands to get list of updated packages available on repo's.

yum repolist
yum check-upgrade

1.2 Upgrade tvault-datamover-api package

• Login to dmapi container from controller node using below command.

  Copy

  lxc-attach -n controller_dmapi

• Take a backup of following file on each dmapi container(s).

tar -czvf dmapi_config.tar.gz /etc/dmapi

• Add the trilio repo and upgrade the dmapi package using below command.Select the appropriate package depending on python version used.

You can check installed package using below command.

yum list installed | grep dmapi
yum check-update dmapi

• Upgrade the dmapi package using below command.

yum upgrade dmapi

• Restore the backed-up config files

tar -xzvf dmapi_config.tar.gz -C /

• Now restart and check the service tvault-datamover-api on the controller.

systemctl restart tvault-datamover-api
systemctl status tvault-datamover-api

Upgrade Horizon Plugin Package

• We need to upgrade below two packages to upgrade horizon plugin.

tvault-horizon-plugin workloadmgrclient

• Login to horizon container from controller node using below command.

  Copy

  lxc-attach -n controller_horizon_container-3990523e

• Add the trilio repo and upgrade the tvault-horizon-plugin & workloadmgrclient packages using below command.Select the appropriate package depending on python version used.

yum list installed | grep trilio

• Depending on the output of above command upgrade appropriate packages.

yum upgrade tvault-horizon-plugin workloadmgrclient

• Restart the httpd service and verify the workloadmgrclient version using below commands.

systemctl restart httpd
workloadmgr --version

Upgrade Tvault-Contego Package

NFS Storage Backend

• Take a backup of following file on each compute node(s) for nfs storage backend.

tar -czvf  contego_config.tar.gz /etc/tvault-contego/ 

• Add the trilio repo and upgrade the tvault-contego package using below command.Select the appropriate package depending on python version used.

yum list installed | grep tvault

• Depending on the output of above command upgrade appropriate packages.

yum upgrade tvault-contego

• Restore the backed-up config files

 tar -xzvf  contego_config.tar.gz -C /

• Now restart the service tvault-contego. Verify the status of the service and check the mount point.

systemctl restart tvault-contego
systemctl status tvault-contego
df -h

S3 Storage Backend

• Take a backup of following file on each compute node(s) for s3 storage backend.

tar -czvf  contego_config.tar.gz /etc/tvault-contego/ /etc/tvault-object-store/

• Upgrade the tvault-contego and s3fuse-plugin packages using below command. Select the appropriate package depending on the python version used.

 yum upgrade tvault-contego python-s3fuse-plugin-cent7

Note*:- If you get prompt while installing, choose the default selection.

• Restore the backed-up config files

 tar -xzvf  contego_config.tar.gz -C /

• Now restart and check the service tvault-contego on compute node(s).

systemctl restart tvault-object-store
systemctl restart tvault-contego 
systemctl status tvault-object-store
systemctl status tvault-contego
df -h

Rolling upgrade on Kolla Openstack

Upgrade tvault-datamover-api package (trilio_datamover_api Container)

Container trilio_datamover_api needs to be redeployed. Follow below steps on all controller nodes :

• Backup Existing conf files/folders

Copy folders /etc/kolla/nova-api & /etc/kolla/trilio-datamover-api/ to some local location.
Eg. cp -r /etc/kolla/nova-api $HOME/4.0-BK/

• Stop and Remove existing container trilio_datamover_api

docker stop trilio_datamover_api
docker rm trilio_datamover_api

• Pull trilio_datamover_api container image

  Copy

  ## Login to docker using credentials: triliodocker/triliopassword
  docker login docker.io
  
  ## Pull Trilio Datamover Api container using following command. You need to edit OS_PLATFORM, TVAULT_VERSION,
   OPENSTACK_RELEASE_NAME.
  
  <OS_PLATFORM> could be ubuntu/centos.
  <OPENSTACK_RELEASE_NAME> could would be train
  <TVAULT_VERSION>    : For exact GA build version check section 1. i) Plan for deployment
  
  docker pull docker.io/trilio/<OS_PLATFORM>-source-trilio-datamover-api:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME>
  
  Example command for train openstack on ubuntu platform with triliovault 4.0 release:
  docker pull docker.io/trilio/centos-source-trilio-datamover-api:4.0.115-train

• Run datamover api container.

  Copy

  You need to edit OS_PLATFORM, TVAULT_VERSION, OPENSTACK_RELEASE_NAME. 
  <OS_PLATFORM> could be ubuntu/centos.
  <OPENSTACK_RELEASE_NAME> would be train
  <TVAULT_VERSION> : For exact TVAULT build version check section 1. i) Plan for deployment
  
  docker run --network host --name trilio_datamover_api -d --restart always -v /etc/kolla/trilio-datamover-api/nova.conf:/etc/nova/nova.conf \
  -v /etc/kolla/trilio-datamover-api/dmapi.conf:/etc/dmapi/dmapi.conf \
  -v /var/log/kolla/:/var/log/kolla/ \
  trilio/<OS_PLATFORM>-source-trilio-datamover-api:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME> /var/lib/kolla/venv/bin/python /usr/bin/dmapi-api 

• Verify deployment

  Copy

  Make sure 'trilio_datamover_api' named container running well and not in 'restarting' state on all controller nodes. 
  'docker ps' command can be used for the same.
  
  # docker ps | grep trilio_datamover_api
  e603ea78094f  trilio/centos-source-trilio-datamover-api:4.0.106-train   "dumb-init --single-…"   5 minutes ago Up 5 minutes
  trilio_datamover_api

Upgrade Tvault-Contego (trilio_datamover Container)

Container trilio_datamover needs to be redeployed. Follow below steps on all Compute nodes :

• Backup Existing conf files/folders

Copy folder /etc/kolla/trilio-datamover to some local location.
Eg. cp -r /etc/kolla/trilio-datamover $HOME/4.0-BK/

• Stop & remove existing trilio_datamover container

docker stop trilio_datamover
docker rm trilio_datamover

• Pull Trilio Datamover container image using the following command:

  Copy

  ## Login to docker using credentials: triliodocker/triliopassword
  docker login docker.io
  
  ## Pull Trilio Datamover Api container using following command. You need to edit OS_PLATFORM, TVAULT_VERSION,
  OPENSTACK_RELEASE_NAME.
  
  <OS_PLATFORM> could be ubuntu/centos.
  <OPENSTACK_RELEASE_NAME> would be train
  <TVAULT_VERSION> : For exact TVAULT build version check section 1. i) Plan for deployment
  
  docker pull docker.io/trilio/<OS_PLATFORM>-source-trilio-datamover:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME>
  
  Example command for train openstack on ubuntu platform with Trilio 4.0 release:
  docker pull docker.io/trilio/centos-source-trilio-datamover:4.0.106-train

• Run datamover container.

  Copy

  -- In following docker run command you need to edit OS_PLATFORM, TVAULT_VERSION,
  OPENSTACK_RELEASE_NAME.
  
  <OS_PLATFORM> could be ubuntu/centos.
  <OPENSTACK_RELEASE_NAME> would be train
  <TVAULT_VERSION> : For exact TVAULT build version check section 1. i) Plan for deployment
  
  Note: If your cloud does not use 'ceph' storage for nova/cinder, remove '/etc/ceph' volume mount option 
  from below commands.
  
  ##If backup storage type is 'NFS'
  docker run --privileged --network host --name trilio_datamover -d \
  --restart always \
  -v /etc/kolla/trilio-datamover/tvault-contego.conf:/etc/tvault-contego/tvault-contego.conf \
  -v /etc/kolla/trilio-datamover/nova.conf:/etc/nova/nova.conf \
  -v /dev:/dev:rw \
  -v /etc/ceph:/etc/ceph:ro \
  -v nova_compute:/var/lib/nova/:rw \
  -v /var/log/kolla/:/var/log/kolla/ \
  -v iscsi_info:/etc/iscsi:rw -v /var/run/libvirt \
  trilio/<OS_PLATFORM>-source-trilio-datamover:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME> /opt/tvault/start_datamover_nfs

• Verify Deployment of trilio_datamover

Make sure 'trilio_datamover' named container running well and not in 'restarting' state on all compute nodes.
   'docker ps' command can be used for the same.
# docker ps | grep trilio_datamover
3e62813e5ade        trilio/centos-source-trilio-datamover:4.0.106-train   "dumb-init --single-…"   7 hours ago         Up 7 hours                              trilio_datamover

Upgrade Horizon Plugin Package

• Login to horizon container

docker exec -it horizon /bin/bash

• Add trilio repo on each controller(s) to get updated packages.

  • Create a file /etc/yum.repos.d/trilio.repo and add below line in it.

    Copy

    [trilio]
    name=Trilio Repository
    baseurl=http://trilio:XpmkpMFviqSe@repos.trilio.io:8283/triliovault-4.0/yum/
    enabled=1
    gpgcheck=0

• Use below commands to get list of updated packages available on repo's.

  Copy

  yum repolist
  yum check-upgrade

• For Horizon plugin upgrade, following packages need to be upgraded.

tvault-horizon-plugin workloadmgrclient

• Select the appropriate package depending on python version used.

 yum list installed | grep trilio

• Depending on the output of above command upgrade appropriate packages

yum upgrade tvault-horizon-plugin workloadmgrclient

• Restart docker container (from controller node) and verify the workloadmgrclient version (inside horizon container).

docker restart horizon
docker exec -it horizon /bin/bash
workloadmgr --version

RedHat Director

Please refer https://triliodata.atlassian.net/wiki/spaces/TRIL/pages/2055864321/Upgrade+of+Trilio+components+on+RHOSP

It is possible to configure Cinder to have multiple configurations and keyrings for CEPH.

In this case, the Trilio Datamover file needs to be extended with the CEPH information.

For Trilio to be able to work in such an environment it is required to put copies of each of these configurations and keyrings into a separate directory, which is then made known to the Trilio Datamover inside a [ceph] block in the tvault-contego.conf.

A tvault-contego.conf file with the extended [ceph] block would look like this.

[DEFAULT]

vault_storage_type = nfs
vault_storage_nfs_export = 192.168.1.34:/mnt/tvault/tvm5
vault_storage_nfs_options = nolock,soft,timeo=180,intr,lookupcache=none

vault_data_directory_old = /var/triliovault
vault_data_directory = /var/trilio/triliovault-mounts
log_file = /var/log/kolla/triliovault-datamover/tvault-contego.log
debug = False
verbose = True
max_uploads_pending = 3
max_commit_pending = 3

dmapi_transport_url = rabbit://openstack:BtsxtLmLM9EvMBduewZOX23rNusiqZqCbhihGIib@192.168.0.51:5672,openstack:BtsxtLmLM9EvMBduewZOX23rNusiqZqCbhihGIib@192.168.0.52:5672,openstack:BtsxtLmLM9EvMBduewZOX23rNusiqZqCbhihGIib@192.168.0.53:5672//

[dmapi_database]
connection = mysql+pymysql://dmapi:x5nvYXnAn4rXmCHfWTK8h3wwShA4vxMq3gE2jH57@kolla-victoriaR-internal.triliodata.demo:3306/dmapi

[libvirt]
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = volumes

[ceph]
keyring_ext = .volumes.keyring
ceph_dir = /etc/ceph/directory1/,/etc/ceph/directory2/

[contego_sys_admin]
helper_command = sudo /usr/bin/privsep-helper

[conductor]
use_local = True

[oslo_messaging_rabbit]
ssl = false

[cinder]
http_retries = 10

The Red Hat Openstack Platform Director is the supported and recommended method to deploy and maintain any RHOSP installation.

Trilio is integrating natively into the RHOSP Director. Manual deployment methods are not supported for RHOSP.

Prepare for deployment

Depending whether the RHOSP environment is already installed or is getting installed for the first time different steps are done to be able to deploy Trilio.

Clone triliovault-cfg-scripts repository and upload Trilio puppet module

The following commands upload the Trilio puppet module to the overcloud. The actual upload happens upon the next deployment.

cd /home/stack
git clone -b <branch> https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/redhat-director-scripts/
# ./upload_puppet_module.sh
Creating tarball...
Tarball created.
Creating heat environment file: /home/stack/.tripleo/environments/puppet-modules-url.yaml
Uploading file to swift: /tmp/puppet-modules-8Qjya2X/puppet-modules.tar.gz
+-----------------------+---------------------+----------------------------------+
| object                | container           | etag                             |
+-----------------------+---------------------+----------------------------------+
| puppet-modules.tar.gz | overcloud-artifacts | 368951f6a4d39cfe53b5781797b133ad |
+-----------------------+---------------------+----------------------------------+

## Make sure that
ls -ll /home/stack/.tripleo/environments/puppet-modules-url.yaml

Update overcloud roles data file to include Trilio services

Trilio contains of multiple services. Add these services to your roles_data.yaml.

Add the following services to the roles_data.yaml

Trilio Datamover Api Service

This service needs to share the same role as the nova-api service. In case of the pre-defined roles will the nova-api service run on the role Controller. In case of custom defined roles, it is necessary to use the role the nova-api service is using.

Add the following line to the identified role:

'OS::TripleO::Services::TrilioDatamoverApi'

Trilio Datamover Service

This service needs to share the same role as the nova-compute service. In case of the pre-defined roles will the nova-compute service run on the role Compute. In case of custom defined roles, it is necessary to use the role the nova-compute service is using.

Add the following line to the identified role:

'OS::TripleO::Services::TrilioDatamover' 

Prepare Trilio container images

Trilio containers are pushed to 'RedHat Container Registry'. Registry URL: 'registry.connect.redhat.com'. Container pull urls are given below.

Trilio Datamover container: registry.connect.redhat.com/trilio/trilio-datamover:<container-tag> Trilio Datamover Api Container: registry.connect.redhat.com/trilio/trilio-datamover-api:<container-tag> Trilio horizon plugin: registry.connect.redhat.com/trilio/trilio-horizon-plugin:<container-tag>

There are three registry methods available in RedHat Openstack Platform.

1. Remote Registry

2. Local Registry

3. Satellite Server

Remote Registry

Follow this section when 'Remote Registry' is used.

For this method it is not necessary to pull the containers in advance. It is only necessary to populate the trilio_env.yaml and overcloud_images.yaml with the RedHat Registry URLs to the right containers.

Populate the trilio_env.yaml with container urls for:

• Trilio Datamover container

• Trilio Datamover api container

$ grep '<container-tag>' trilio_env.yaml
   DockerTrilioDatamoverImage: registry.connect.redhat.com/trilio/trilio-datamover:<container-tag>
   DockerTrilioDmApiImage: registry.connect.redhat.com/trilio/trilio-datamover-api:<container-tag>

In the overcloud_images.yaml replace the standard Horizon Container with the Trilio Horizon container. This is necessary to gain access to the Trilio Horizon Dashboard.

$ grep 'trilio' /home/stack/templates/overcloud_images.yaml
   DockerHorizonImage: registry.connect.redhat.com/trilio/trilio-horizon-plugin:<container-tag>

Local Registry

Follow this section when 'local registry' is used on the undercloud.

In this case it is necessary to pull the Trilio containers to the undercloud registry. Trilio provides shell scripts which will pull the containers from 'registry.connect.redhat.com' to the undercloud and updates the trilio_env.yaml with the values for the datamover and datamover api containers.

## Script argument details: 
./prepare_trilio_images.sh <undercloud_ip> <container_tag>

## Example execution:
./prepare_trilio_images.sh 192.168.13.34 4.0.92-rhosp13

The changes can be verified in the trilio_env.yaml.

[stack@undercloud$ grep '4.0.92-rhosp13' trilio_env.yaml
   DockerTrilioDatamoverImage: 192.168.122.10:8787/trilio/trilio-datamover:4.0.92-rhosp13
   DockerTrilioDmApiImage: 192.168.122.10:8787/trilio/trilio-datamover-api:4.0.92-rhosp13

In the overcloud_images.yaml replace the standard Horizon Container with the Trilio Horizon container. This is necessary to gain access to the Trilio Horizon Dashboard.

$ grep 'trilio' /home/stack/templates/overcloud_images.yaml
  DockerHorizonImage: 192.168.122.10:8787/trilio/trilio-horizon-plugin:4.0.92-rhosp13

Satellite Server

Follow this section when a Satellite Server is used for the container registry.

Pull the Trilio containers on the Red Hat Satellite using the given Red Hat registry URLs.

Populate the trilio_env.yaml with container urls for:

• Trilio Datamover container

• Trilio Datamover api container

$ grep '4.0.92-rhosp13' trilio_env.yaml
   DockerTrilioDatamoverImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover:4.0.92-rhosp13
   DockerTrilioDmApiImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover-api:4.0.92-rhosp13

In the overcloud_images.yaml replace the standard Horizon Container with the Trilio Horizon container. This is necessary to gain access to the Trilio Horizon Dashboard.

$ grep 'trilio' /home/stack/templates/overcloud_images.yaml
   DockerHorizonImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-horizon-plugin:4.0.92-rhosp13

Provide environment details in trilio-env.yaml

Provide backup target details and other necessary details in the provided environment file. This environment file will be used in the overcloud deployment to configure Trilio components. Container image names have already been populated in the preparation of the container images. Still it is recommended to verify the container URLs.

The following information are required additionally:

• Network for the datamover api

• Backup target type {nfs/s3}

• In case of NFS

  • list of NFS Shares

  • NFS options

• In case of S3

  • s3 type {amazon_s3/ceph_s3}

  • Access key

  • Secret key

  • S3 Region name

  • S3 Bucket name

  • S3 Endpoint URL

  • S3 SSL Enabled {true/false}