1 of 8

Installing Trilio Components

Once the Trilio VM or the Cluster of Trilio VMs has been spun, the installation process can begin. This process contains the following steps:

Install the Trilio dm-api service on the control plane.
Install the Trilio datamover service on the compute plane.
Install the Trilio Horizon plugin into the Horizon service.

How these steps look in detail is dependent on the Openstack distribution that Trilio is installed in. Each supported Openstack distribution has its own deployment tools. Trilio integrates into these deployment tools to provide a native integration from the beginning to the end.

Installing on RHOSP

The is the supported and recommended method to deploy and maintain any RHOSP installation. Trilio integrates natively into the RHOSP Director and manual deployment methods are not supported for RHOSP.

1. Prepare for deployment

1.1] Select 'backup target' type

Backup target storage is used to store backup images taken by Trilio and also associated configuration needs. The following backup target types are supported by Trilio:

Backup Target Types

Required Configuration

1.2] Clone triliovault-cfg-scripts repository

The overcloud-deploy command must already have been run successfully prior to this point and overcloud should be available. Perform the following steps for 'undercloud' node on an existing RHOSP environment:

All commands need to be run as user 'stack' on undercloud node

RHOSP 16.0 is not supported anymore as RedHat has officially stopped supporting it. However, Trilio maintained it for some time and stopped the support from 4.1HF11 onwards. The latest hotfix available for RHOSP16.0 is 41.HF10. Reach out to the Support team for any help.

Ensure that the Trilio appliance connected to this installation is on the latest Hotfix version. Failure to ensure this may lead to your installation not working as expected.
1. Refer to this doc :
Run the following command to clone the triliovault-cfg-scripts github repository:

If your backup target type is 'Ceph-based S3' with SSL, skip this step. Otherwise, access the Red Hat Director scripts according to the RHOSP version being used:
- RHOSP 13 - cd triliovault-cfg-scripts/redhat-director-scripts/rhosp13/
- RHOSP 16.1 - cd triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/
- RHOSP 16.2 - cd triliovault-cfg-scripts/redhat-director-scripts/rhosp16.2/
If your backup target is Ceph S3 with SSL and your SSL certificates are self-signed or authorized by private CA, you must provide the CA chain certificate to validate the SSL requests. Otherwise, skip this step. To do this:
1. Rename your CA chain cert file to s3-cert.pem.
2. Copy the renamed file into the following directory: triliovault-cfg-scripts/redhat-director-scripts/redhat-director-scripts/<RHOSP_RELEASE_Directory>/puppet/trilio/filesIf your overcloud deploy command uses any other deploy artifact through an environment file, then you must merge Trilio deploy artifact url and your url in a single file.
3. Then access the Red Hat Director scripts according to the version being used:
  - RHOSP 13 - cp s3-cert.pem /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp13/puppet/trilio/files/
  - RHOSP 16.1 - cp s3-cert.pem /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/puppet/trilio/files/
  - RHOSP 16.2 - cp s3-cert.pem /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.2/puppet/trilio/files/

From this point onwards in the documentation, only the following path will be used for examples: cd triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/

2] Upload Trilio-Puppet module

RHOSP 16.1

The following commands upload the Trilio puppet module to the overcloud registry. The upload only happens upon the next deployment.

Step 1: -

The output of the above command looks like the following.

Trilio puppet module is uploaded to overcloud as a swift deploy artifact with heat resource name 'DeployArtifactURLs'.

Step 2: - Check Trilio's Puppet module artifact file and ensure that it looks like the following:

Step 3: -

Firstly, check to make sure that your overcloud deploy environment files uses deploy artifacts. To do this check string DeployArtifactURLs in your environment files (only those mentioned in the overcloud deploy command with -e option). If you find any environment file with the -e option, then your deploy command is using deploy artifacts.
If your deploy command is using deploy artifact, you must merge all deploy artifacts in a single file. For example, if your artifact file path is /home/stack/templates/user-artifacts.yaml, then perform the following steps to merge both urls in single file, which is passed to the overcloud deploy command with the -e option.

3] Update overcloud roles data file to include Trilio services

Trilio contains multiple services. Add these services to your roles_data.yaml.

In the case of uncustomized roles_data.yaml can the default file be found on the undercloud at:

/usr/share/openstack-tripleo-heat-templates/roles_data.yaml

Add the following services to the roles_data.yaml

All commands need to be run as user 'stack'

3.1] Add Trilio Datamover Api Service to role data file

This service needs to share the same role as the keystone and database service. In case of the pre-defined roles will these services run on the role Controller. In case of custom-defined roles, it is necessary to use the same role where OS::TripleO::Services::Keystone service installed.

Add the following line to the identified role:

3.2] Add Trilio Datamover Service to role data file

This service needs to share the same role as the nova-compute service. In case of the pre-defined roles will the nova-compute service run on the role Compute. In case of custom-defined roles, it is necessary to use the role that the nova-compute service is using.

Add the following line to the identified role:

4] Prepare Trilio container images

All commands need to be run as user 'stack'

Trilio containers are pushed to 'RedHat Container Registry'. Registry URL: registry.connect.redhat.com Container pull URLs are given below.

Please note that using the hotfix containers requires that the Trilio Appliance is getting upgraded to the desired hotfix level as well.

Refer to the word <HOTFIX-TAG-VERSION> as 4.1.94-hotfix-16 in the below sections

RHOSP 13

RHOSP 16.1

RHOSP 16.2

There are three registry methods available in the RedHat OpenStack Platform.

Remote Registry
Local Registry
Satellite Server

4.1] Remote Registry

Follow this section when 'Remote Registry' is used.

In this method, container images get downloaded directly on overcloud nodes during overcloud deploy/update command execution. User can set the remote registry to the RedHat registry or any other private registry that he wants to use. The user needs to provide credentials for the registry in containers-prepare-parameter.yaml file.

Make sure other OpenStack service images are also using the same method to pull container images. If it's not the case you can not use this method.
Populate containers-prepare-parameter.yaml with content like following. Important parameters are 'push_destination: false', ContainerImageRegistryLogin: true and registry credentials. Trilio container images are published to the registry registry.connect.redhat.com Credentials of registry 'registry.redhat.io' will work for registry.connect.redhat.com registry too.

Note: This file - containers-prepare-parameter.yaml

Note: File 'containers-prepare-parameter.yaml' gets created as output of command 'openstack tripleo container image prepare'. Refer above document by RedHat

3. Make sure you have network connectivity to the above registries from all overcloud nodes. Otherwise, image pull operation will fail.

4. Populate the trilio_env.yaml with Trilio container image URLs:

Trilio Datamover container
Trilio Datamover API container
Trilio Horizon Plugin

trilio_env.yaml will be available in

4.2] Local Registry

Follow this section when 'local registry' is used on the undercloud.

In this case, it is necessary to push the Trilio containers to the undercloud registry. Trilio provides shell scripts that will pull the containers from 'registry.connect.redhat.com' and push them to the undercloud and update the trilio_env.yaml.

RHOSP13

Verify the changes

RHOSP16.1

Verify the changes:

RHOSP16.2

Verify the changes

The changes can be verified using the following commands.

4.3] Red Hat Satellite Server

Follow this section when a Satellite Server is used for the container registry.

Populate the trilio_env.yaml with container urls.

RHOSP 13

RHOSP16.1

RHOSP16.2

5] Provide environment details in trilio-env.yaml

Provide backup target details and other necessary details in the provided environment file. This environment file will be used in the overcloud deployment to configure Trilio components. Container image names have already been populated in the preparation of the container images. Still it is recommended to verify the container URLs.

The following information are required additionally:

Network for the datamover api
datamover password
Backup target type {nfs/s3}
In case of NFS
- list of NFS Shares
- NFS options
In case of S3
- S3 type {amazon_s3/ceph_s3}
- S3 Access key
- S3 Secret key
- S3 Region name
- S3 Bucket
- S3 Endpoint URL
- S3 Signature Version
- S3 Auth Version
- S3 SSL Enabled {true/false}
- S3 SSL Cert

Use ceph_s3 for any non-aws S3 backup targets.

6. Advanced Settings/Configuration

6.1 Haproxy customized configuration for Trilio dmapi service __

The existing default haproxy configuration works fine with most of the environments. Only when timeout issues with the dmapi are observed or other reasons are known, change the configuration as described here.

Following is the haproxy conf file location on haproxy nodes of the overcloud. Trilio datamover api service haproxy configuration gets added to this file.

Trilio datamover haproxy default configuration from the above file looks as follows:

The user can change the following configuration parameter values.

To change these default values, you need to do the following steps. i) On the undercloud node, open the following file for edit (Edit <RHOSP_RELEASE> with your cloud's release information. Valid values are - rhosp13, rhosp16, rhosp16.1)

For RHOSP13

For RHOSP16.0

For RHOSP16.1

For RHOSP16.2

ii) Search the following entries and edit as required

iii) Save the changes.

7] Deploy overcloud with trilio environment

Use the following heat environment file and roles data file in overcloud deploy command:

trilio_env.yaml
roles_data.yaml
Use the correct Trilio endpoint map file as per available Keystone endpoint configuration
1. Instead of tls-endpoints-public-dns.yaml file, use environments/trilio_env_tls_endpoints_public_dns.yaml
2. Instead of tls-endpoints-public-ip.yaml file, useenvironments/trilio_env_tls_endpoints_public_ip.yaml
3. Instead of tls-everywhere-endpoints-dns.yaml file, useenvironments/trilio_env_tls_everywhere_dns.yaml

To include new environment files use '-e' option and for roles data file use '-r' option. An example overcloud deploy command is shown below:

8] Verify deployment

If the containers are in restarting state or not listed by the following command then your deployment is not done correctly. Please recheck if you followed the complete documentation.

8.1] On Controller node

Make sure Trilio dmapi and horizon containers are in a running state and no other Trilio container is deployed on controller nodes. When the role for these containers is not "controller" check on respective nodes according to configured roles_data.yaml.

Verify the haproxy configuration under:

8.2] On Compute node

Make sure Trilio datamover container is in running state and no other Trilio container is deployed on compute nodes.

8.3] On the node with Horizon service

Make sure horizon container is in running state. Please note that 'Horizon' container is replaced with Trilio Horizon container. This container will have latest OpenStack horizon + Trilio's horizon plugin.

9] Additional Steps on Trilio Appliance

9.1] Change the nova user id on the Trilio Nodes

In RHOSP, nova user id on nova-compute docker container is set to '42436'. The 'nova' user id on the Trilio nodes needs to be set the same. Do the following steps on all Trilio nodes:

Download the shell script that will change the user id
Assign executable permissions
Execute the script
Verify that nova user and group id have changed to '42436'

10] Troubleshooting for overcloud deployment failures

Trilio components will be deployed using puppet scripts.

Installing on Canonical Openstack

Trilio and Canonical have started a partnership to ensure a native deployment of Trilio using JuJu Charms.

Those JuJu Charms are publicly available as Open Source Charms.

Trilio is not providing the JuJu Charms to deploy Trilio 4.1 in Canonical Openstack. These are developed and maintained by Canonical.

Canonical Openstack doesn't require the Trilio Cluster. The required services are installed and managed via JuJu Charms.

The following charms exist:

trilio-wlm Installs and manages Trilio Controller services.
trilio-dm-api Installs and manages the Trilio Datamover API service.
trilio-data-mover Installs and manages the Trilio Datamover service.
trilio-horizon-plugin Installs and manages the Trilio Horizon Plugin.

The documentation of the charms can be found here:

Installing on Kolla Ussuri

This page lists all steps required to deploy Trilio components on Kolla-ansible deployed OpenStack cloud.

1] Plan for Deployment

Please ensure that the Trilio Appliance has been updated to the latest hotfix before continuing the installation.

Refer to the below-mentioned acceptable values for the placeholders in this document as per the Openstack environment: kolla_base_distro : ubuntu / centos triliovault_tag : 4.1.94-hotfix-13-ussuri / 4.1.94-hotfix-12-victoria

1.1] Select backup target type

Backup target storage is used to store backup images taken by Trilio and details needed for configuration:

Following backup target types are supported by Trilio. Select one of them and get it ready before proceeding to the next step.

a) NFS

Need NFS share path

b) Amazon S3

- S3 Access Key - Secret Key - Region - Bucket name

c) Other S3 compatible storage (Like, Ceph based S3)

- S3 Access Key - Secret Key - Region - Endpoint URL (Valid for S3 other than Amazon S3) - Bucket name

2] Clone Trilio Deployment Scripts

Clone triliovault-cfg-scripts GitHub repository on Kolla ansible server at '/root' or any other directory of your preference. Afterward, copy the Trilio Ansible role into the Kolla-ansible roles directory

git clone -b hotfix-13-TVO/4.1 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/kolla-ansible/

# For Centos and Ubuntu
cp -R ansible/roles/triliovault /usr/local/share/kolla-ansible/ansible/roles/

3] Hook Trilio deployment scripts to Kolla-ansible deploy scripts

3.1] Add Trilio global variables to globals.yml

## For Centos and Ubuntu
## Take backup of globals.yml
cp /etc/kolla/globals.yml /opt/

## Append Trilio global variables to globals.yml 
cat ansible/triliovault_globals.yml >> /etc/kolla/globals.yml

3.2] Add Trilio passwords to kolla passwords.yaml

Append triliovault_passwords.yml to /etc/kolla/passwords.yml. Passwords are empty. Set these passwords manually in the /etc/kolla/passwords.yml.

## For Centos and Ubuntu
## Take backup of passwords.yml
cp /etc/kolla/passwords.yml /opt/

## Append Trilio global variables to passwords.yml 
cat ansible/triliovault_passwords.yml >> /etc/kolla/passwords.yml

Edit /etc/kolla/passwords.yml, go to the end of the file and set trilio passwords.

3.3] Append Trilio site.yml content to kolla ansible’s site.yml

## For Centos and Ubuntu
## Take backup of site.yml
cp /usr/local/share/kolla-ansible/ansible/site.yml /opt/

## Append Trilio code to site.yml
cat ansible/triliovault_site.yml >> /usr/local/share/kolla-ansible/ansible/site.yml

3.4] Append triliovault_inventory.txt to your cloud’s kolla-ansible inventory file.

For example:
If your inventory file name path '/root/multinode' then use following command.

cat ansible/triliovault_inventory.txt >> /root/multinode

4] Edit globals.yml to set Trilio parameters

Edit /etc/kolla/globals.yml file to fill Trilio backup target and build details. You will find the Trilio related parameters at the end of globals.yml file. Details like Trilio build version, backup target type, backup target details, etc need to be filled out.

Following is the list of parameters that the usr needs to edit.

In the case of a different registry than docker hub, Trilio containers need to be pulled from docker.io and pushed to preferred registries.

Following are the triliovault container image URLs. Replace kolla_base_distro and triliovault_tag variables with their values.

docker.io/trilio/<kolla_base_distro>-binary-trilio-datamover-api:<triliovault_tag>
docker.io/trilio/<kolla_base_distro>-binary-trilio-datamover:<triliovault_tag>
docker.io/trilio/<kolla_base_distro>-binary-trilio-horizon-plugin:<triliovault_tag>

5] Enable Trilio Snapshot mount feature

To enable Trilio's Snapshot mount feature it is necessary to make the Trilio Backup target available to the nova-compute and nova-libvirt containers.

Edit /usr/local/share/kolla-ansible/ansible/roles/nova-cell/defaults/main.yml and find nova_libvirt_default_volumes variable. Append the Trilio mount bind /var/trilio:/var/trilio:shared to the list of already existing volumes.

For a default Kolla installation, will the variable look as follows afterward:

nova_libvirt_default_volumes:
  - "{{ node_config_directory }}/nova-libvirt/:{{ container_config_directory }}/:ro"
  - "/etc/localtime:/etc/localtime:ro"
  - "{{ '/etc/timezone:/etc/timezone:ro' if ansible_os_family == 'Debian' else '' }}"
  - "/lib/modules:/lib/modules:ro"
  - "/run/:/run/:shared"
  - "/dev:/dev"
  - "/sys/fs/cgroup:/sys/fs/cgroup"
  - "kolla_logs:/var/log/kolla/"
  - "libvirtd:/var/lib/libvirt"
  - "{{ nova_instance_datadir_volume }}:/var/lib/nova/"
  - "
{% if enable_shared_var_lib_nova_mnt | bool %}/var/lib/nova/mnt:/var/lib/nova/mnt:shared{% endif %}
"
  - "nova_libvirt_qemu:/etc/libvirt/qemu"
  - "{{ kolla_dev_repos_directory ~ '/nova/nova:/var/lib/kolla/venv/lib/python' ~ distro_python_version ~ '/site-packages/nova' if nova_dev_mode | bool else '' }
  - "/var/trilio:/var/trilio:shared"

Next, find the variable nova_compute_default_volumes in the same file and append the mount bind /var/trilio:/var/trilio:shared to the list.

After the change will the variable look for a default Kolla installation as follows:

nova_compute_default_volumes:
  - "{{ node_config_directory }}/nova-compute/:{{ container_config_directory }}/:ro"
  - "/etc/localtime:/etc/localtime:ro"
  - "{{ '/etc/timezone:/etc/timezone:ro' if ansible_os_family == 'Debian' else '' }}"
  - "/lib/modules:/lib/modules:ro"
  - "/run:/run:shared"
  - "/dev:/dev"
  - "kolla_logs:/var/log/kolla/"
  - "
{% if enable_iscsid | bool %}iscsi_info:/etc/iscsi{% endif %}"
  - "libvirtd:/var/lib/libvirt"
  - "{{ nova_instance_datadir_volume }}:/var/lib/nova/"
  - "{% if enable_shared_var_lib_nova_mnt | bool %}/var/lib/nova/mnt:/var/lib/nova/mnt:shared{% endif %}"
  - "{{ kolla_dev_repos_directory ~ '/nova/nova:/var/lib/kolla/venv/lib/python' ~ distro_python_version ~ '/site-packages/nova' if nova_dev_mode | bool else '' }}"
  - "/var/trilio:/var/trilio:shared"

In case of using Ironic compute nodes, one more entry needs to be adjusted in the same file. Find the variable nova_compute_ironic_default_volumes and append trilio mount /var/trilio:/var/trilio:shared to the list.

After the changes the variable will look like the following:

nova_compute_ironic_default_volumes:
  - "{{ node_config_directory }}/nova-compute-ironic/:{{ container_config_directory }}/:ro"
  - "/etc/localtime:/etc/localtime:ro"
  - "{{ '/etc/timezone:/etc/timezone:ro' if ansible_os_family == 'Debian' else '' }}"
  - "kolla_logs:/var/log/kolla/"
  - "{{ kolla_dev_repos_directory ~ '/nova/nova:/var/lib/kolla/venv/lib/python' ~ distro_python_version ~ '/site-packages/nova' if nova_dev_mode | bool else '' }}"
  - "/var/trilio:/var/trilio:shared"

6] Pull Trilio container images

Pull the Trilio container images from the Dockerhub based on the existing inventory file. In the example is the inventory file named multinode.

kolla-ansible -i multinode pull --tags triliovault

7] Deploy Trilio

All that is left, is to run the deploy command using the existing inventory file. In the example is the inventory file named 'multinode'.

This is just an example command. You need to use your cloud deploy command.

kolla-ansible -i multinode deploy

8] Verify Trilio deployment

Verify on the nodes that are supposed to run the Trilio containers, that those are available and healthy.

[root@controller ~]# docker ps | grep triliovault_datamover_api
f00781997bc3        trilio/centos-binary-trilio-datamover-api:<triliovualt_tag>    "dumb-init --single-…"   2 minutes ago       Up 2 minutes                            triliovault_datamover_api

[root@compute ~]# docker ps | grep triliovault_datamover
84831db5d215        trilio/centos-binary-trilio-datamover:<triliovualt_tag>     "dumb-init --single-…"   5 minutes ago       Up 4 minutes                            triliovault_datamover

[root@controller ~]# docker ps | grep horizon
f3647e0fff27        trilio/centos-binary-trilio-horizon-plugin:<triliovualt_tag>   "dumb-init --single-…"   8 minutes ago       Up 8 minutes                            horizon

9] Troubleshooting Tips

9.1 ] Check Trilio containers and their startup logs

To see all TriloVault containers running on a specific node use the docker ps command.

docker ps -a | grep trilio

To check the startup logs use the docker logs <container name> command.

docker logs trilio_datamover_api
docker logs trilio_datamover

9.2] Trilio Horizon tabs are not visible in Openstack

Verify that the Trilio Appliance is configured. The Horizon tabs are only shown when a configured Trilio appliance is available.

Verify that the Trilio horizon container is installed and in a running state.

docker ps | grep horizon

9.3] Trilio Service logs

Trilio datamover api service logs on datamover api node

/var/log/kolla/triliovault-datamover-api/dmapi.log

Trilio datamover service logs on datamover node

/var/log/kolla/triliovault-datamover/tvault-contego.log

10. Change the nova user id on the Trilio Nodes

Note: This step needs to be done on Trilio Appliance node. Not on OpenStack node.

Pre-requisite: You should have already launched Trilio appliance VM

In Kolla OpenStack distribution, nova user id on nova-compute docker container is set to '42436'. The nova user id on the Trilio nodes needs to be set the same. Do the following steps on all Trilio nodes:

Download the shell script that will change the user id
Assign executable permissions
Execute the script
Verify that nova user and group id have changed to '42436'
After this step, you can proceed to the 'Configuring Trilio' section.

## Download the shell script
$ curl -O https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/master/common/nova_userid.sh

## Assign executable permissions
$ chmod +x nova_userid.sh

## Execute the shell script to change 'nova' user and group id to '42436'
$ ./nova_userid.sh

## Ignore any errors and verify that 'nova' user and group id has changed to '42436'
$ id nova
   uid=42436(nova) gid=42436(nova) groups=42436(nova),990(libvirt),36(kvm)

Installing on Kolla Victoria

The installation of Trilio for Openstack on Kolla Victoria with Trilio 4.1 is following this procedure:

Deplo T4O-4.1 GA Appliance
Upgrade to 4.1 HF5 or higher on the appliance
Deploy Trilio components of 4.1 HF5 or higher on the Kolla Openstack Victoria
Configure the Trilio appliance

Deplo T4O-4.1 GA Appliance

Please follow to spin up the base Trilio 4.1GA appliance.

Upgrade the T4O appliance to the latest 4.1 HF

Trilio supports Kolla Victoria from 4.1HF5 onwards, so it is recommended to upgrade to the latest available hotfix on 4.1 to make deployment successful. Please follow to upgrade the appliance to the latest 4.1 Hotfix.

Deploy Trilio components of 4.1 HF11 or higher

Run the deployment of the components following using the following values:

Variable

Value

Configure the Trilio Appliance

Installing on Ansible Openstack Ussuri

Please ensure that the Trilio Appliance has been updated to the latest hotfix before continuing the installation.

Change the nova user id on the Trilio Nodes

Trilio is by default using the nova user id and group id 997:998 Ansible Openstack is not always 'nova' user id 162 on nova-compute containers. The 'nova' user id on the Trilio nodes need to be set the same as in the nova-compute containers. Do the following steps on all Trilio nodes in case of nova id not being 162:162:

Download the shell script that will change the user-id
Assign executable permissions
Edit script to use the correct nova id
Execute the script
Verify that 'nova' user and group id has changed to the desired value

curl -O https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/master/common/nova_userid.sh
chmod +x nova_userid.sh
vi nova_userid.sh  # change nova user_id and group_id to uid & gid present on compute nodes. 
./nova_userid.sh
id nova

Prepare deployment host

Clone triliovault-cfg-scripts from github repository on Ansible Host.

git clone -b <branch> https://github.com/trilioData/triliovault-cfg-scripts.git

Available values for <branch>:

Copy Ansible roles and vars to required places.

cd triliovault-cfg-scripts/
cp -R ansible/roles/* /opt/openstack-ansible/playbooks/roles/
cp ansible/main-install.yml   /opt/openstack-ansible/playbooks/os-tvault-install.yml
cp ansible/environments/group_vars/all/vars.yml /etc/openstack_deploy/user_tvault_vars.yml

In case of installing on OSA Victora edit OPENSTACK_DIST in the file /etc/openstack_/user_tvault_vars.yml to Victoria

Add Trilio playbook to /opt/openstack-ansible/playbooks/setup-openstack.ymlat the end of the file.

- import_playbook: os-tvault-install.yml

Add the following content at the end of the file /etc/openstack_deploy/user_variables.yml

# Datamover haproxy setting
haproxy_extra_services:
  - service:
      haproxy_service_name: datamover_service
      haproxy_backend_nodes: "{{ groups['dmapi_all'] | default([]) }}"
      haproxy_ssl: "{{ haproxy_ssl }}"
      haproxy_port: 8784
      haproxy_balance_type: http
      haproxy_balance_alg: roundrobin
      haproxy_timeout_client: 10m
      haproxy_timeout_server: 10m
      haproxy_backend_options:
        - "httpchk GET / HTTP/1.0\\r\\nUser-agent:\\ osa-haproxy-healthcheck"

Create the following file /opt/openstack-ansible/inventory/env.d/tvault-dmapi.yml

cat > /opt/openstack-ansible/inventory/env.d/tvault-dmapi.yml 
component_skel:
  dmapi_api:
    belongs_to:
      - dmapi_all

container_skel:
  dmapi_container:
    belongs_to:
      - tvault-dmapi_containers
    contains:
      - dmapi_api

physical_skel:
  tvault-dmapi_containers:
    belongs_to:
      - all_containers
  tvault-dmapi_hosts:
    belongs_to:
      - hosts

Edit the file /etc/openstack_deploy/openstack_user_config.yml according to the example below to set host entries for Trilio components.

#tvault-dmapi
tvault-dmapi_hosts:   # Add controller details in this section as tvault DMAPI is resides on controller nodes.
  infra-1:            # controller host name. 
    ip: 172.26.0.3    # Ip address of controller
  infra-2:            # If we have multiple controllers add controllers details in same manner as shown in Infra-2  
    ip: 172.26.0.4    
    
#tvault-datamover
tvault_compute_hosts: # Add compute details in this section as tvault datamover is resides on compute nodes.
  infra-1:            # compute host name. 
    ip: 172.26.0.7    # Ip address of compute node
  infra-2:            # If we have multiple compute nodes add compute details in same manner as shown in Infra-2
    ip: 172.26.0.8

Edit the common editable parameter section in the file /etc/openstack_deploy/user_tvault_vars.yml

Append the required details like Trilio Appliance IP address, Trilio package version, Openstack distribution, snapshot storage backend, SSL related information, etc.

The possible package versions are:

GA Trilio 4.1: 4.1.94

##common editable parameters required for installing tvault-horizon-plugin, tvault-contego and tvault-datamover-api
#ip address of TVM
IP_ADDRESS: sample_tvault_ip_address

##Time Zone
TIME_ZONE: "Etc/UTC"

#Update TVAULT package version here, we will install mentioned version plugins for Example# TVAULT_PACKAGE_VERSION: 3.3.36
TVAULT_PACKAGE_VERSION: 4.1.94  #GA build version

# Update Openstack dist code name like ussuri etc.
OPENSTACK_DIST: ussuri

#Need to add the following statement in nova sudoers file
#nova ALL = (root) NOPASSWD: /home/tvault/.virtenv/bin/privsep-helper *
#These changes require for Datamover, Otherwise Datamover will not work
#Are you sure? Please set variable to
#  UPDATE_NOVA_SUDOERS_FILE: proceed
#other wise ansible tvault-contego installation will exit
UPDATE_NOVA_SUDOERS_FILE: proceed

##### Select snapshot storage type #####
#Details for NFS as snapshot storage , NFS_SHARES should begin with "-".
##True/False
NFS: False
NFS_SHARES:
          - sample_nfs_server_ip1:sample_share_path
          - sample_nfs_server_ip2:sample_share_path

#if NFS_OPTS is empty then default value will be "nolock,soft,timeo=180,intr,lookupcache=none"
NFS_OPTS: ""

#### Details for S3 as snapshot storage
##True/False
S3: False
VAULT_S3_ACCESS_KEY: sample_s3_access_key
VAULT_S3_SECRET_ACCESS_KEY: sample_s3_secret_access_key
VAULT_S3_REGION_NAME: sample_s3_region_name
VAULT_S3_BUCKET: sample_s3_bucket
VAULT_S3_SIGNATURE_VERSION: default
#### S3 Specific Backend Configurations
#### Provide one of follwoing two values in s3_type variable, string's case should be match
#Amazon/Other_S3_Compatible
s3_type: sample_s3_type
#### Required field(s) for all S3 backends except Amazon
VAULT_S3_ENDPOINT_URL: ""
#True/False
VAULT_S3_SECURE: True
VAULT_S3_SSL_CERT: ""

###details of datamover API
##If SSL is enabled "DMAPI_ENABLED_SSL_APIS" value should be dmapi.
#DMAPI_ENABLED_SSL_APIS: dmapi
##If SSL is disabled "DMAPI_ENABLED_SSL_APIS" value should be empty.
DMAPI_ENABLED_SSL_APIS: ""
DMAPI_SSL_CERT: ""
DMAPI_SSL_KEY: ""

#### Any service is using Ceph Backend then set ceph_backend_enabled value to True
#True/False
ceph_backend_enabled: False

#Set verbosity level and run playbooks with -vvv option to display custom debug messages
verbosity_level: 3

Deploy Trilio components

Run the following commands to deploy only Trilio components in case of an already deployed Ansible Openstack.

cd /opt/openstack-ansible/playbooks

# To create Dmapi container
openstack-ansible lxc-containers-create.yml 

#To Deploy Trilio Components
openstack-ansible os-tvault-install.yml

#To configure Haproxy for Dmapi
openstack-ansible haproxy-install.yml

If Ansible Openstack is not already deployed then run the native Openstack deployment commands to deploy Openstack and Trilio Components together. An example for the native deployment command is given below:

openstack-ansible setup-infrastructure.yml --syntax-check
openstack-ansible setup-hosts.yml
openstack-ansible setup-infrastructure.yml
openstack-ansible setup-openstack.yml

Verify the Trilio deployment

Verify triliovault datamover api service deployed and started well. Run the below commands on controller node(s).

lxc-ls                                           # Check the dmapi container is present on controller node.
lxc-info -s controller_dmapi_container-a11984bf  # Confirm running status of the container

Verify triliovault datamover service deployed and started well on compute node(s). Run the following command oncompute node(s).

systemctl status tvault-contego.service
systemctl status tvault-objest-store  # If Storage backend is S3
df -h                                 # Verify the mount point is mounted on compute node(s)

Verify that triliovault horizon plugin, contegoclient, and workloadmgrclient are installed on the Horizon container.

Run the following command on Horizon container.

lxc-attach -n controller_horizon_container-1d9c055c                                   # To login on horizon container
apt list | egrep 'tvault-horizon-plugin|workloadmgrclient|contegoclient'              # For ubuntu based container
yum list installed |egrep 'tvault-horizon-plugin|workloadmgrclient|contegoclient'     # For CentOS based container

Verify that haproxy setting on controller node using below commands.

 haproxy -c -V -f /etc/haproxy/haproxy.cfg # Verify the keyword datamover_service-back is present in output.

Update to the latest hotfix

After the deployment has been verified it is recommended to update to the latest hotfix to ensure the best possible experience.

To update the environment follow this procedure.

Installing on Ansible Openstack Victoria

The installation of Trilio for Openstack on Kolla Victoria with Trilio 4.1 is following this procedure:

Deplo T4O-4.1 GA Appliance
Upgrade to 4.1 HF5 packages on the appliance
Deploy Trilio components on Openstack Victoria
Update Trilio components on Openstack Victora
Configure the Trilio appliance

Deplo T4O-4.1 GA Appliance

Please follow this deployment guide to spin up the base Trilio 4.1GA appliance.

Upgrade to the latest 4.1 HF Appliance

Trilio supports Ansible OpenStack Victoria from 4.1HF5 onwards, so it is recommended to upgrade to the latest available hotfix on 4.1 to make deployment successful. Please follow this upgrade guide to upgrade the appliance to the latest 4.1 Hotfix.

Deploy Trilio components on Openstack

Run the deployment of the components following this guide using the following values:

Variable

Value

Change parameterOPENSTACK_DISTin the file/etc/openstack_deploy/user_tvault_vars.ymlto victoria

Update Trilio components

Follow this guide to update the packages on the OpenStack environment.

Configure the Trilio Appliance

Please follow this guide to configure the upgraded Trilio 4.1 appliance.

Installing on TripleO Train

1. Prepare for deployment

1.1] Select 'backup target' type

Backup target storage is used to store backup images taken by Trilio and details needed for configuration:

Following backup target types are supported by Trilio

a) NFS

Need NFS share path

b) Amazon S3

- S3 Access Key - Secret Key - Region - Bucket name

c) Other S3 compatible storage (Like, Ceph based S3)

- S3 Access Key - Secret Key - Region - Endpoint URL (Valid for S3 other than Amazon S3) - Bucket name

1.2] Clone triliovault-cfg-scripts repository

The following steps are to be done on 'undercloud' node on an already installed RHOSP environment. The overcloud-deploy command has to be run successfully already and the overcloud should be available.

All commands need to be run as user 'stack' on undercloud node

The following command clones the triliovault-cfg-scripts github repository.

cd /home/stack
git clone -b hotfix-13-TVO/4.1 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/

Please note that the Trilio Appliance needs to get updated to hf3 as well.

1.3] If backup target type is 'Ceph based S3' with SSL:

If your backup target is ceph S3 with SSL and SSL certificates are self signed or authorized by private CA, then user needs to provide CA chain certificate to validate the SSL requests. For that, user needs to rename his ca chain cert file to 's3-cert.pem' and copy it into directory - 'triliovault-cfg-scripts/redhat-director-scripts/redhat-director-scripts/<RHOSP_RELEASE___Directory/puppet/trilio/files'

cp s3-cert.pem /home/stack/triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/puppet/trilio/files/

2] Upload Trilio puppet module

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/scripts/
chmod +x *.sh
./upload_puppet_module.sh

## Output of above command looks like following.
Creating tarball...
Tarball created.
Creating heat environment file: /home/stack/.tripleo/environments/puppet-modules-url.yaml
Uploading file to swift: /tmp/puppet-modules-8Qjya2X/puppet-modules.tar.gz
+-----------------------+---------------------+----------------------------------+
| object                | container           | etag                             |
+-----------------------+---------------------+----------------------------------+
| puppet-modules.tar.gz | overcloud-artifacts | 368951f6a4d39cfe53b5781797b133ad |
+-----------------------+---------------------+----------------------------------+

## Above command creates following file.
ls -ll /home/stack/.tripleo/environments/puppet-modules-url.yaml

3] Update overcloud roles data file to include Trilio services

Trilio contains multiple services. Add these services to your roles_data.yaml.

In the case of uncustomized roles_data.yaml can the default file be found on the undercloud at:

/usr/share/openstack-tripleo-heat-templates/roles_data.yaml

Add the following services to the roles_data.yaml

All commands need to be run as user 'stack'

3.1] Add Trilio Datamover Api Service to role data file

This service needs to share the same role as the keystone and database service. In case of the pre-defined roles will these services run on the role Controller. In case of custom defined roles, it is necessary to use the same role where 'OS::TripleO::Services::Keystone' service installed.

Add the following line to the identified role:

'OS::TripleO::Services::TrilioDatamoverApi'

3.2] Add Trilio Datamover Service to role data file

This service needs to share the same role as the nova-compute service. In case of the pre-defined roles will the nova-compute service run on the role Compute. In case of custom defined roles, it is necessary to use the role the nova-compute service is using.

Add the following line to the identified role:

'OS::TripleO::Services::TrilioDatamover'

4] Prepare Trilio container images

All commands need to be run as user 'stack'

Refer to the below-mentioned value of the respective placeholder in this document. HOTFIX-TAG-VERSION : 4.1.94-hotfix-12-tripleo

Trilio containers are pushed to 'Dockerhub'. Registry URL: 'docker.io'. Container pull URLs are given below.

CentOS7

Trilio Datamover container:       docker.io/trilio/tripleo-train-centos7-trilio-datamover:<HOTFIX-TAG-VERSION>
Trilio Datamover Api Container:   docker.io/trilio/tripleo-train-centos7-trilio-datamover-api:<HOTFIX-TAG-VERSION>
Trilio horizon plugin:            docker.io/trilio/tripleo-train-centos7-trilio-horizon-plugin:<HOTFIX-TAG-VERSION>

CentOS8

Trilio Datamover container:        docker.io/trilio/tripleo-train-centos8-trilio-datamover:<HOTFIX-TAG-VERSION>
Trilio Datamover Api Container:   docker.io/trilio/tripleo-train-centos8-trilio-datamover-api:<HOTFIX-TAG-VERSION>
Trilio horizon plugin:            docker.io/trilio/tripleo-train-centos8-trilio-horizon-plugin:<HOTFIX-TAG-VERSION>

There are two registry methods available in TripleO Openstack Platform.

Remote Registry
Local Registry

4.1] Remote Registry

Follow this section when 'Remote Registry' is used.

For this method it is not necessary to pull the containers in advance. It is only necessary to populate the trilio_env.yaml file with the Trilio container URLs from Dockerhub registry.

Populate the trilio_env.yaml with container URLs for:

Trilio Datamover container
Trilio Datamover api container
Trilio Horizon Plugin

trilio_env.yaml will be available in __triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/environments

# For TripleO Train Centos7
$ grep 'Image' trilio_env.yaml
   DockerTrilioDatamoverImage: docker.io/trilio/tripleo-train-centos7-trilio-datamover:<HOTFIX-TAG-VERSION>
   DockerTrilioDmApiImage: docker.io/trilio/tripleo-train-centos7-trilio-datamover-api:<HOTFIX-TAG-VERSION>
   DockerHorizonImage: docker.io/trilio/tripleo-train-centos7-trilio-horizon-plugin:<HOTFIX-TAG-VERSION>

# For Tripleo Train Centos8
$ grep 'Image' trilio_env.yaml
   DockerTrilioDatamoverImage: docker.io/trilio/tripleo-train-centos8-trilio-datamover:<HOTFIX-TAG-VERSION>
   DockerTrilioDmApiImage: docker.io/trilio/tripleo-train-centos8-trilio-datamover-api:<HOTFIX-TAG-VERSION>
   ContainerHorizonImage: docker.io/trilio/tripleo-train-centos8-trilio-horizon-plugin:<HOTFIX-TAG-VERSION>

4.2] Local Registry

Follow this section when 'local registry' is used on the undercloud.

Run the following script. Script pulls the triliovault containers and updates the triliovault environment file with URLs.

Acceptable values for the below two parameters:

OS_platform: [centos7, centos8]

container_tool_available_on_undercloud: [docker, podman]

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/scripts/

sudo ./prepare_trilio_images.sh <undercloud_registry_hostname_or_ip> <OS_platform> <4.1-TRIPLEO-CONTAINER> <container_tool_available_on_undercloud>

## To get undercloud registry hostname/ip, we have two approaches. Use either one.
1. openstack tripleo container image list

2. find your 'containers-prepare-parameter.yaml' (from overcloud deploy command) and search for 'push_destination'
cat /home/stack/containers-prepare-parameter.yaml | grep push_destination
 - push_destination: "undercloud.ctlplane.ooo.prod1:8787"

Here, 'undercloud.ctlplane.ooo.prod1' is undercloud registry hostname. Use it in our command like following example.

# Command Example:
sudo ./prepare_trilio_images.sh undercloud.ctlplane.ooo.prod1 centos7 4.1.94-hotfix-1-tripleo podman

## Verify changes
# For TripleO Train Centos7
$ grep 'Image' ../environments/trilio_env.yaml
   DockerTrilioDatamoverImage: prod1-undercloud.demo:8787/trilio/tripleo-train-centos7-trilio-datamover:<HOTFIX-TAG-VERSION>
   DockerTrilioDmApiImage: prod1-undercloud.demo:8787/trilio/tripleo-train-centos7-trilio-datamover-api:<HOTFIX-TAG-VERSION>
   DockerHorizonImage: prod1-undercloud.demo:8787/trilio/tripleo-train-centos7-trilio-horizon-plugin:<HOTFIX-TAG-VERSION>

# For Tripleo Train Centos8
$ grep 'Image' trilio_env.yaml
   DockerTrilioDatamoverImage: prod1-undercloud.demo:8787/trilio/tripleo-train-centos8-trilio-datamover:<HOTFIX-TAG-VERSION>
   DockerTrilioDmApiImage: prod1-undercloud.demo:8787/trilio/tripleo-train-centos8-trilio-datamover-api:<HOTFIX-TAG-VERSION>
   ContainerHorizonImage: prod1-undercloud.demo:8787/trilio/tripleo-train-centos8-trilio-horizon-plugin:<HOTFIX-TAG-VERSION>

The changes can be verified using the following commands.

## For Centos7 Train

(undercloud) [stack@undercloud redhat-director-scripts/tripleo-train]$ openstack tripleo container image list | grep trilio
| docker://undercloud.ctlplane.localdomain:8787/trilio/tripleo-train-centos7-trilio-datamover:<HOTFIX-TAG-VERSION>                       |                        |
| docker://undercloud.ctlplane.localdomain:8787/trilio/tripleo-train-centos7-trilio-datamover-api:<HOTFIX-TAG-VERSION>                  |                   |
| docker://undercloud.ctlplane.localdomain:8787/trilio/tripleo-train-centos7-trilio-horizon-plugin:<HOTFIX-TAG-VERSION>                  |

-----------------------------------------------------------------------------------------------------
## For Centos8 Train
(undercloud) [stack@undercloud redhat-director-scripts/tripleo-train]$ openstack tripleo container image list | grep trilio
| docker://undercloud.ctlplane.localdomain:8787/trilio/tripleo-train-centos8-trilio-datamover:<HOTFIX-TAG-VERSION>                       |                        |
| docker://undercloud.ctlplane.localdomain:8787/trilio/tripleo-train-centos8-trilio-datamover-api:<HOTFIX-TAG-VERSION>                  |                   |
| docker://undercloud.ctlplane.localdomain:8787/trilio/tripleo-train-centos8-trilio-horizon-plugin:<HOTFIX-TAG-VERSION>                 |

5] Fill in triliovault environment details

Fill triliovault details in file - '/home/stack/triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/environments/trilio_env.yaml', triliovault environment file is self explanatory. Fill details of backup target, verify image urls and other details.

6] Install Trilio on Overcloud

Use the following heat environment file and roles data file in overcloud deploy command

trilio_env.yaml: This environment file contains Trilio backup target details and Trilio container image locations
roles_data.yaml: This file contains overcloud roles data with Trilio roles added.
Use the correct trilio endpoint map file as per your keystone endpoint configuration. - Instead of tls-endpoints-public-dns.yaml this file, use ‘environments/trilio_env_tls_endpoints_public_dns.yaml’ - Instead of tls-endpoints-public-ip.yaml this file, use ‘environments/trilio_env_tls_endpoints_public_ip.yaml’ - Instead of tls-everywhere-endpoints-dns.yaml this file, use ‘environments/trilio_env_tls_everywhere_dns.yaml’

Deploy command with triliovault environment file looks like following.

openstack overcloud deploy --templates \
  -e /home/stack/templates/node-info.yaml \
  -e /home/stack/templates/overcloud_images.yaml \
  -e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/environments/trilio_env.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-tls.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/inject-trust-anchor.yaml \
  -e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/environments/trilio_env_tls_endpoints_public_dns.yaml \
  --ntp-server 192.168.1.34 \
  --libvirt-type qemu \
  --log-file overcloud_deploy.log \
  -r /usr/share/openstack-tripleo-heat-templates/roles_data.yaml

7] Verify deployment

If the containers are in restarting state or not listed by the following command then your deployment is not done correctly. Please recheck if you followed the complete documentation.

7.1] On Controller node

[root@overcloud-controller-0 heat-admin]# podman ps | grep trilio
26fcb9194566  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>        kolla_start           5 days ago  Up 5 days ago         trilio_dmapi
094971d0f5a9  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>       kolla_start           5 days ago  Up 5 days ago         horizon

Verify the haproxy configuration under:

/var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg

7.2] On Compute node

Make sure Trilio datamover container is in running state and no other Trilio container is deployed on compute nodes.

[root@overcloud-novacompute-0 heat-admin]# podman ps | grep trilio
b1840444cc59  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>                    kolla_start           5 days ago  Up 5 days ago         trilio_datamover

7.3] On the node with Horizon service

[root@overcloud-controller-0 heat-admin]# podman ps | grep horizon
094971d0f5a9  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>       kolla_start           5 days ago  Up 5 days ago         horizon

8] Deplo T4O-4.1 GA Appliance

Please follow this deployment guide to spin up the base Trilio 4.1GA appliance.

9] Upgrade to the latest 4.1 HF on the appliance

Trilio supports TripleO Train from 4.1HF5 onwards, so it is recommended to upgrade to the latest available hotfix on 4.1 to make deployment successful. Please follow this upgrade guide to upgrade the appliance to the latest 4.1 Hotfix.

10] Configure the Trilio Appliance

Please follow this guide to configure the upgraded Trilio 4.1 appliance.

Installing on TripleO Train

1. Prepare for deployment

1.1] Select 'backup target' type

Backup target storage is used to store backup images taken by Trilio and details needed for configuration:

Following backup target types are supported by Trilio

a) NFS

Need NFS share path

b) Amazon S3

- S3 Access Key - Secret Key - Region - Bucket name

c) Other S3 compatible storage (Like, Ceph based S3)

- S3 Access Key - Secret Key - Region - Endpoint URL (Valid for S3 other than Amazon S3) - Bucket name

1.2] Clone triliovault-cfg-scripts repository

All commands need to be run as user 'stack' on undercloud node

The following command clones the triliovault-cfg-scripts github repository.

cd /home/stack
git clone -b hotfix-13-TVO/4.1 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/

Please note that the Trilio Appliance needs to get updated to hf3 as well.

1.3] If backup target type is 'Ceph based S3' with SSL:

cp s3-cert.pem /home/stack/triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/puppet/trilio/files/

2] Upload Trilio puppet module

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/scripts/
chmod +x *.sh
./upload_puppet_module.sh

## Output of above command looks like following.
Creating tarball...
Tarball created.
Creating heat environment file: /home/stack/.tripleo/environments/puppet-modules-url.yaml
Uploading file to swift: /tmp/puppet-modules-8Qjya2X/puppet-modules.tar.gz
+-----------------------+---------------------+----------------------------------+
| object                | container           | etag                             |
+-----------------------+---------------------+----------------------------------+
| puppet-modules.tar.gz | overcloud-artifacts | 368951f6a4d39cfe53b5781797b133ad |
+-----------------------+---------------------+----------------------------------+

## Above command creates following file.
ls -ll /home/stack/.tripleo/environments/puppet-modules-url.yaml

3] Update overcloud roles data file to include Trilio services

Trilio contains multiple services. Add these services to your roles_data.yaml.

In the case of uncustomized roles_data.yaml can the default file be found on the undercloud at:

/usr/share/openstack-tripleo-heat-templates/roles_data.yaml

Add the following services to the roles_data.yaml

All commands need to be run as user 'stack'

3.1] Add Trilio Datamover Api Service to role data file

This service needs to share the same role as the keystone and database service. In case of the pre-defined roles will these services run on the role Controller. In case of custom defined roles, it is necessary to use the same role where 'OS::TripleO::Services::Keystone' service installed.

Add the following line to the identified role:

'OS::TripleO::Services::TrilioDatamoverApi'

3.2] Add Trilio Datamover Service to role data file

This service needs to share the same role as the nova-compute service. In case of the pre-defined roles will the nova-compute service run on the role Compute. In case of custom defined roles, it is necessary to use the role the nova-compute service is using.

Add the following line to the identified role:

'OS::TripleO::Services::TrilioDatamover'

4] Prepare Trilio container images

All commands need to be run as user 'stack'

Refer to the below-mentioned value of the respective placeholder in this document. HOTFIX-TAG-VERSION : 4.1.94-hotfix-12-tripleo

Trilio containers are pushed to 'Dockerhub'. Registry URL: 'docker.io'. Container pull URLs are given below.

CentOS7

Trilio Datamover container:       docker.io/trilio/tripleo-train-centos7-trilio-datamover:<HOTFIX-TAG-VERSION>
Trilio Datamover Api Container:   docker.io/trilio/tripleo-train-centos7-trilio-datamover-api:<HOTFIX-TAG-VERSION>
Trilio horizon plugin:            docker.io/trilio/tripleo-train-centos7-trilio-horizon-plugin:<HOTFIX-TAG-VERSION>

CentOS8

Trilio Datamover container:        docker.io/trilio/tripleo-train-centos8-trilio-datamover:<HOTFIX-TAG-VERSION>
Trilio Datamover Api Container:   docker.io/trilio/tripleo-train-centos8-trilio-datamover-api:<HOTFIX-TAG-VERSION>
Trilio horizon plugin:            docker.io/trilio/tripleo-train-centos8-trilio-horizon-plugin:<HOTFIX-TAG-VERSION>

There are two registry methods available in TripleO Openstack Platform.

Remote Registry
Local Registry

4.1] Remote Registry

Follow this section when 'Remote Registry' is used.

For this method it is not necessary to pull the containers in advance. It is only necessary to populate the trilio_env.yaml file with the Trilio container URLs from Dockerhub registry.

Populate the trilio_env.yaml with container URLs for:

Trilio Datamover container
Trilio Datamover api container
Trilio Horizon Plugin

trilio_env.yaml will be available in __triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/environments

# For TripleO Train Centos7
$ grep 'Image' trilio_env.yaml
   DockerTrilioDatamoverImage: docker.io/trilio/tripleo-train-centos7-trilio-datamover:<HOTFIX-TAG-VERSION>
   DockerTrilioDmApiImage: docker.io/trilio/tripleo-train-centos7-trilio-datamover-api:<HOTFIX-TAG-VERSION>
   DockerHorizonImage: docker.io/trilio/tripleo-train-centos7-trilio-horizon-plugin:<HOTFIX-TAG-VERSION>

# For Tripleo Train Centos8
$ grep 'Image' trilio_env.yaml
   DockerTrilioDatamoverImage: docker.io/trilio/tripleo-train-centos8-trilio-datamover:<HOTFIX-TAG-VERSION>
   DockerTrilioDmApiImage: docker.io/trilio/tripleo-train-centos8-trilio-datamover-api:<HOTFIX-TAG-VERSION>
   ContainerHorizonImage: docker.io/trilio/tripleo-train-centos8-trilio-horizon-plugin:<HOTFIX-TAG-VERSION>

4.2] Local Registry

Follow this section when 'local registry' is used on the undercloud.

Run the following script. Script pulls the triliovault containers and updates the triliovault environment file with URLs.

Acceptable values for the below two parameters:

OS_platform: [centos7, centos8]

container_tool_available_on_undercloud: [docker, podman]

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/scripts/

sudo ./prepare_trilio_images.sh <undercloud_registry_hostname_or_ip> <OS_platform> <4.1-TRIPLEO-CONTAINER> <container_tool_available_on_undercloud>

## To get undercloud registry hostname/ip, we have two approaches. Use either one.
1. openstack tripleo container image list

2. find your 'containers-prepare-parameter.yaml' (from overcloud deploy command) and search for 'push_destination'
cat /home/stack/containers-prepare-parameter.yaml | grep push_destination
 - push_destination: "undercloud.ctlplane.ooo.prod1:8787"

Here, 'undercloud.ctlplane.ooo.prod1' is undercloud registry hostname. Use it in our command like following example.

# Command Example:
sudo ./prepare_trilio_images.sh undercloud.ctlplane.ooo.prod1 centos7 4.1.94-hotfix-1-tripleo podman

## Verify changes
# For TripleO Train Centos7
$ grep 'Image' ../environments/trilio_env.yaml
   DockerTrilioDatamoverImage: prod1-undercloud.demo:8787/trilio/tripleo-train-centos7-trilio-datamover:<HOTFIX-TAG-VERSION>
   DockerTrilioDmApiImage: prod1-undercloud.demo:8787/trilio/tripleo-train-centos7-trilio-datamover-api:<HOTFIX-TAG-VERSION>
   DockerHorizonImage: prod1-undercloud.demo:8787/trilio/tripleo-train-centos7-trilio-horizon-plugin:<HOTFIX-TAG-VERSION>

# For Tripleo Train Centos8
$ grep 'Image' trilio_env.yaml
   DockerTrilioDatamoverImage: prod1-undercloud.demo:8787/trilio/tripleo-train-centos8-trilio-datamover:<HOTFIX-TAG-VERSION>
   DockerTrilioDmApiImage: prod1-undercloud.demo:8787/trilio/tripleo-train-centos8-trilio-datamover-api:<HOTFIX-TAG-VERSION>
   ContainerHorizonImage: prod1-undercloud.demo:8787/trilio/tripleo-train-centos8-trilio-horizon-plugin:<HOTFIX-TAG-VERSION>

The changes can be verified using the following commands.

## For Centos7 Train

(undercloud) [stack@undercloud redhat-director-scripts/tripleo-train]$ openstack tripleo container image list | grep trilio
| docker://undercloud.ctlplane.localdomain:8787/trilio/tripleo-train-centos7-trilio-datamover:<HOTFIX-TAG-VERSION>                       |                        |
| docker://undercloud.ctlplane.localdomain:8787/trilio/tripleo-train-centos7-trilio-datamover-api:<HOTFIX-TAG-VERSION>                  |                   |
| docker://undercloud.ctlplane.localdomain:8787/trilio/tripleo-train-centos7-trilio-horizon-plugin:<HOTFIX-TAG-VERSION>                  |

-----------------------------------------------------------------------------------------------------
## For Centos8 Train
(undercloud) [stack@undercloud redhat-director-scripts/tripleo-train]$ openstack tripleo container image list | grep trilio
| docker://undercloud.ctlplane.localdomain:8787/trilio/tripleo-train-centos8-trilio-datamover:<HOTFIX-TAG-VERSION>                       |                        |
| docker://undercloud.ctlplane.localdomain:8787/trilio/tripleo-train-centos8-trilio-datamover-api:<HOTFIX-TAG-VERSION>                  |                   |
| docker://undercloud.ctlplane.localdomain:8787/trilio/tripleo-train-centos8-trilio-horizon-plugin:<HOTFIX-TAG-VERSION>                 |

5] Fill in triliovault environment details

6] Install Trilio on Overcloud

Use the following heat environment file and roles data file in overcloud deploy command

trilio_env.yaml: This environment file contains Trilio backup target details and Trilio container image locations
roles_data.yaml: This file contains overcloud roles data with Trilio roles added.
Use the correct trilio endpoint map file as per your keystone endpoint configuration. - Instead of tls-endpoints-public-dns.yaml this file, use ‘environments/trilio_env_tls_endpoints_public_dns.yaml’ - Instead of tls-endpoints-public-ip.yaml this file, use ‘environments/trilio_env_tls_endpoints_public_ip.yaml’ - Instead of tls-everywhere-endpoints-dns.yaml this file, use ‘environments/trilio_env_tls_everywhere_dns.yaml’

Deploy command with triliovault environment file looks like following.

openstack overcloud deploy --templates \
  -e /home/stack/templates/node-info.yaml \
  -e /home/stack/templates/overcloud_images.yaml \
  -e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/environments/trilio_env.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-tls.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/inject-trust-anchor.yaml \
  -e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/environments/trilio_env_tls_endpoints_public_dns.yaml \
  --ntp-server 192.168.1.34 \
  --libvirt-type qemu \
  --log-file overcloud_deploy.log \
  -r /usr/share/openstack-tripleo-heat-templates/roles_data.yaml

7] Verify deployment

If the containers are in restarting state or not listed by the following command then your deployment is not done correctly. Please recheck if you followed the complete documentation.

7.1] On Controller node

[root@overcloud-controller-0 heat-admin]# podman ps | grep trilio
26fcb9194566  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>        kolla_start           5 days ago  Up 5 days ago         trilio_dmapi
094971d0f5a9  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>       kolla_start           5 days ago  Up 5 days ago         horizon

Verify the haproxy configuration under:

/var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg

7.2] On Compute node

Make sure Trilio datamover container is in running state and no other Trilio container is deployed on compute nodes.

[root@overcloud-novacompute-0 heat-admin]# podman ps | grep trilio
b1840444cc59  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>                    kolla_start           5 days ago  Up 5 days ago         trilio_datamover

7.3] On the node with Horizon service

[root@overcloud-controller-0 heat-admin]# podman ps | grep horizon
094971d0f5a9  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>       kolla_start           5 days ago  Up 5 days ago         horizon

8] Deplo T4O-4.1 GA Appliance

Please follow this deployment guide to spin up the base Trilio 4.1GA appliance.

9] Upgrade to the latest 4.1 HF on the appliance

10] Configure the Trilio Appliance

Please follow this guide to configure the upgraded Trilio 4.1 appliance.

Installing on Kolla Ussuri

This page lists all steps required to deploy Trilio components on Kolla-ansible deployed OpenStack cloud.

1] Plan for Deployment

Please ensure that the Trilio Appliance has been updated to the latest hotfix before continuing the installation.

1.1] Select backup target type

Backup target storage is used to store backup images taken by Trilio and details needed for configuration:

Following backup target types are supported by Trilio. Select one of them and get it ready before proceeding to the next step.

a) NFS

Need NFS share path

b) Amazon S3

- S3 Access Key - Secret Key - Region - Bucket name

c) Other S3 compatible storage (Like, Ceph based S3)

- S3 Access Key - Secret Key - Region - Endpoint URL (Valid for S3 other than Amazon S3) - Bucket name

2] Clone Trilio Deployment Scripts

git clone -b hotfix-13-TVO/4.1 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/kolla-ansible/

# For Centos and Ubuntu
cp -R ansible/roles/triliovault /usr/local/share/kolla-ansible/ansible/roles/

3] Hook Trilio deployment scripts to Kolla-ansible deploy scripts

3.1] Add Trilio global variables to globals.yml

## For Centos and Ubuntu
## Take backup of globals.yml
cp /etc/kolla/globals.yml /opt/

## Append Trilio global variables to globals.yml 
cat ansible/triliovault_globals.yml >> /etc/kolla/globals.yml

3.2] Add Trilio passwords to kolla passwords.yaml

Append triliovault_passwords.yml to /etc/kolla/passwords.yml. Passwords are empty. Set these passwords manually in the /etc/kolla/passwords.yml.

## For Centos and Ubuntu
## Take backup of passwords.yml
cp /etc/kolla/passwords.yml /opt/

## Append Trilio global variables to passwords.yml 
cat ansible/triliovault_passwords.yml >> /etc/kolla/passwords.yml

Edit /etc/kolla/passwords.yml, go to the end of the file and set trilio passwords.

3.3] Append Trilio site.yml content to kolla ansible’s site.yml

## For Centos and Ubuntu
## Take backup of site.yml
cp /usr/local/share/kolla-ansible/ansible/site.yml /opt/

## Append Trilio code to site.yml
cat ansible/triliovault_site.yml >> /usr/local/share/kolla-ansible/ansible/site.yml

3.4] Append triliovault_inventory.txt to your cloud’s kolla-ansible inventory file.

For example:
If your inventory file name path '/root/multinode' then use following command.

cat ansible/triliovault_inventory.txt >> /root/multinode

4] Edit globals.yml to set Trilio parameters

Following is the list of parameters that the usr needs to edit.

Parameter

Defaults/choices

comments

In the case of a different registry than docker hub, Trilio containers need to be pulled from docker.io and pushed to preferred registries.

Following are the triliovault container image URLs. Replace kolla_base_distro and triliovault_tag variables with their values.

docker.io/trilio/<kolla_base_distro>-binary-trilio-datamover-api:<triliovault_tag>
docker.io/trilio/<kolla_base_distro>-binary-trilio-datamover:<triliovault_tag>
docker.io/trilio/<kolla_base_distro>-binary-trilio-horizon-plugin:<triliovault_tag>

5] Enable Trilio Snapshot mount feature

To enable Trilio's Snapshot mount feature it is necessary to make the Trilio Backup target available to the nova-compute and nova-libvirt containers.

For a default Kolla installation, will the variable look as follows afterward:

nova_libvirt_default_volumes:
  - "{{ node_config_directory }}/nova-libvirt/:{{ container_config_directory }}/:ro"
  - "/etc/localtime:/etc/localtime:ro"
  - "{{ '/etc/timezone:/etc/timezone:ro' if ansible_os_family == 'Debian' else '' }}"
  - "/lib/modules:/lib/modules:ro"
  - "/run/:/run/:shared"
  - "/dev:/dev"
  - "/sys/fs/cgroup:/sys/fs/cgroup"
  - "kolla_logs:/var/log/kolla/"
  - "libvirtd:/var/lib/libvirt"
  - "{{ nova_instance_datadir_volume }}:/var/lib/nova/"
  - "
{% if enable_shared_var_lib_nova_mnt | bool %}/var/lib/nova/mnt:/var/lib/nova/mnt:shared{% endif %}
"
  - "nova_libvirt_qemu:/etc/libvirt/qemu"
  - "{{ kolla_dev_repos_directory ~ '/nova/nova:/var/lib/kolla/venv/lib/python' ~ distro_python_version ~ '/site-packages/nova' if nova_dev_mode | bool else '' }
  - "/var/trilio:/var/trilio:shared"

Next, find the variable nova_compute_default_volumes in the same file and append the mount bind /var/trilio:/var/trilio:shared to the list.

After the change will the variable look for a default Kolla installation as follows:

nova_compute_default_volumes:
  - "{{ node_config_directory }}/nova-compute/:{{ container_config_directory }}/:ro"
  - "/etc/localtime:/etc/localtime:ro"
  - "{{ '/etc/timezone:/etc/timezone:ro' if ansible_os_family == 'Debian' else '' }}"
  - "/lib/modules:/lib/modules:ro"
  - "/run:/run:shared"
  - "/dev:/dev"
  - "kolla_logs:/var/log/kolla/"
  - "
{% if enable_iscsid | bool %}iscsi_info:/etc/iscsi{% endif %}"
  - "libvirtd:/var/lib/libvirt"
  - "{{ nova_instance_datadir_volume }}:/var/lib/nova/"
  - "{% if enable_shared_var_lib_nova_mnt | bool %}/var/lib/nova/mnt:/var/lib/nova/mnt:shared{% endif %}"
  - "{{ kolla_dev_repos_directory ~ '/nova/nova:/var/lib/kolla/venv/lib/python' ~ distro_python_version ~ '/site-packages/nova' if nova_dev_mode | bool else '' }}"
  - "/var/trilio:/var/trilio:shared"

After the changes the variable will look like the following:

nova_compute_ironic_default_volumes:
  - "{{ node_config_directory }}/nova-compute-ironic/:{{ container_config_directory }}/:ro"
  - "/etc/localtime:/etc/localtime:ro"
  - "{{ '/etc/timezone:/etc/timezone:ro' if ansible_os_family == 'Debian' else '' }}"
  - "kolla_logs:/var/log/kolla/"
  - "{{ kolla_dev_repos_directory ~ '/nova/nova:/var/lib/kolla/venv/lib/python' ~ distro_python_version ~ '/site-packages/nova' if nova_dev_mode | bool else '' }}"
  - "/var/trilio:/var/trilio:shared"

6] Pull Trilio container images

Pull the Trilio container images from the Dockerhub based on the existing inventory file. In the example is the inventory file named multinode.

kolla-ansible -i multinode pull --tags triliovault

7] Deploy Trilio

All that is left, is to run the deploy command using the existing inventory file. In the example is the inventory file named 'multinode'.

This is just an example command. You need to use your cloud deploy command.

kolla-ansible -i multinode deploy

8] Verify Trilio deployment

Verify on the nodes that are supposed to run the Trilio containers, that those are available and healthy.

[root@controller ~]# docker ps | grep triliovault_datamover_api
f00781997bc3        trilio/centos-binary-trilio-datamover-api:<triliovualt_tag>    "dumb-init --single-…"   2 minutes ago       Up 2 minutes                            triliovault_datamover_api

[root@compute ~]# docker ps | grep triliovault_datamover
84831db5d215        trilio/centos-binary-trilio-datamover:<triliovualt_tag>     "dumb-init --single-…"   5 minutes ago       Up 4 minutes                            triliovault_datamover

[root@controller ~]# docker ps | grep horizon
f3647e0fff27        trilio/centos-binary-trilio-horizon-plugin:<triliovualt_tag>   "dumb-init --single-…"   8 minutes ago       Up 8 minutes                            horizon

9] Troubleshooting Tips

9.1 ] Check Trilio containers and their startup logs

To see all TriloVault containers running on a specific node use the docker ps command.

docker ps -a | grep trilio

To check the startup logs use the docker logs <container name> command.

docker logs trilio_datamover_api
docker logs trilio_datamover

9.2] Trilio Horizon tabs are not visible in Openstack

Verify that the Trilio Appliance is configured. The Horizon tabs are only shown when a configured Trilio appliance is available.

Verify that the Trilio horizon container is installed and in a running state.

docker ps | grep horizon

9.3] Trilio Service logs

Trilio datamover api service logs on datamover api node

/var/log/kolla/triliovault-datamover-api/dmapi.log

Trilio datamover service logs on datamover node

/var/log/kolla/triliovault-datamover/tvault-contego.log

10. Change the nova user id on the Trilio Nodes

Note: This step needs to be done on Trilio Appliance node. Not on OpenStack node.

Pre-requisite: You should have already launched Trilio appliance VM

Download the shell script that will change the user id
Assign executable permissions
Execute the script
Verify that nova user and group id have changed to '42436'
After this step, you can proceed to the 'Configuring Trilio' section.

## Download the shell script
$ curl -O https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/master/common/nova_userid.sh

## Assign executable permissions
$ chmod +x nova_userid.sh

## Execute the shell script to change 'nova' user and group id to '42436'
$ ./nova_userid.sh

## Ignore any errors and verify that 'nova' user and group id has changed to '42436'
$ id nova
   uid=42436(nova) gid=42436(nova) groups=42436(nova),990(libvirt),36(kvm)

Installing on RHOSP

1. Prepare for deployment

1.1] Select 'backup target' type

Backup target storage is used to store backup images taken by Trilio and also associated configuration needs. The following backup target types are supported by Trilio:

Backup Target Types

Required Configuration

1.2] Clone triliovault-cfg-scripts repository

All commands need to be run as user 'stack' on undercloud node

Ensure that the Trilio appliance connected to this installation is on the latest Hotfix version. Failure to ensure this may lead to your installation not working as expected.
1. Refer to this doc :
Run the following command to clone the triliovault-cfg-scripts github repository:

If your backup target type is 'Ceph-based S3' with SSL, skip this step. Otherwise, access the Red Hat Director scripts according to the RHOSP version being used:
- RHOSP 13 - cd triliovault-cfg-scripts/redhat-director-scripts/rhosp13/
- RHOSP 16.1 - cd triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/
- RHOSP 16.2 - cd triliovault-cfg-scripts/redhat-director-scripts/rhosp16.2/
If your backup target is Ceph S3 with SSL and your SSL certificates are self-signed or authorized by private CA, you must provide the CA chain certificate to validate the SSL requests. Otherwise, skip this step. To do this:
1. Rename your CA chain cert file to s3-cert.pem.
2. Copy the renamed file into the following directory: triliovault-cfg-scripts/redhat-director-scripts/redhat-director-scripts/<RHOSP_RELEASE_Directory>/puppet/trilio/filesIf your overcloud deploy command uses any other deploy artifact through an environment file, then you must merge Trilio deploy artifact url and your url in a single file.
3. Then access the Red Hat Director scripts according to the version being used:
  - RHOSP 13 - cp s3-cert.pem /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp13/puppet/trilio/files/
  - RHOSP 16.1 - cp s3-cert.pem /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/puppet/trilio/files/
  - RHOSP 16.2 - cp s3-cert.pem /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.2/puppet/trilio/files/

From this point onwards in the documentation, only the following path will be used for examples: cd triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/

2] Upload Trilio-Puppet module

RHOSP 16.1

cd triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/

The following commands upload the Trilio puppet module to the overcloud registry. The upload only happens upon the next deployment.

Step 1: -

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/scripts/
./upload_puppet_module.sh

The output of the above command looks like the following.

./upload_puppet_module.sh
Creating tarball...
Tarball created.
Creating heat environment file: /home/stack/.tripleo/environments/puppet-modules-url.yaml
Uploading file to swift: /tmp/puppet-modules-8Qjya2X/puppet-modules.tar.gz
+-----------------------+---------------------+----------------------------------+
| object                | container           | etag                             |
+-----------------------+---------------------+----------------------------------+
| puppet-modules.tar.gz | overcloud-artifacts | 368951f6a4d39cfe53b5781797b133ad |
+-----------------------+---------------------+----------------------------------+

## Above command creates the following file.
ls -ll /home/stack/.tripleo/environments/puppet-modules-url.yaml

Trilio puppet module is uploaded to overcloud as a swift deploy artifact with heat resource name 'DeployArtifactURLs'.

Step 2: - Check Trilio's Puppet module artifact file and ensure that it looks like the following:

(undercloud) [stack@ucloud161 ~]$ cat /home/stack/.tripleo/environments/puppet-modules-url.yaml
# Heat environment to deploy artifacts via Swift Temp URL(s)
parameter_defaults:
    DeployArtifactURLs:
    - 'http://172.25.0.103:8080/v1/AUTH_46ba596219d143c8b076e9fcc4139fed/overcloud-artifacts/puppet-modules.tar.gz?temp_url_sig=c3972b7ce75226c278ab3fa8237d31cc1f2115bd&temp_url_expires=1646738377'

Step 3: -

Firstly, check to make sure that your overcloud deploy environment files uses deploy artifacts. To do this check string DeployArtifactURLs in your environment files (only those mentioned in the overcloud deploy command with -e option). If you find any environment file with the -e option, then your deploy command is using deploy artifacts.
If your deploy command is using deploy artifact, you must merge all deploy artifacts in a single file. For example, if your artifact file path is /home/stack/templates/user-artifacts.yaml, then perform the following steps to merge both urls in single file, which is passed to the overcloud deploy command with the -e option.

(undercloud) [stack@ucloud161 ~]$ cat /home/stack/.tripleo/environments/puppet-modules-url.yaml | grep http >> /home/stack/templates/user-artifacts.yaml
(undercloud) [stack@ucloud161 ~]$ cat /home/stack/templates/user-artifacts.yaml
# Heat environment to deploy artifacts via Swift Temp URL(s)
parameter_defaults:
    DeployArtifactURLs:
    - 'http://172.25.0.103:8080/v1/AUTH_57ba596219d143c8b076e9fcc4139f3g/overcloud-artifacts/some-artifact.tar.gz?temp_url_sig=dc972b7ce75226c278ab3fa8237d31cc1f2115sc&temp_url_expires=3446738365'
    - 'http://172.25.0.103:8080/v1/AUTH_46ba596219d143c8b076e9fcc4139fed/overcloud-artifacts/puppet-modules.tar.gz?temp_url_sig=c3972b7ce75226c278ab3fa8237d31cc1f2115bd&temp_url_expires=1646738377'

3] Update overcloud roles data file to include Trilio services

Trilio contains multiple services. Add these services to your roles_data.yaml.

In the case of uncustomized roles_data.yaml can the default file be found on the undercloud at:

/usr/share/openstack-tripleo-heat-templates/roles_data.yaml

Add the following services to the roles_data.yaml

All commands need to be run as user 'stack'

3.1] Add Trilio Datamover Api Service to role data file

Add the following line to the identified role:

'OS::TripleO::Services::TrilioDatamoverApi'

3.2] Add Trilio Datamover Service to role data file

Add the following line to the identified role:

'OS::TripleO::Services::TrilioDatamover'

4] Prepare Trilio container images

All commands need to be run as user 'stack'

Trilio containers are pushed to 'RedHat Container Registry'. Registry URL: registry.connect.redhat.com Container pull URLs are given below.

Please note that using the hotfix containers requires that the Trilio Appliance is getting upgraded to the desired hotfix level as well.

Refer to the word <HOTFIX-TAG-VERSION> as 4.1.94-hotfix-16 in the below sections

RHOSP 13

Trilio Datamover container:
    registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp13
Trilio Datamover Api Container:   
    registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp13
Trilio horizon plugin:            
    registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp13

RHOSP 16.1

Trilio Datamover container:
    registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1
Trilio Datamover Api Container: 
    registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1
Trilio horizon plugin:
    registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1

RHOSP 16.2

Trilio Datamover container:        
    registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.2
Trilio Datamover Api Container:
    registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.2
Trilio horizon plugin:            
    registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.2

There are three registry methods available in the RedHat OpenStack Platform.

Remote Registry
Local Registry
Satellite Server

4.1] Remote Registry

Follow this section when 'Remote Registry' is used.

Make sure other OpenStack service images are also using the same method to pull container images. If it's not the case you can not use this method.
Populate containers-prepare-parameter.yaml with content like following. Important parameters are 'push_destination: false', ContainerImageRegistryLogin: true and registry credentials. Trilio container images are published to the registry registry.connect.redhat.com Credentials of registry 'registry.redhat.io' will work for registry.connect.redhat.com registry too.

Note: This file - containers-prepare-parameter.yaml

File name: containers-prepare-parameter.yaml

parameter_defaults:
  ContainerImagePrepare:
  - push_destination: false
    set:
      namespace: registry.redhat.io/...
      ...
  ...
  ContainerImageRegistryCredentials:
    registry.redhat.io:
      myuser: 'p@55w0rd!'
    registry.connect.redhat.com:
      myuser: 'p@55w0rd!'
  ContainerImageRegistryLogin: true

Redhat document for remote registry method:

Note: File 'containers-prepare-parameter.yaml' gets created as output of command 'openstack tripleo container image prepare'. Refer above document by RedHat

3. Make sure you have network connectivity to the above registries from all overcloud nodes. Otherwise, image pull operation will fail.

4. Populate the trilio_env.yaml with Trilio container image URLs:

Trilio Datamover container
Trilio Datamover API container
Trilio Horizon Plugin

trilio_env.yaml will be available in

cd triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_Directory/
vi trilio_env.yaml

# For RHOSP13
$ grep 'Image' trilio_env.yaml
   DockerTrilioDatamoverImage: registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp13
   DockerTrilioDmApiImage: registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp13
   DockerHorizonImage: registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp13

# For RHOSP16.1
$ grep 'Image' trilio_env.yaml
   DockerTrilioDatamoverImage: registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1
   DockerTrilioDmApiImage: registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1
   ContainerHorizonImage: registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1
   
# For RHOSP16.2
$ grep 'Image' trilio_env.yaml
   DockerTrilioDatamoverImage: registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.2
   DockerTrilioDmApiImage: registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.2
   ContainerHorizonImage: registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.2

4.2] Local Registry

Follow this section when 'local registry' is used on the undercloud.

RHOSP13

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp13/scripts/
./prepare_trilio_images.sh <undercloud_ip/hostname> <HOTFIX-TAG-VERSION>-rhosp13

Verify the changes

$ grep 'Image' ../environments/trilio_env.yaml
   DockerTrilioDatamoverImage: 172.25.2.2:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp13
   DockerTrilioDmApiImage: 172.25.2.2:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp13
   DockerHorizonImage: 172.25.2.2:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp13

RHOSP16.1

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/scripts/
sudo ./prepare_trilio_images.sh <undercloud_ip/hostname> <HOTFIX-TAG-VERSION>-rhosp16.1

Verify the changes:

$ grep 'Image' ../environments/trilio_env.yaml
   DockerTrilioDatamoverImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1
   DockerTrilioDmApiImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1
   ContainerHorizonImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1

RHOSP16.2

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.2/scripts/
sudo ./prepare_trilio_images.sh <undercloud_ip/hostname> <HOTFIX-TAG-VERSION>-rhosp16.2

Verify the changes

$ grep 'Image' ../environments/trilio_env.yaml
   DockerTrilioDatamoverImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.2
   DockerTrilioDmApiImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.2
   ContainerHorizonImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.2

The changes can be verified using the following commands.

(undercloud) [stack@undercloud redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>]$ openstack tripleo container image list | grep trilio
| docker://undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1                       |                        |
| docker://undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1                   |                   |
| docker://undercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1                  |

-----------------------------------------------------------------------------------------------------

(undercloud) [stack@undercloud redhat-director-scripts]$ grep 'Image' ../environments/trilio_env.yaml
   DockerTrilioDatamoverImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1
   DockerTrilioDmApiImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1
   ContainerHorizonImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1

4.3] Red Hat Satellite Server

Follow this section when a Satellite Server is used for the container registry.

Pull the Trilio containers on the Red Hat Satellite using the given

Populate the trilio_env.yaml with container urls.

RHOSP 13

$ grep 'Images' ../environments/trilio_env.yaml
   DockerTrilioDatamoverImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp13
   DockerTrilioDmApiImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp13
   DockerHorizonImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp13

RHOSP16.1

$ grep 'Image' trilio_env.yaml
   DockerTrilioDatamoverImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1
   DockerTrilioDmApiImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1
   ContainerHorizonImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1

RHOSP16.2

$ grep 'Image' trilio_env.yaml
   DockerTrilioDatamoverImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.2
   DockerTrilioDmApiImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.2
   ContainerHorizonImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.2

5] Provide environment details in trilio-env.yaml

The following information are required additionally:

Network for the datamover api
datamover password
Backup target type {nfs/s3}
In case of NFS
- list of NFS Shares
- NFS options
In case of S3
- S3 type {amazon_s3/ceph_s3}
- S3 Access key
- S3 Secret key
- S3 Region name
- S3 Bucket
- S3 Endpoint URL
- S3 Signature Version
- S3 Auth Version
- S3 SSL Enabled {true/false}
- S3 SSL Cert

Use ceph_s3 for any non-aws S3 backup targets.

resource_registry:
  OS::TripleO::Services::TrilioDatamover: ../services/trilio-datamover.yaml
  OS::TripleO::Services::TrilioDatamoverApi: ../services/trilio-datamover-api.yaml
  # NOTE: If there are addition customizations to the endpoint map (e.g. for
  # other integration), this will need to be regenerated.
  OS::TripleO::EndpointMap: endpoint_map.yaml

parameter_defaults:

   ## Enable Trilio's quota functionality on horizon
   ExtraConfig:
     horizon::customization_module: 'dashboards.overrides'

   ## Define network map for trilio datamover api service
   ServiceNetMap:
       TrilioDatamoverApiNetwork: internal_api

   ## Trilio Datamover Password for keystone and database
   TrilioDatamoverPassword: "test1234"

   ## Trilio container pull urls
   DockerTrilioDatamoverImage: devundercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1
   DockerTrilioDmApiImage: devundercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1

   ## If you do not want Trilio's horizon plugin to replace your horizon container, just comment following line.
   ContainerHorizonImage: devundercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1

   ## Backup target type nfs/s3, used to store snapshots taken by triliovault
   BackupTargetType: 'nfs'

   ## For backup target 'nfs'
   NfsShares: '192.168.122.101:/opt/tvault'
   NfsOptions: 'nolock,soft,timeo=180,intr,lookupcache=none'

   ## For backup target 's3'
   ## S3 type: amazon_s3/ceph_s3
   S3Type: 'amazon_s3'

   ## S3 access key
   S3AccessKey: ''

   ## S3 secret key
   S3SecretKey: ''

   ## S3 region, if your s3 does not have any region, just keep the parameter as it is
   S3RegionName: ''

   ## S3 bucket name
   S3Bucket: ''

   ## S3 endpoint url, not required for Amazon S3, keep it as it is
   S3EndpointUrl: ''

   ## S3 signature version
   S3SignatureVersion: 'default'

   ## S3 Auth version
   S3AuthVersion: 'DEFAULT'

   ## If S3 backend is not Amazon S3 and SSL is enabled on S3 endpoint url then change it to 'True', otherwise keep it as 'False'
   S3SslEnabled: False

   ## If S3 backend is not Amazon S3 and SSL is enabled on S3 endpoint URL and SSL certificates are self signed, then
   ## user need to set this parameter value to: '/etc/tvault-contego/s3-cert.pem', otherwise keep it's value as empty string.
   S3SslCert: ''

   ## Don't edit following parameter
   EnablePackageInstall: True

6. Advanced Settings/Configuration

6.1 Haproxy customized configuration for Trilio dmapi service __

Following is the haproxy conf file location on haproxy nodes of the overcloud. Trilio datamover api service haproxy configuration gets added to this file.

/var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg

Trilio datamover haproxy default configuration from the above file looks as follows:

listen trilio_datamover_api
  bind 172.25.0.107:13784 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 172.25.0.107:8784 transparent
  balance roundrobin
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  http-request set-header X-Forwarded-Port %[dst_port]
  maxconn 50000
  option httpchk
  option httplog
  retries 5
  timeout check 10m
  timeout client 10m
  timeout connect 10m
  timeout http-request 10m
  timeout queue 10m
  timeout server 10m
  server overcloud-controller-0.internalapi.localdomain 172.25.0.106:8784 check fall 5 inter 2000 rise 2

The user can change the following configuration parameter values.

retries 5
timeout http-request 10m
timeout queue 10m
timeout connect 10m
timeout client 10m
timeout server 10m
timeout check 10m
balance roundrobin
maxconn 50000

For RHOSP13

/home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp13/services/trilio-datamover-api.yaml

For RHOSP16.0

/home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16/services/trilio-datamover-api.yaml

For RHOSP16.1

/home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/services/trilio-datamover-api.yaml

For RHOSP16.2

/home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.2/services/trilio-datamover-api.yaml

ii) Search the following entries and edit as required

          tripleo::haproxy::trilio_datamover_api::options:
             'retries': '5'
             'maxconn': '50000'
             'balance': 'roundrobin'
             'timeout http-request': '10m'
             'timeout queue': '10m'
             'timeout connect': '10m'
             'timeout client': '10m'
             'timeout server': '10m'
             'timeout check': '10m'

iii) Save the changes.

7] Deploy overcloud with trilio environment

Use the following heat environment file and roles data file in overcloud deploy command:

trilio_env.yaml
roles_data.yaml
Use the correct Trilio endpoint map file as per available Keystone endpoint configuration
1. Instead of tls-endpoints-public-dns.yaml file, use environments/trilio_env_tls_endpoints_public_dns.yaml
2. Instead of tls-endpoints-public-ip.yaml file, useenvironments/trilio_env_tls_endpoints_public_ip.yaml
3. Instead of tls-everywhere-endpoints-dns.yaml file, useenvironments/trilio_env_tls_everywhere_dns.yaml

To include new environment files use '-e' option and for roles data file use '-r' option. An example overcloud deploy command is shown below:

openstack overcloud deploy --templates \
  -e /home/stack/templates/node-info.yaml \
  -e /home/stack/templates/overcloud_images.yaml \
  -e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/environments/trilio_env.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-tls.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/inject-trust-anchor.yaml \
  -e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/environments/trilio_env_tls_endpoints_public_dns.yaml \
  --ntp-server 192.168.1.34 \
  --libvirt-type qemu \
  --log-file overcloud_deploy.log \
  -r /usr/share/openstack-tripleo-heat-templates/roles_data.yaml

8] Verify deployment

If the containers are in restarting state or not listed by the following command then your deployment is not done correctly. Please recheck if you followed the complete documentation.

8.1] On Controller node

[root@overcloud-controller-0 heat-admin]# podman ps | grep trilio
26fcb9194566  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-REL-VERSION>-rhosp16.1        kolla_start           5 days ago  Up 5 days ago         trilio_dmapi
094971d0f5a9  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-REL-VERSION>-rhosp16.1       kolla_start           5 days ago  Up 5 days ago         horizon

Verify the haproxy configuration under:

/var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg

8.2] On Compute node

Make sure Trilio datamover container is in running state and no other Trilio container is deployed on compute nodes.

[root@overcloud-novacompute-0 heat-admin]# podman ps | grep trilio
b1840444cc59  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-REL-VERSION>-rhosp16.1                    kolla_start           5 days ago  Up 5 days ago         trilio_datamover

8.3] On the node with Horizon service

[root@overcloud-controller-0 heat-admin]# podman ps | grep horizon
094971d0f5a9  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-REL-VERSION>-rhosp16.1       kolla_start           5 days ago  Up 5 days ago         horizon

9] Additional Steps on Trilio Appliance

9.1] Change the nova user id on the Trilio Nodes

In RHOSP, nova user id on nova-compute docker container is set to '42436'. The 'nova' user id on the Trilio nodes needs to be set the same. Do the following steps on all Trilio nodes:

Download the shell script that will change the user id
Assign executable permissions
Execute the script
Verify that nova user and group id have changed to '42436'

## Download the shell script
$ curl -O https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/master/common/nova_userid.sh

## Assign executable permissions
$ chmod +x nova_userid.sh

## Execute the shell script to change 'nova' user and group id to '42436'
$ ./nova_userid.sh

## Ignore any errors and verify that 'nova' user and group id has changed to '42436'
$ id nova
   uid=42436(nova) gid=42436(nova) groups=42436(nova),990(libvirt),36(kvm)

10] Troubleshooting for overcloud deployment failures

Trilio components will be deployed using puppet scripts.

In case if the overcloud deployment is failing, do the following command to provide the list of errors. The following document also provides valuable insights:

openstack stack failures list overcloud
heat stack-list --show-nested -f "status=FAILED"
heat resource-list --nested-depth 5 overcloud | grep FAILED

=> If trilio datamover api containers does not start well or in restarting state, use following logs to debug.


docker logs trilio_dmapi

tailf /var/log/containers/trilio-datamover-api/dmapi.log

 

=> If trilio datamover containers does not start well or in restarting state, use following logs to debug.


docker logs trilio_datamover

tailf /var/log/containers/trilio-datamover/tvault-contego.log