Trilio supports the upgrade of charms and Trilio components from older releases (4.0 onwards) to the latest release. More information about the latest 4.2 Trilio charms for the Trilio-4.2 release can be found here.

The following charms exist:

• trilio-wlm Installs and manages Trilio Controller services.

• trilio-dm-api Installs and manages the Trilio Datamover API service.

• trilio-data-mover Installs and manages the Trilio Datamover service.

• trilio-horizon-plugin Installs and manages the Trilio Horizon Plugin.

The documentation of the charms can be found here:

Upgrading the Trilio Charms

More detailed information about the latest 4.2 Trilio charms for the Trilio-4.2 release can be found here.

Upgrading the Trilio Charms up to OpenStack Wallaby release

Follow the steps mentioned in this document to upgrade the charms to the latest 4.2 charms before upgrading the Trilio packages.

Upgrading the Trilio Charms to OpenStack Yoga release onwards

Follow the steps mentioned below to upgrade the charms to the latest 4.2 charms before upgrading the Trilio packages.

General Prerequisites

1. No snapshot OR restore are to be running during this process.

2. Global job scheduler should be disabled

1. Upgrade juju charms

juju [-m <model>] upgrade-charm trilio-wlm --switch trilio-charmers-trilio-wlm-focal --channel latest/edge

juju [-m <model>] upgrade-charm trilio-horizon-plugin --switch trilio-charmers-trilio-horizon-plugin-focal --channel latest/edge

juju [-m <model>] upgrade-charm trilio-dm-api --switch trilio-charmers-trilio-dm-api-focal --channel latest/edge

juju [-m <model>] upgrade-charm trilio-data-mover --switch trilio-charmers-trilio-data-mover-focal --channel latest/edge

2. Wait till all trilio units are active/idle then Restart all the trilio services

juju [-m <model>] exec --application trilio-dm-api "sudo systemctl restart tvault-datamover-api"

juju [-m <model>] exec --application trilio-data-mover "sudo systemctl restart tvault-contego"

juju [-m <model>] exec --application trilio-horizon-plugin "sudo systemctl restart apache2"

3. Select Single Node or HA steps as appropriate to restart services

For setup with single node wlm unit

juju [-m <model>] exec --application trilio-wlm "sudo systemctl restart wlm-api wlm-scheduler wlm-workloads wlm-cron"

For setups with 3 node HA enabled wlm

juju [-m <model>] exec --application trilio-wlm "systemctl restart wlm-api wlm-scheduler wlm-workloads"

juju [-m <model>] exec --unit trilio-wlm/leader "sudo crm resource restart res_trilio_wlm_wlm_cron"

4. Ensure that wlm-cron is only running on a single Unit in the output of this command

juju [-m <model>] exec --application trilio-wlm "sudo systemctl status wlm-cron"

If wlm-cron is running in more than one unit, or nowhere at all, this can be fixed by following the below steps:

juju [-m <model>] exec --unit trilio-wlm/leader "sudo crm resource stop res_trilio_wlm_wlm_cron" juju [-m <model>] exec --application trilio-wlm "sudo systemctl stop wlm-cron" juju [-m <model>] exec --application trilio-wlm "sudo ps -ef | grep workloadmgr-cron | grep -v grep"

No workloadmgr-cron process should be running. If it's still running somewhere, login to that unit and stop the service manually with 'sudo systemctl stop wlm-cron'

juju [-m <model>] exec --unit trilio-wlm/leader "sudo crm resource start res_trilio_wlm_wlm_cron" juju [-m <model>] exec --application trilio-wlm "sudo systemctl status wlm-cron"

5. Check upgraded juju charm & trilio services

Check the trilio unit's status in juju status [-m <model>] | grep trilio output.

All the trilio units will be reporting the new juju charm version.

trilio-data-mover 4.2.64.26 active 3 trilio-charmers-trilio-data-mover-jammy latest/edge 4 no Unit is ready trilio-dm-api 4.2.64.2 active 1 trilio-charmers-trilio-dm-api-jammy latest/edge 2 no Unit is ready trilio-horizon-plugin 4.2.64.5 active 1 trilio-charmers-trilio-horizon-plugin-jammy latest/edge 4 no Unit is ready trilio-wlm 4.2.64.20 active 1 trilio-charmers-trilio-wlm-jammy latest/edge 3 no Unit is ready

6. Check that all Trilio services are running

juju [-m <model>] exec --application trilio-dm-api "sudo systemctl status tvault-datamover-api"

juju [-m <model>] exec --application trilio-data-mover "sudo systemctl status tvault-contego"

juju [-m <model>] exec --application trilio-horizon-plugin "sudo systemctl status apache2"

juju [-m <model>] exec --application trilio-wlm "sudo systemctl status wlm-api wlm-scheduler wlm-workloads wlm-cron"

Next move on the Updating the Trilio packages.

Upgrading the Trilio packages

Trilio is releasing hotfixes, which require updating the packages inside the containers. These hotfixes can not be installed using the Juju charms as they don't require an update to the charms.

Generic Prerequisites

1. Trilio packages can be upgraded after deployment. Trilio supports upgrade to the latest 4.2 releases from as old as the Trilio 4.0 release.

2. No snapshot OR restore are to be running during this process.

3. Global job scheduler should be disabled.

4. wlm-cron should be disabled ( Following sets of commands are to be run on MAAS node).

5. If trilio-wlm is HA enabled, set the cluster configuration to maintenance mode ( this command will fail for single node deployment).

juju exec [-m <model>] --unit trilio-wlm/leader "sudo crm configure property maintenance-mode=true"

1. Stop the wlm-cron service

juju exec [-m <model>] --application trilio-wlm "sudo systemctl stop wlm-cron"

1. Ensure that no stale wlm-cron processes exist

juju exec [-m <model>] --application trilio-wlm "sudo ps -ef | grep [w]orkloadmgr-cron"

If any stale process are found, they should be stopped manually.

Upgrade package on trilio units

The deployed Trilio version is controlled by the triliovault-pkg-source charm configuration option.

The gemfury repository should be accessible from all Trilio units. For each trilio charm, it should be pointing to the following Gemfury repository:

deb [trusted=yes] https://apt.fury.io/trilio-4-3/ /

This can be checked via juju [-m ] config trilio-wlm triliovault-pkg-source command output.

The preferred, recommended, and tested method to update the packages is through the Juju command line.

Run the below commands from MAAS node

On Ubuntu Bionic environments

juju exec [-m <model>] --application trilio-wlm 'sudo apt-get update && sudo apt-get install -y --only-upgrade -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" workloadmgr python3-workloadmgrclient python3-contegoclient s3-fuse-plugin'
juju exec [-m <model>] --application trilio-horizon-plugin 'sudo apt-get update && sudo apt-get install -y --only-upgrade -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" tvault-horizon-plugin python-workloadmgrclient'
juju exec [-m <model>] --application trilio-dm-api 'sudo apt-get update && sudo apt-get install -y --only-upgrade -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" python3-dmapi'
juju exec [-m <model>] --application trilio-data-mover 'sudo apt-get update && sudo apt-get install -y --only-upgrade -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" tvault-contego s3-fuse-plugin'

On other (not Ubuntu Bionic) environments

juju exec [-m <model>] --application trilio-wlm 'sudo apt-get update && sudo apt-get install -y --only-upgrade -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" workloadmgr python3-workloadmgrclient python3-contegoclient python3-s3-fuse-plugin'
juju exec [-m <model>] --application trilio-horizon-plugin 'sudo apt-get update && sudo apt-get install -y --only-upgrade -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" python3-tvault-horizon-plugin python3-workloadmgrclient'
juju exec [-m <model>] --application trilio-dm-api 'sudo apt-get update && sudo apt-get install -y --only-upgrade -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" python3-dmapi'
juju exec [-m <model>] --application trilio-data-mover 'sudo apt-get update && sudo apt-get install -y --only-upgrade -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" python3-tvault-contego python3-s3-fuse-plugin'

Check the trilio unit's status in juju status [-m ] | grep trilio output. All the trilio units will be with the new packages.

trilio-data-mover      <package version>  active      3  trilio-data-mover      jujucharms    8  ubuntu
trilio-dm-api          <package version>  active      1  trilio-dm-api          jujucharms    5  ubuntu
trilio-horizon-plugin  <package version>  active      1  trilio-horizon-plugin  jujucharms    4  ubuntu
trilio-wlm             <package version>  active      3  trilio-wlm             jujucharms    7  ubuntu

Update the DB schema

Run the below command to update the schema

juju exec [-m <model>] --unit trilio-wlm/leader "alembic -c /etc/workloadmgr/alembic.ini upgrade heads"

Check the schema head with below command. It should point to latest schema head.

juju exec [-m <model>] --unit trilio-wlm/leader "alembic -c /etc/workloadmgr/alembic.ini current"

Enable mount-bind for NFS

T4O 4.2 is changing how the mount point is calculated. It is required to setup a mount bind to make T4O 4.1 or older backups available.

Please refer to this page for detailed steps to set up the mount bind.

Post-Upgrade steps

1. If the trilio-wlm nodes are HA enabled:

   1. Make sure the wlm-cron services are down after the pkg upgrade. Run the following command for the same:

juju exec [-m <model>] --application trilio-wlm "sudo systemctl stop wlm-cron"

1. Unset the cluster maintenance mode

juju exec [-m <model>] --unit trilio-wlm/leader "sudo crm configure property maintenance-mode=false"

1. Make sure the wlm-cron service up and running on any one node.

juju exec [-m <model>] --application trilio-wlm "sudo systemctl status wlm-cron"

1. Set the Global Job Scheduler to the original state.

Troubleshooting

If any trilio unit gets into an error state with the message :

hook failed: "update-status"

One of the reasons could be the package installation did not finish correctly. One way to verify that is by logging into that unit and following the below steps;

Login to trilio unit and run "sudo dpkg --configure -a"
It will ask for user input, hit enter and log out from the unit.
From mass node run command "juju resolve <trilio unit name>"

Generic Prerequisites

• Please ensure to complete the upgrade of all the Trilio components on the Openstack controller & compute nodes before starting the rolling upgrade of Trilio.

• The mentioned Gemfury repository should be accessible from Trilio VM.

• Either 4.2 GA OR any hotfix patch against 4.2 should be already deployed for performing upgrades mentioned in the current document.

• Please ensure the following points before starting the upgrade process:

  • No snapshot OR restore to be running.

  • Global job-scheduler should be disabled.

  • wlm-cron should be disabled and any lingering process should be killed.

Deactivating the wlm-cron service

The following sets of commands will disable the wlm-cron service and verify that it is has been completly shut-down.

[root@TVM2 ~]# pcs resource disable wlm-cron
[root@TVM2 ~]# systemctl status wlm-cron
● wlm-cron.service - workload's scheduler cron service
   Loaded: loaded (/etc/systemd/system/wlm-cron.service; disabled; vendor preset                                                                                                                                                                                                                                                                                                                                                           : disabled)
   Active: inactive (dead)

Jun 11 08:27:06 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:06 - INFO - 1...t
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 140686268624368 Child 11389 ki...5
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - 1...5
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: Shutting down thread pool
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - S...l
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: Stopping the threads
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - S...s
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: All threads are stopped succes...y
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - A...y
Jun 11 08:27:09 TVM2 systemd[1]: Stopped workload's scheduler cron service.
Hint: Some lines were ellipsized, use -l to show in full.
[root@TVM2 ~]# pcs resource show wlm-cron
 Resource: wlm-cron (class=systemd type=wlm-cron)
  Meta Attrs: target-role=Stopped
  Operations: monitor interval=30s on-fail=restart timeout=300s (wlm-cron-monito                                                                                                                                                                                                                                                                                                                                                           r-interval-30s)
              start interval=0s on-fail=restart timeout=300s (wlm-cron-start-int                                                                                                                                                                                                                                                                                                                                                           erval-0s)
              stop interval=0s timeout=300s (wlm-cron-stop-interval-0s)
[root@TVM2 ~]# ps -ef | grep -i workloadmgr-cron
root     15379 14383  0 08:27 pts/0    00:00:00 grep --color=auto -i workloadmgr                                                                                                                                                                                                                                                                                                                                                           -cron

Steps to upgrade Trilio having Internet access

1. Download the hf_upgrade.sh script on all T4O nodes where the upgrade is to be done.

cd /opt/ 
wget https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/4.3.2/TVOAppliance/hf_upgrade.sh
chmod +x hf_upgrade.sh

You can check the usage of this script here.

1. Run the following command to download and Install the upgraded packages

./hf_upgrade.sh --all  
OR 
./hf_upgrade.sh -a

1. Run the following command to enable the wlm-cron service

pcs resource enable wlm-cron

Steps to upgrade Trilio having No Internet access

Here the packages need to be downloaded on a separate host which has internet access. The script ./hf_upgrade.sh can be used with the below-mentioned option to download the required package

1. Download the hf_upgrade.sh script and use the script to download the upgraded packages. You can check the usage of this script here.\

   Copy

   cd /<tvo_packages_download_path>/ 
   wget https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/4.3.2/TVOAppliance/hf_upgrade.sh
   chmod +x hf_upgrade.sh
   
   # Download the upgraded packages
   ./hf_upgrade.sh --downloadonly
   

2. Copy the downloaded 4.3-offlinePkgs.tar.gz package and hf_upgrade.sh script to all the Trilio nodes\

   Copy

   scp <tvo_packages_download_path>/* root@<TrilioVault_node_IP>:/<path_to_upgrade_package>/

3. Now access the appliance VMs and install the copied offline packages on each of the Trilio appliances by running the below-mentioned command:\

   Copy

   cd <path_to_upgrade_package>/
   ./hf_upgrade.sh --installonly
   

4. Once the installation is done, run the following command to enable the wlm-cron service\

   Copy

   pcs resource enable wlm-cron
   

Post upgrade steps

1. Verify all wlm services on all Trilio nodes

   Copy

   systemctl status tvault-config wlm-workloads wlm-api wlm-scheduler
   pcs status (on primary node)
   systemctl status wlm-cron (on primary node)
   systemctl status tvault-object-store (only if Trilio configured with S3 backend storage)

2. Make sure all wlm services are up and running from python version 3.8.x

3. Check the mount point using “df -h” command

4. Grafana dashboard shows the correct wlm service status on all T4O nodes.

5. Enable Global Job Scheduler

   ---

Additional check for wlm-cron on the primary node

ps -ef | grep workloadmgr-cron | grep -v grep
# Above command should show only 2 processes running; sample below

[root@tvm6 ~]# ps -ef | grep workloadmgr-cron | grep -v grep
nova      8841     1  2 Jul28 ?        00:40:44 /home/stack/myansible/bin/python3 /home/stack/myansible/bin/workloadmgr-cron --config-file=/etc/workloadmgr/workloadmgr.conf
nova      8898  8841  0 Jul28 ?        00:07:03 /home/stack/myansible/bin/python3 /home/stack/myansible/bin/workloadmgr-cron --config-file=/etc/workloadmgr/workloadmgr.conf

[RHOSP, TripleO and Kolla only] Verify nova UID/GID for nova user on the Appliance

Red Hat OpenStack, TripleO, and Kolla Ansible Openstack are using the nova UID/GID of 42436 inside their containers instead of 162:162 which is the standard in other Openstack environments.

Please verify that the nova UID/GID on the Trilio Appliance is still 42436, to do so follow the below document provided here:

• Update nova UID/GID on the appliance

0] Pre-requisites

Please ensure the following points are met before starting the upgrade process:

• No Snapshot or Restore is running

• Global job scheduler is disabled

• wlm-cron is disabled on the Trilio Appliance

Deactivating the wlm-cron service

The following sets of commands will disable the wlm-cron service and verify that it is has been completely shut-down.

[root@TVM2 ~]# pcs resource disable wlm-cron
[root@TVM2 ~]# systemctl status wlm-cron
● wlm-cron.service - workload's scheduler cron service
   Loaded: loaded (/etc/systemd/system/wlm-cron.service; disabled; vendor preset                                                                                                                                                                                                                                                                                                                                                           : disabled)
   Active: inactive (dead)

Jun 11 08:27:06 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:06 - INFO - 1...t
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 140686268624368 Child 11389 ki...5
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - 1...5
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: Shutting down thread pool
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - S...l
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: Stopping the threads
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - S...s
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: All threads are stopped succes...y
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - A...y
Jun 11 08:27:09 TVM2 systemd[1]: Stopped workload's scheduler cron service.
Hint: Some lines were ellipsized, use -l to show in full.
[root@TVM2 ~]# pcs resource show wlm-cron
 Resource: wlm-cron (class=systemd type=wlm-cron)
  Meta Attrs: target-role=Stopped
  Operations: monitor interval=30s on-fail=restart timeout=300s (wlm-cron-monito                                                                                                                                                                                                                                                                                                                                                           r-interval-30s)
              start interval=0s on-fail=restart timeout=300s (wlm-cron-start-int                                                                                                                                                                                                                                                                                                                                                           erval-0s)
              stop interval=0s timeout=300s (wlm-cron-stop-interval-0s)
[root@TVM2 ~]# ps -ef | grep -i workloadmgr-cron
root     15379 14383  0 08:27 pts/0    00:00:00 grep --color=auto -i workloadmgr                                                                                                                                                                                                                                                                                                                                                           -cron

1] [On Undercloud node] Clone latest Trilio repository and upload Trilio puppet module

1.1] Clone Trilio cfg scripts repository

cd /home/stack
mv triliovault-cfg-scripts triliovault-cfg-scripts-old
git clone -b 4.3.2 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/

Separate directories are created as per Redhat OpenStack release under 'triliovault-cfg-scripts/redhat-director-scripts/' directory. Use all scripts/templates from respective directory. For ex, if your RHOSP release is 13, then use scripts/templates from 'triliovault-cfg-scripts/redhat-director-scripts/rhosp13' directory only.

1.2] If backup target type is 'Ceph based S3' with SSL:

If your backup target is ceph S3 with SSL and SSL certificates are self signed or authorized by private CA, then user needs to provide CA chain certificate to validate the SSL requests. For that, user needs to rename his ca chain cert file to 's3-cert.pem' and copy it into the puppet directory of the right release.

cp s3-cert.pem /home/stack/triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/puppet/trilio/files/

2] Upload Trilio puppet module

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/scripts/
./upload_puppet_module.sh

## Output of above command looks like following for RHOSP13, RHOSP16.1 and RHOSP16.2
Creating tarball...
Tarball created.
Creating heat environment file: /home/stack/.tripleo/environments/puppet-modules-url.yaml
Uploading file to swift: /tmp/puppet-modules-8Qjya2X/puppet-modules.tar.gz
+-----------------------+---------------------+----------------------------------+
| object                | container           | etag                             |
+-----------------------+---------------------+----------------------------------+
| puppet-modules.tar.gz | overcloud-artifacts | 368951f6a4d39cfe53b5781797b133ad |
+-----------------------+---------------------+----------------------------------+

## Output of above command looks like following for RHOSP17.0
Creating tarball...
Tarball created.
renamed '/tmp/puppet-modules-P3duCg9/puppet-modules.tar.gz' -> '/var/lib/tripleo/artifacts/overcloud-artifacts/puppet-modules.tar.gz'
Creating heat environment file: /home/stack/.tripleo/environments/puppet-modules-url.yaml

## Above command creates following file.
ls -ll /home/stack/.tripleo/environments/puppet-modules-url.yaml

3] Update overcloud roles data file to include Trilio services

Trilio has two services as explained below. You need to add these two services to your roles_data.yaml. If you do not have customized roles_data file, you can find your default roles_data.yaml file at /usr/share/openstack-tripleo-heat-templates/roles_data.yaml on undercloud.

You need to find that role_data file and edit it to add the following Trilio services.

i) Trilio Datamover Api Service:

Service Entry in roles_data yaml: OS::TripleO::Services::TrilioDatamoverApi

Typically this service should be deployed on controller nodes where keystone and database runs. If you are using RHOSP's pre-defined roles, you need to addOS::TripleO::Services::TrilioDatamoverApiservice to Controller role.

ii) Trilio Datamover Service: Service Entry in roles_data yaml: OS::TripleO::Services::TrilioDatamover This service should be deployed on role where nova-compute service is running.

If you are using RHOSP's pre-defined roles, you need to add our OS::TripleO::Services::TrilioDatamover service to Compute role.

If you have defined your custom roles, then you need to identify the role name where in 'nova-compute' service is running and then you need to add 'OS::TripleO::Services::TrilioDatamover' service to that role.

iii) Trilio Horizon Service:

This service needs to share the same role as the OpenStack Horizon server. In the case of the pre-defined roles will the Horizon service run on the role Controller. Add the following to the identified role OS::TripleO::Services::TrilioHorizon

4] Prepare latest Trilio container images

Trilio containers are pushed to 'RedHat Container Registry'. Registry URL is 'registry.connect.redhat.com'. The Trilio container URLs are as follows:

4.1] available container images

RHOSP 13

Trilio Datamover container:       registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp13
Trilio Datamover Api Container:   registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp13
Trilio horizon plugin:            registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp13

RHOSP 16.1

Trilio Datamover container:       registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1
Trilio Datamover Api Container:   registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1
Trilio horizon plugin:            registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1

RHOSP 16.2

Trilio Datamover container:       registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.2
Trilio Datamover Api Container:   registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.2
Trilio horizon plugin:            registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.2

RHOSP 17.0

Trilio Datamover container:       registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp17.0
Trilio Datamover Api Container:   registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp17.0
Trilio horizon plugin:            registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp17.0

There are three registry methods available in RedHat Openstack Platform.

1. Remote Registry

2. Local Registry

3. Satellite Server

4.2] Remote Registry

Follow this section when 'Remote Registry' is used.

For this method it is not necessary to pull the containers in advance. It is only necessary to populate the trilio_env.yaml file with the Trilio container URLs from Redhat registry.

Populate the trilio_env.yaml with container URLs for:

• Trilio Datamover container

• Trilio Datamover api container

• Trilio Horizon Plugin

Example

# For RHOSP13
$ grep 'Image' trilio_env.yaml
   DockerTrilioDatamoverImage: registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp13
   DockerTrilioDmApiImage: registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp13
   DockerHorizonImage: registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp13

# For RHOSP16.1
$ grep 'Image' trilio_env.yaml
   DockerTrilioDatamoverImage: registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1
   DockerTrilioDmApiImage: registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1
   ContainerHorizonImage: registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1
   
# For RHOSP16.2
$ grep 'Image' trilio_env.yaml
   DockerTrilioDatamoverImage: registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.2
   DockerTrilioDmApiImage: registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.2
   ContainerHorizonImage: registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.2

# For RHOSP17.0
$ grep 'Image' trilio_env.yaml
   DockerTrilioDatamoverImage: registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp17.0
   DockerTrilioDmApiImage: registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp17.0
   ContainerHorizonImage: registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp17.0

4.2] Local Registry

Follow this section when 'local registry' is used on the undercloud.

In this case it is necessary to push the Trilio containers to the undercloud registry. Trilio provides shell scripts which will pull the containers from 'registry.connect.redhat.com' and push them to the undercloud and updates the trilio_env.yaml.

RHOSP13 example

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/scripts/

./prepare_trilio_images.sh <undercloud_ip> <container_tag>

# Example:
./prepare_trilio_images.sh 192.168.13.34 <HOTFIX-TAG-VERSION>-rhosp13

## Verify changes
# For RHOSP13
$ grep '<HOTFIX-TAG-VERSION>-rhosp13' ../environments/trilio_env.yaml
   DockerTrilioDatamoverImage: 172.25.2.2:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp13
   DockerTrilioDmApiImage: 172.25.2.2:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp13
   DockerHorizonImage: 172.25.2.2:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp13

RHOSP 16.1, RHOSP16.2 and RHOSP17.0 example

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/scripts/

sudo ./prepare_trilio_images.sh <UNDERCLOUD_REGISTRY_HOSTNAME> <CONTAINER_TAG> 

## Run following command to find 'UNDERCLOUD_REGISTRY_HOSTNAME'. 
-- In the below example 'trilio-undercloud.ctlplane.localdomain' is <UNDERCLOUD_REGISTRY_HOSTNAME>
$ openstack tripleo container image list | grep keystone
| docker://trilio-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-keystone:16.0-82                       |
| docker://trilio-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-barbican-keystone-listener:16.0-84   

## 'CONTAINER_TAG' format for RHOSP16.1: <<HOTFIX-TAG-VERSION>>-rhosp16.1
## 'CONTAINER_TAG' format for RHOSP16.2: <<HOTFIX-TAG-VERSION>>-rhosp16.2
## 'CONTAINER_TAG' format for RHOSP17.0: <<HOTFIX-TAG-VERSION>>-rhosp17.0

## Example
sudo ./prepare_trilio_images.sh trilio-undercloud.ctlplane.localdomain <HOTFIX-TAG-VERSION>-rhosp16.1

The changes can be verified using the following commands.

(undercloud) [stack@undercloud redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>]$ openstack tripleo container image list | grep trilio
| docker://undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1                       |                        |
| docker://undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1                   |                   |
| docker://undercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1                  |

-----------------------------------------------------------------------------------------------------

(undercloud) [stack@undercloud redhat-director-scripts]$ grep 'Image' ../environments/trilio_env.yaml
   DockerTrilioDatamoverImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1
   DockerTrilioDmApiImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1
   ContainerHorizonImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1

4.3] Red Hat Satellite Server

Follow this section when a Satellite Server is used for the container registry.

Pull the Trilio containers on the Red Hat Satellite using the given Red Hat registry URLs.

Populate the trilio_env.yaml with container urls.

RHOSP 13 example

$ grep '<HOTFIX-TAG-VERSION>-rhosp13' ../environments/trilio_env.yaml
   DockerTrilioDatamoverImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp13
   DockerTrilioDmApiImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp13
   DockerHorizonImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp13

RHOSP 16.1, RHOSP 16.2 and RHOSP 17.0

## For RHOSP16.1
$ grep 'Image' trilio_env.yaml
   DockerTrilioDatamoverImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1
   DockerTrilioDmApiImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1
   ContainerHorizonImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1

## For RHOSP16.2
$ grep 'Image' trilio_env.yaml
   DockerTrilioDatamoverImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.2
   DockerTrilioDmApiImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.2
   ContainerHorizonImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.2

## For RHOSP7.0
$ grep 'Image' trilio_env.yaml
   DockerTrilioDatamoverImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp17.0
   DockerTrilioDmApiImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp17.0
   ContainerHorizonImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp17.0

5. Verify Trilio environment details

It is recommended to re-populate the backup target details in the freshly downloaded trilio_env.yaml file. This will ensure that parameters that have been added since the last update/installation of Trilio are available and will be filled out too.

Locations of the trilio_env.yaml:

RHOSP13: /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp13/environments/trilio_env.yaml
RHOSP16.1: /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/environments/trilio_env.yaml
RHOSP16.2: /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.2/environments/trilio_env.yaml
RHOSP17.0: /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp17.0/environments/trilio_env.yaml

For more details about the trilio_env.yaml please check here.

6] Configure multi-IP NFS

On Undercloud node, change directory

cd triliovault-cfg-scripts/common/

Edit file 'triliovault_nfs_map_input.yml' in the current directory and provide compute host and NFS share/ip map.

Get the compute hostnames from the following command. Check ‘Name' column. Use exact hostnames in 'triliovault_nfs_map_input.yml' file.

(undercloud) [stack@ucqa161 ~]$ openstack server list
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+
| ID                                   | Name                          | Status | Networks             | Image          | Flavor  |
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+
| 8c3d04ae-fcdd-431c-afa6-9a50f3cb2c0d | overcloudtrain1-controller-2  | ACTIVE | ctlplane=172.30.5.18 | overcloud-full | control |
| 103dfd3e-d073-4123-9223-b8cf8c7398fe | overcloudtrain1-controller-0  | ACTIVE | ctlplane=172.30.5.11 | overcloud-full | control |
| a3541849-2e9b-4aa0-9fa9-91e7d24f0149 | overcloudtrain1-controller-1  | ACTIVE | ctlplane=172.30.5.25 | overcloud-full | control |
| 74a9f530-0c7b-49c4-9a1f-87e7eeda91c0 | overcloudtrain1-novacompute-0 | ACTIVE | ctlplane=172.30.5.30 | overcloud-full | compute |
| c1664ac3-7d9c-4a36-b375-0e4ee19e93e4 | overcloudtrain1-novacompute-1 | ACTIVE | ctlplane=172.30.5.15 | overcloud-full | compute |
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+

Edit input map file and fill all the details. Refer to the this page for details about the structure.

vi triliovault_nfs_map_input.yml

Update pyyaml on the undercloud node only

## On Python3 env 
sudo pip3 install PyYAML==5.1 3 

## On Python2 env 
sudo pip install PyYAML==5.1

Expand the map file to create a one-to-one mapping of the compute nodes and the nfs shares.

## On Python3 env 
python3 ./generate_nfs_map.py 
 
## On Python2 env 
python ./generate_nfs_map.py

The result will be in file - 'triliovault_nfs_map_output.yml'

Validate output map file

Open file 'triliovault_nfs_map_output.yml

vi triliovault_nfs_map_output.yml

available in the current directory and validate that all compute nodes are covered with all the necessary NFS shares.

Append this output map file to 'triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_nfs_map.yaml'

grep ':.*:' triliovault_nfs_map_output.yml >> ../redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_nfs_map.yaml

Validate the changes in file 'triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_nfs_map.yaml'

Include the environment file (trilio_nfs_map.yaml) in overcloud deploy command with '-e' option as shown below.

-e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_nfs_map.yaml

Ensure that MultiIPNfsEnabled is set to true in trilio_env.yaml file and that nfs is used as backup target.

7] Update Overcloud Trilio components

Use the following heat environment file and roles data file in overcloud deploy command:

1. trilio_env.yaml

2. roles_data.yaml

3. Use correct Trilio endpoint map file as per available Keystone endpoint configuration

   1. Instead of tls-endpoints-public-dns.yaml file, use environments/trilio_env_tls_endpoints_public_dns.yaml

   2. Instead of tls-endpoints-public-ip.yaml file, useenvironments/trilio_env_tls_endpoints_public_ip.yaml

   3. Instead of tls-everywhere-endpoints-dns.yaml file, useenvironments/trilio_env_tls_everywhere_dns.yaml

To include new environment files use '-e' option and for roles data file use '-r' option. An example overcloud deploy command is shown below:

openstack overcloud deploy --templates \
  -e /home/stack/templates/node-info.yaml \
  -e /home/stack/templates/overcloud_images.yaml \
  -e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/environments/trilio_env.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-tls.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/inject-trust-anchor.yaml \
  -e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/environments/trilio_env_tls_endpoints_public_dns.yaml \
  -e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/environments/trilio_nfs_map.yaml \
  --ntp-server 192.168.1.34 \
  --libvirt-type qemu \
  --log-file overcloud_deploy.log \
  -r /home/stack/templates/roles_data.yaml

8] Verify deployment

8.1] On Controller node

Make sure Trilio dmapi and horizon containers are in a running state and no other Trilio container is deployed on controller nodes. When the role for these containers is not "controller" check on respective nodes according to configured roles_data.yaml.

[root@overcloud-controller-0 heat-admin]# podman ps | grep trilio
26fcb9194566  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.2        kolla_start           5 days ago  Up 5 days ago         trilio_dmapi
094971d0f5a9  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.2       kolla_start           5 days ago  Up 5 days ago         horizon

8.2] On Compute node

Make sure Trilio datamover container is in running state and no other Trilio container is deployed on compute nodes.

[root@overcloud-novacompute-0 heat-admin]# podman ps | grep trilio
b1840444cc59  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.2                    kolla_start           5 days ago  Up 5 days ago         trilio_datamover

8.3] On the node with Horizon service

Make sure horizon container is in running state. Please note that 'Horizon' container is replaced with Trilio Horizon container. This container will have latest OpenStack horizon + Trilio's horizon plugin.

[root@overcloud-controller-0 heat-admin]# podman ps | grep horizon
094971d0f5a9  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.2       kolla_start           5 days ago  Up 5 days ago         horizon

If Trilio Horizon container is in restarted state on RHOSP 16.1.8/RHSOP 16.2.4 then use below workaroud

## Either of the below workarounds should be performed on all the controller nodes where issue occurs for horizon pod.

option-1: Restart the memcached service on controller using systemctl (command: systemctl restart tripleo_memcached.service)

option-2: Restart the memcached pod (command: podman restart memcached)

9] Enable mount-bind for NFS

In T4O 4.2 and later releases, deriving of the mount point has been changed. It is necessary to set up the mount-bind to make T4O 4.1 or older backups available for T4O 4.2 and onwards releases.

Please follow this documentation to set up the mount bind for RHOSP.

Starting Trilio for Openstack 4.0 does Trilio for Openstack allow in-place upgrades.

The following versions can be upgraded to each other:

The upgrade process contains upgrading the Trilio appliance and the Openstack components and is dependent on the underlying operating system.

Enable mount-bind for NFS

In T4O 4.2 and later releases, calculation of the mount point has been changed. It is necessary to set up the mount-bind to make T4O 4.1 or older backups available for T4O 4.2 and T4O 4.2 onwards releases. Please follow this documentation to set up the mount bind for Canonical OpenStack.

Trilio supports the upgrade of Trilio-Openstack components from any of the older releases (4.1 onwards) to the latest 4.2 hotfix releases without ripping up the older deployments.

Refer to the below-mentioned acceptable values for the placeholders kolla_base_distro and triliovault_tag in this document as per the Openstack environment:

1] Pre-requisites

Please ensure the following points are met before starting the upgrade process:

• Either 4.1 or 4.2 GA OR any hotfix patch against 4.1/4.2 should be already deployed

• No Snapshot OR Restore is running

• Global job scheduler should be disabled

• wlm-cron is disabled on the primary Trilio Appliance

• Access to the gemfury repository to fetch new packages

1.1] Deactivating the wlm-cron service

The following sets of commands will disable the wlm-cron service and verify that it has been completly shut-down.

[root@TVM2 ~]# pcs resource disable wlm-cron
[root@TVM2 ~]# systemctl status wlm-cron
● wlm-cron.service - workload's scheduler cron service
   Loaded: loaded (/etc/systemd/system/wlm-cron.service; disabled; vendor preset                                                                                                                                                                                                                                                                                                                                                           : disabled)
   Active: inactive (dead)

Jun 11 08:27:06 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:06 - INFO - 1...t
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 140686268624368 Child 11389 ki...5
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - 1...5
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: Shutting down thread pool
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - S...l
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: Stopping the threads
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - S...s
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: All threads are stopped succes...y
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - A...y
Jun 11 08:27:09 TVM2 systemd[1]: Stopped workload's scheduler cron service.
Hint: Some lines were ellipsized, use -l to show in full.
[root@TVM2 ~]# pcs resource show wlm-cron
 Resource: wlm-cron (class=systemd type=wlm-cron)
  Meta Attrs: target-role=Stopped
  Operations: monitor interval=30s on-fail=restart timeout=300s (wlm-cron-monito                                                                                                                                                                                                                                                                                                                                                           r-interval-30s)
              start interval=0s on-fail=restart timeout=300s (wlm-cron-start-int                                                                                                                                                                                                                                                                                                                                                           erval-0s)
              stop interval=0s timeout=300s (wlm-cron-stop-interval-0s)
[root@TVM2 ~]# ps -ef | grep -i workloadmgr-cron
root     15379 14383  0 08:27 pts/0    00:00:00 grep --color=auto -i workloadmgr                                                                                                                                                                                                                                                                                                                                                           -cron

2] Clone latest configuration scripts

Before the latest configuration script is loaded it is recommended to take a backup of the existing config scripts' folder & Trilio ansible roles. The following command can be used for this purpose:

mv triliovault-cfg-scripts triliovault-cfg-scripts_old
mv /usr/local/share/kolla-ansible/ansible/roles/triliovault /opt/triliovault_old

Clone the latest configuration scripts of the required branch and access the deployment script directory for Kolla Ansible Openstack.

git clone -b 4.3.2 https://github.com/trilioData/triliovault-cfg-scripts.git

cd triliovault-cfg-scripts/kolla-ansible/

Copy the downloaded Trilio ansible role into the Kolla-Ansible roles directory.

cp -R ansible/roles/triliovault /usr/local/share/kolla-ansible/ansible/roles/

3] Append Trilio variables

3.1] Clean old Trilio variables and append new Trilio variables

This is required, in case of some variable names changed, some new variables have been added, or old variables removed in the latest triliovault_globals.yml they need to be updated in /etc/kolla/globals.yml file.

#copy the backed-up original globals.yml which is not having triliovault variables iniside current globals.yml
cp /opt/globals.yml /etc/kolla/globals.yml

#Append Trilio global variables to globals.yml
cat ansible/triliovault_globals.yml >> /etc/kolla/globals.yml

3.2] Clean old Trilio passwords and append new Trilio password variables

This step is required, when some password variable names have been added, changed, or removed in the latest triliovault_passwords.yml. In this case, the /etc/kolla/passwords.yml needs to be updated.

Take backup of current password file
cp /etc/kolla/password.yml /opt/password-<CURRENT-RELEASE>.yml

#Reset the passwords file to default one by reverting the backed-up original password.yml. This backup would have been taken during previous install/upgrade.
cp /opt/password.yml /etc/kolla/password.yml

#Append Trilio password variables to passwords.yml 
cat ansible/triliovault_passwords.yml >> /etc/kolla/passwords.yml

#File /etc/kolla/passwords.yml to be edited to set passwords.
#To set the passwords, it's recommended to use the same passwords as done during previous T4O deployment, as present in the password file backup (/opt/password-<CURRENT-RELEASE>.yml). 
#Any additional passwords (in triliovault_passwords.yml), should be set by the user in /etc/kolla/passwords.yml.

3.3] Append triliovault_site.yml content to kolla ansible's site.yml

This is required because, in case of some variable names changed, some new variables have been added, or old variables removed in the latest triliovault_site.yml they need to be updated in /usr/local/share/kolla-ansible/ansible/site.yml file.

#Take backup of current site.yml file
cp /usr/local/share/kolla-ansible/ansible/site.yml /opt/site-<CURRENT-RELEASE>.yml

#Reset the site.yml to default one by reverting the backed-up original site.yml inside current site.yml. This backup would have been taken during previous install/upgrade.
cp /opt/site.yml /usr/local/share/kolla-ansible/ansible/site.yml

# If the OpenStack release is ‘yoga' append below Trilio code to site.yml  
cat ansible/triliovault_site_yoga.yml >> /usr/local/share/kolla-ansible/ansible/site.yml    

# If the OpenStack release is other than 'yoga' append below Trilio code to site.yml 
cat ansible/triliovault_site.yml >> /usr/local/share/kolla-ansible/ansible/site.yml                               

3.4] Append triliovault_inventory.txt to the kolla-ansible inventory file

By default, the triliovault-datamover-api service gets installed on ‘control' hosts and the trilio-datamover service gets installed on 'compute’ hosts. You can edit the T4O groups in the inventory file as per your cloud architecture.

T4O group names are ‘triliovault-datamover-api’ and ‘triliovault-datamover’

For example:
If your inventory file name path '/root/multinode' then use following
#cleanup old T4O groups from /root/multinode and copy latest triliovault inventory file
cat ansible/triliovault_inventory.txt >> /root/multinode

4] Configure multi-IP NFS as Trilio backup target

On kolla-ansible server node, change directory

cd triliovault-cfg-scripts/common/

Edit file 'triliovault_nfs_map_input.yml' in the current directory and provide compute host and NFS share/ip map.

vi triliovault_nfs_map_input.yml

The triliovault_nfs_map_input.yml is explained here.

Update PyYAML on the kolla-ansible server node only

pip3 install -U pyyaml

Expand the map file to create a one-to-one mapping of compute and NFS share.

python ./generate_nfs_map.py

The result will be in file - 'triliovault_nfs_map_output.yml'

Validate output map file

Open file 'triliovault_nfs_map_output.yml

vi triliovault_nfs_map_output.yml

available in the current directory and validate that all compute nodes are covered with all necessary NFS shares.

Append this output map file to triliovault_globals.yml File Path: /home/stack/triliovault-cfg-scripts/kolla-ansible/ansible/triliovault_globals.yml

cat triliovault_nfs_map_output.yml >> ../kolla-ansible/ansible/triliovault_globals.yml

Ensure to set multi_ip_nfs_enabled in _`_triliovault_globals.yml` file to yes

• A new parameter is added to triliovault_globals.yml , set this parameter value to 'yes' if backup target NFS supports multiple endpoints/IPs. File Path: '/home/stack/triliovault-cfg-scripts/kolla-ansible/ansible/triliovault_globals.yml’ multi_ip_nfs_enabled: 'yes'

• Later append triliovault_globals.yaml file to /etc/kolla/globals.yml

5] Edit globals.yml to set T4O parameters

Edit /etc/kolla/globals.yml file to fill triliovault backup target and build details. You will find the triliovault related parameters at the end of globals.yml. The user needs to fill in details like triliovault build version, backup target type, backup target details, etc.

Following is the list of parameters that the user needs to edit.

6] Enable T4O Snapshot mount feature

To enable Trilio's Snapshot mount feature it is necessary to make the Trilio Backup target available to the nova-compute and nova-libvirt containers.

Edit /usr/local/share/kolla-ansible/ansible/roles/nova-cell/defaults/main.yml and find nova_libvirt_default_volumes variable. Append the Trilio mount bind /var/trilio:/var/trilio:shared to the list of already existing volumes.

For a default Kolla installation, will the variable look as follows afterward:

nova_libvirt_default_volumes:
  - "{{ node_config_directory }}/nova-libvirt/:{{ container_config_directory }}/:ro"
  - "/etc/localtime:/etc/localtime:ro"
  - "{{ '/etc/timezone:/etc/timezone:ro' if ansible_os_family == 'Debian' else '' }}"
  - "/lib/modules:/lib/modules:ro"
  - "/run/:/run/:shared"
  - "/dev:/dev"
  - "/sys/fs/cgroup:/sys/fs/cgroup"
  - "kolla_logs:/var/log/kolla/"
  - "libvirtd:/var/lib/libvirt"
  - "{{ nova_instance_datadir_volume }}:/var/lib/nova/"
  - "
{% if enable_shared_var_lib_nova_mnt | bool %}/var/lib/nova/mnt:/var/lib/nova/mnt:shared{% endif %}




"
  - "nova_libvirt_qemu:/etc/libvirt/qemu"
  - "{{ kolla_dev_repos_directory ~ '/nova/nova:/var/lib/kolla/venv/lib/python' ~ distro_python_version ~ '/site-packages/nova' if nova_dev_mode | bool else '' }
  - "/var/trilio:/var/trilio:shared"

Next, find the variable nova_compute_default_volumes in the same file and append the mount bind /var/trilio:/var/trilio:shared to the list.

After the change will the variable look for a default Kolla installation as follows:

nova_compute_default_volumes:
  - "{{ node_config_directory }}/nova-compute/:{{ container_config_directory }}/:ro"
  - "/etc/localtime:/etc/localtime:ro"
  - "{{ '/etc/timezone:/etc/timezone:ro' if ansible_os_family == 'Debian' else '' }}"
  - "/lib/modules:/lib/modules:ro"
  - "/run:/run:shared"
  - "/dev:/dev"
  - "kolla_logs:/var/log/kolla/"
  - "
{% if enable_iscsid | bool %}iscsi_info:/etc/iscsi{% endif %}"
  - "libvirtd:/var/lib/libvirt"
  - "{{ nova_instance_datadir_volume }}:/var/lib/nova/"
  - "{% if enable_shared_var_lib_nova_mnt | bool %}/var/lib/nova/mnt:/var/lib/nova/mnt:shared{% endif %}


"
  - "{{ kolla_dev_repos_directory ~ '/nova/nova:/var/lib/kolla/venv/lib/python' ~ distro_python_version ~ '/site-packages/nova' if nova_dev_mode | bool else '' }}"
  - "/var/trilio:/var/trilio:shared"

In the case of using Ironic compute nodes one more entry needs to be adjusted in the same file. Find the variable nova_compute_ironic_default_volumes and append trilio mount /var/trilio:/var/trilio:shared to the list.

After the changes the variable will look like the following:

nova_compute_ironic_default_volumes:
  - "{{ node_config_directory }}/nova-compute-ironic/:{{ container_config_directory }}/:ro"
  - "/etc/localtime:/etc/localtime:ro"
  - "{{ '/etc/timezone:/etc/timezone:ro' if ansible_os_family == 'Debian' else '' }}"
  - "kolla_logs:/var/log/kolla/"
  - "{{ kolla_dev_repos_directory ~ '/nova/nova:/var/lib/kolla/venv/lib/python' ~ distro_python_version ~ '/site-packages/nova' if nova_dev_mode | bool else '' }}"
  - "/var/trilio:/var/trilio:shared"

7] Pull containers in case of private repository

In case, the user doesn’t want to use the docker hub registry for triliovault containers during cloud deployment, then the user can pull triliovault images before starting cloud deployment and push them to other preferred registries.

Following are the triliovault container image URLs for 4.3.2 releases. Replace kolla_base_distro and triliovault_tag variables with their values.

This {{ kolla_base_distro }} variable can be either 'centos' or 'ubuntu' depends on your base OpenStack distro This {{ triliovault_tag }} is mentioned at the start of this document.

Below are the Source-based OpenStack deployment images

docker.io/trilio/kolla-{{ kolla_base_distro }}-trilio-datamover:{{ triliovault_tag }}
docker.io/trilio/kolla-{{ kolla_base_distro }}-trilio-datamover-api:{{ triliovault_tag }}
docker.io/trilio/kolla-{{ kolla_base_distro }}-trilio-horizon-plugin:{{ triliovault_tag }}

## EXAMPLE from Kolla Ubuntu source based OpenStack
docker.io/trilio/kolla-ubuntu-trilio-datamover:{{ triliovault_tag }}
docker.io/trilio/kolla-ubuntu-trilio-datamover-api:{{ triliovault_tag }}
docker.io/trilio/kolla-ubuntu-trilio-horizon-plugin:{{ triliovault_tag }}

Below are the Binary-based OpenStack deployment images

docker.io/trilio/kolla-{{ kolla_base_distro }}-trilio-datamover:{{ triliovault_tag }}
docker.io/trilio/kolla-{{ kolla_base_distro }}-trilio-datamover-api:{{ triliovault_tag }}
docker.io/trilio/{{ kolla_base_distro }}-binary-trilio-horizon-plugin:{{ triliovault_tag }}

## EXAMPLE from Kolla Ubuntu binary based OpenStack
docker.io/trilio/kolla-ubuntu-trilio-datamover:{{ triliovault_tag }}
docker.io/trilio/kolla-ubuntu-trilio-datamover-api:{{ triliovault_tag }}
docker.io/trilio/ubuntu-binary-trilio-horizon-plugin:{{ triliovault_tag }}

8] Pull T4O container images

Activate the login into dockerhub for Trilio tagged containers.

Please get the Dockerhub login credentials from Trilio Sales/Support team

ansible -i multinode control -m shell -a "docker login -u <docker-login-username> -p <docker-login-password> docker.io"

Run the below command from the directory with the multinode file tull pull the required images.

kolla-ansible -i multinode pull --tags triliovault

9] Run Kolla-Ansible upgrade command

Run the below command from the directory with the multinode file to start the upgrade process.

kolla-ansible -i multinode upgrade

10] Verify Trilio deployment

Verify on the controller and compute nodes that the Trilio containers are in UP state.

Following is a sample output of commands from controller and compute nodes. triliovault_tag will have value as per the openstack release where deployment being done.

[controller] docker ps  | grep "trilio-"
a2a3593f76db   trilio/kolla-centos-trilio-datamover-api:<triliovault_tag>       "dumb-init --single-…"   23 hours ago    Up 23 hours    triliovault_datamover_api
5f573caa7b02   trilio/kolla-centos-trilio-horizon-plugin:<triliovault_tag>      "dumb-init --single-…"   23 hours ago    Up 23 hours              horizon

[compute] docker ps | grep "trilio-"
f6d443c2942c   trilio/kolla-centos-trilio-datamover:<triliovault_tag>          "dumb-init --single-…"   23 hours ago    Up 23 hours    triliovault_datamover

11] Advance settings/configuration for Trilio services

11.1] Customize HAproxy configuration parameters for Trilio datamover api service

Following are the default haproxy conf parameters set against triliovault datamover api service.

retries 5
timeout http-request 10m
timeout queue 10m
timeout connect 10m
timeout client 10m
timeout server 10m
timeout check 10m
balance roundrobin
maxconn 50000

These values work best for triliovault dmapi service. It’s not recommended to change these parameter values. However, in some exceptional cases, If needed to change any of the above parameter values then same can be done on kolla-ansible server in the following file.

/usr/local/share/kolla-ansible/ansible/roles/triliovault/defaults/main.yml

After editing, run kolla-ansible deploy command again to push these changes to openstack cloud.

Post kolla-ansible deploy, to verify the changes, please check following file, available on all controller/haproxy nodes.

/etc/kolla/haproxy/services.d/triliovault-datamover-api.cfg

12] Enable mound-bind for NFS

In T4O 4.2 and later releases, deriving of the mount point has been changed. It is necessary to set up the mount-bind to make T4O 4.1 or older backups available for T4O 4.2 and onwards releases.

Please follow this documentation to ensure that Backups taken from T4O 4.1 or older can be used with T4O 4.2 and onwards releases.

1. Generic Pre-requisites

1. Please ensure following points before starting the upgrade process:

   1. Either 4.1 GA OR any hotfix patch against 4.1 should be already deployed for performing upgrades mentioned in the current document.

   2. No snapshot OR restore to be running.

   3. Global job scheduler should be disabled.

   4. wlm-cron should be disabled (on primary T4O node).

      1. pcs resource disable wlm-cron

      2. Check : systemctl status wlm-cron OR pcs resource show wlm-cron

      3. Additional step : To ensure that cron is actually stopped, search for any lingering processes against wlm-cron and kill them. [Cmd : ps -ef | grep -i workloadmgr-cron]

2. [On Undercloud node] Clone triliovault repo and upload trilio puppet module

2.1 Clone the latest configuration scripts

2.2 Backup target is “Ceph based S3” with SSL

If the backup target is Ceph S3 with SSL and SSL certificates are self-signed or authorized by private CA, then the user needs to provide a CA chain certificate to validate the SSL requests. For that, user needs to rename his CA chain cert file to 's3-cert.pem' and copy it into directory - 'triliovault-cfg-scripts/redhat-director-scripts/redhat-director-scripts/puppet/trilio/files'.

2.3 Upload triliovault puppet module

3. Prepare Trilio container images

In this step, we are going to pull triliovault container images to the user’s registry.

Trilio containers are pushed to ‘Dockerhub'. The registry URL is ‘docker.io’. Following are the triliovault container pull URLs.

3.1 Trilio container URLs

Trilio container URLs for TripleO Train CentOS7:

There are two registry methods available in the TripleO Openstack Platform.

1. Remote Registry

2. Local Registry

Identify which method you are using. Below we have explained all three methods to pull and configure trilioVault's container images for overcloud deployment.

3.2 Remote Registry

If you are using the 'Remote Registry' method follow this section. You don't need to pull anything. You just need to populate the following container URLs in trilio env yaml.

• Populate 'environments/trilio_env.yaml' file with triliovault container urls. Changes look like the following.

3.3 Registry on Undercloud

If you are using 'local registry' on undercloud, follow this section.

• Run the following script. Script pulls the triliovault containers and updates the triliovault environment file with URLs.

## Above script pushes trilio container images to undercloud registry and sets correct trilio images URLs in ‘environments/trilio_env.yaml’. Verify the changes using the following command.

4. Configure multi-IP NFS

On Undercloud node, change directory

Edit file 'triliovault_nfs_map_input.yml' in the current directory and provide compute host and NFS share/ip map.

Get the compute hostnames from the following command. Check ‘Name' column. Use exact hostnames in 'triliovault_nfs_map_input.yml' file.

vi triliovault_nfs_map_input.yml

Update pyyaml on the undercloud node only

Expand the map file to create a one-to-one mapping of the compute nodes and the nfs shares.

The result will be in file - 'triliovault_nfs_map_output.yml'

Validate output map file

Open file 'triliovault_nfs_map_output.yml

vi triliovault_nfs_map_output.yml

available in the current directory and validate that all compute nodes are covered with all the necessary NFS shares.

Append this output map file to 'triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_nfs_map.yaml'

Validate the changes in file 'triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_nfs_map.yaml'

Include the environment file (trilio_nfs_map.yaml) in overcloud deploy command with '-e' option as shown below.

Ensure that MultiIPNfsEnabled is set to true in trilio_env.yaml file and that nfs is used as backup target.

5. Fill in triliovault environment details

• Refer old 'trilio_env.yaml' - (/home/stack/triliovault-cfg-scripts-old/redhat-director-scripts/tripleo-train/environments/trilio_env.yaml) file and update new trilio_env.yaml file.

• Fill triliovault details in file - '/home/stack/triliovault-cfg-scripts/redhat-director-scripts/tripleo-train/environments/trilio_env.yaml', triliovault environment file is self explanatory. Fill details of backup target, verify image urls and other details.

6. roles_data.yaml file changes

A new separate triliovault service is introduced in Trilio 4.3 release for Trilio Horizon plugin. User need to add following service to roles_data.yaml file and this service need to be co-located with openstack horizon service.

OS::TripleO::Services::TrilioHorizon

7. Install Trilio on Overcloud

Use the following heat environment file and roles data file in overcloud deploy command

1. trilio_env.yaml: This environment file contains Trilio backup target details and Trilio container image locations

2. roles_data.yaml: This file contains overcloud roles data with Trilio roles added. This file need not be changed, you can use the old role_data.yaml file

3. Use the correct trilio endpoint map file as per your keystone endpoint configuration. - Instead of tls-endpoints-public-dns.yaml this file, use ‘environments/trilio_env_tls_endpoints_public_dns.yaml’ - Instead of tls-endpoints-public-ip.yaml this file, use ‘environments/trilio_env_tls_endpoints_public_ip.yaml’ - Instead of tls-everywhere-endpoints-dns.yaml this file, use ‘environments/trilio_env_tls_everywhere_dns.yaml’

Deploy command with triliovault environment file looks like following.

8. Steps to verify correct deployment

8.1 On overcloud controller node(s)

Make sure Trilio dmapi and horizon containers(shown below) are in a running state and no other Trilio container is deployed on controller nodes. If the containers are in restarting state or not listed by the following command then your deployment is not done correctly. You need to revisit the above steps

8.2 On overcloud compute node(s)

Make sure the Trilio datamover container (shown below) is in a running state and no other Trilio container is deployed on compute nodes. If the containers are in restarting state or not listed by the following command then your deployment is not done correctly. You need to revisit the above steps.

8.3 On OpenStack node where OpenStack Horizon Service is running

Make sure the horizon container is in a running state. Please note that the 'Horizon' container is replaced with the Trilio Horizon container. This container will have the latest OpenStack horizon + Trilio's horizon plugin

9. Troubleshooting if any failures

Trilio components will be deployed using puppet scripts.

10. Known Issues/Limitations

10.1 Overcloud deploy fails with the following error. Valid for Train CentOS7 only.

Workaround:

11. Enable mount-bind for NFS

Note : Below mentioned steps required only if target backend is NFS.

Upgrading from Trilio 4.1 to a higher version

Trilio 4.1 can be upgraded without reinstallation to a higher version of T4O if available.

Pre-requisites

Please ensure the following points are met before starting the upgrade process:

• No Snapshot or Restore is running

• The Global-Job-Scheduler is disabled

• wlm-cron is disabled on the Trilio Appliance

• Access to the Gemfury repository to fetch new packages \

  1. Note: For single IP-based NFS share as a backup target refer to this rolling upgrade on the Ansible Openstack document. User needs to follow the Ansible Openstack installation document if having multiple IP-based NFS share

Deactivating the wlm-cron service

The following sets of commands will disable the wlm-cron service and verify that it has been completely shut down.

Update the repositories

Deb-based (Ubuntu)

Add the Gemfury repository on each of the DMAPI containers, Horizon containers & Compute nodes.

Create a file /etc/apt/sources.list.d/fury.list and add the below line to it.

The following commands can be used to verify the connection to the Gemfury repository and to check for available packages.

RPM-based (CentOS)

Add Trilio repo on each of the DMAPI containers, Horizon containers & Compute nodes.

Modify the file /etc/yum.repos.d/trilio.repo and add the below line in it.

The following commands can be used to verify the connection to the Gemfury repository and to check for available packages.

Upgrade tvault-datamover-api package

The following steps represent the best practice procedure to upgrade the DMAPI service.

1. Login to DMAPI container

2. Take a backup of the DMAPI configuration in /etc/dmapi/

3. use apt list --upgradeable to identify the package used for the dmapi service

4. Update the DMAPI package

5. restore the backed-up config files into /etc/dmapi/

6. Restart the DMAPI service

7. Check the status of the DMAPI service

These steps are done with the following commands. This example is assuming that the more common python3 packages are used.

Deb-based (Ubuntu)

RPM-based (CentOS)

Upgrade Horizon plugin

The following steps represent the best practice procedure to update the Horizon plugin.

1. Login to Horizon Container

2. use apt list --upgradeable to identify the package the Trilio packages for the workloadmgrclient, contegoclient and tvault-horizon-plugin

3. Install the tvault-horizon-plugin package in the required python version

4. Install the workloadmgrclient package

5. Install the contegoclient

6. Restart the Horizon webserver

7. Check the installed version of the workloadmgrclient

These steps are done with the following commands. This example is assuming that the more common python3 packages are used.

Deb-based (Ubuntu)

RPM-based (CentOS)

Upgrade the tvault-contego service

The following steps represent the best practice procedure to update the tvault-contego service on the compute nodes.

1. Login into the compute node

2. Take a backup of the config files at and /etc/tvault-contego/ and /etc/tvault-object-store (if S3)

3. Unmount storage mount path

4. Upgrade the tvault-contego package in the required python version

5. (S3 only) upgrade the s3-fuse-plugin package

6. Restore the config files

7. (S3 only) Restart the tvault-object-store service

8. Restart the tvault-contego service

9. Check the status of the service(s)

These steps are done with the following commands. This example is assuming that the more common python3 packages are used.

NFS as Storage Backend

• Take a backup of the config files

• Check the mount path of the NFS storage using the command df -h and unmount the path using umount command. e.g.
• Upgrade the Trilio packages:

Deb-based (Ubuntu):

RPM-based (CentOS):

• Restore the config files, restart the service and verify the mount point

S3 as Storage Backend

• Take a backup of the config files

• Check the mount path of the S3 storage using the command df -h and unmount the path using umount command.

• Upgrade the Trilio packages

Deb-based (Ubuntu):

RPM-based (CentOS):

• Restore the config files, restart the service and verify the mount point

Advance settings/configuration

Customize HAproxy cfg parameters for Trilio datamover api service

Following are the haproxy cfg parameters recommended for optimal performance of dmapi service. File location on controller /etc/haproxy/haproxy.cfg

Parameters timeout client, timeout server, and balance for DMAPI service

Remove the below content, if present in the file/etc/openstack_deploy/user_variables.ymlon the ansible host.

Add the below lines at end of the file /etc/openstack_deploy/user_variables.yml on the ansible host.

Update Haproxy configuration using the below command on the ansible host.

Enable mount-bind for NFS

T4O 4.2 has changed the calculation of the mount point. It is necessary to set up the mount-bind to make T4O 4.1 or older backups available for T4O 4.2