1 of 1

Upgrade Trilio Appliance

The Trilio appliance of T4O 4.2 is running a different Kernel than the Trilio appliance for T4O 4.1 or older.

When upgrading from T4O 4.1 or older it is recommended to replace the Trilio appliance entirely and not do an in-place upgrade. This document provides both online/offline upgrades of Trilio Appliance from any of the older TVO-4.2-based releases to the latest TVO-4.2 release.

Generic Prerequisites

The prerequisites should already be fulfilled from upgrading the Trilio components on the Controller and Compute nodes.

Please ensure to complete the upgrade of all the Trilio components on the Openstack controller & compute nodes before starting the rolling upgrade of Trilio.
The mentioned Gemfury repository should be accessible from Trilio VM.
Either 4.2 GA OR any hotfix patch against 4.2 should be already deployed for performing upgrades mentioned in the current document.
Please ensure the following points before starting the upgrade process:
- No snapshot OR restore to be running.
- Global job-scheduler should be disabled.
- wlm-cron should be disabled and any lingering process should be killed.

Deactivating the wlm-cron service

The following sets of commands will disable the wlm-cron service and verify that it is has been completly shut-down.

[root@TVM2 ~]# pcs resource disable wlm-cron
[root@TVM2 ~]# systemctl status wlm-cron
● wlm-cron.service - workload's scheduler cron service
   Loaded: loaded (/etc/systemd/system/wlm-cron.service; disabled; vendor preset                                                                                                                                                                                                                                                                                                                                                           : disabled)
   Active: inactive (dead)

Jun 11 08:27:06 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:06 - INFO - 1...t
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 140686268624368 Child 11389 ki...5
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - 1...5
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: Shutting down thread pool
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - S...l
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: Stopping the threads
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - S...s
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: All threads are stopped succes...y
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - A...y
Jun 11 08:27:09 TVM2 systemd[1]: Stopped workload's scheduler cron service.
Hint: Some lines were ellipsized, use -l to show in full.
[root@TVM2 ~]# pcs resource show wlm-cron
 Resource: wlm-cron (class=systemd type=wlm-cron)
  Meta Attrs: target-role=Stopped
  Operations: monitor interval=30s on-fail=restart timeout=300s (wlm-cron-monito                                                                                                                                                                                                                                                                                                                                                           r-interval-30s)
              start interval=0s on-fail=restart timeout=300s (wlm-cron-start-int                                                                                                                                                                                                                                                                                                                                                           erval-0s)
              stop interval=0s timeout=300s (wlm-cron-stop-interval-0s)
[root@TVM2 ~]# ps -ef | grep -i workloadmgr-cron
root     15379 14383  0 08:27 pts/0    00:00:00 grep --color=auto -i workloadmgr                                                                                                                                                                                                                                                                                                                                                           -cron

Steps to upgrade Trilio having Internet access

1. Download the `hf_upgrade.sh` script on all T4O nodes where the upgrade is to be done.

You can check the usage of this script here.

Either perform the Step 1.1 Upgrade to the latest Hotfix/Maintenance release or 1.2 Upgrade to the intermediate Hotfix/Maintenance release depending on your requirement.

1.1 Upgrade to the latest Hotfix/Maintenance release

cd /opt/ 
wget https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/TVO/4.2.8/TVOAppliance/hf_upgrade.sh
chmod +x hf_upgrade.sh

1.2 Upgrade to the intermediate Hotfix/Maintenance release

a. Download the hf_upgrade.sh of desired Hotfix/Maintenance release

You have to replace the Gitbranch in below command with the actual branch name of the Hotfix/Maintenance release you wish to upgrade.

You will get that git branch name from Release Notes page of that particular Hotfix/Maintenance release

cd /opt/ 
wget https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/<Gitbranch>/TVOAppliance/hf_upgrade.sh
chmod +x hf_upgrade.sh

b. Edit downloaded hf_upgrade.sh and set the BRANCH_NAME as triliodata-hotfix-4-2 for any Hotfix/Maintenance release and OFFLINE_PKG_NAME must be set to particular hotfix package name

You would find the offline package name for particular release at http://repos.trilio.io:8283/triliodata-hotfix-4-2/offlinePkgs/

For example, if you wish to upgarde to 4.2.7 release then BRANCH_NAME would be triliodata-hotfix-4-2 and OFFLINE_PKG_NAME must be set to 4.2.7-offlinePkgs.tar.gz in the script.

2. Run the following command to download and Install the upgraded packages

./hf_upgrade.sh --all  
OR 
./hf_upgrade.sh -a

3. Run the following command to enable the wlm-cron service

pcs resource enable wlm-cron

Steps to upgrade Trilio having No Internet access

Here the packages need to be downloaded on a separate host which has internet access. The script ./hf_upgrade.sh can be used with the below-mentioned option to download the required package

1. Download the `hf_upgrade.sh` script on a separate host which has internet access

Use this script to download the upgraded packages.

You can check the usage of this script here.

Either perform the Step 1.1 Upgrade to the latest Hotfix/Maintenance release or 1.2 Upgrade to the intermediate Hotfix/Maintenance release depending on your requirement.

1.1 Upgrade to the latest Hotfix/Maintenance release

cd /<tvo_packages_download_path>/ 
wget https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/TVO/4.2.8/TVOAppliance/hf_upgrade.sh

# Download the upgraded packages
./hf_upgrade.sh --downloadonly

1.2 Upgrade to the intermediate Hotfix/Maintenance release.

a. Download the hf_upgrade.sh of desired Hotfix/Maintenance release

You have to replace the Gitbranch in below command with the actual branch name of the Hotfix/Maintenance release you wish to upgrade.

You will get that git branch name from Release Notes page of that particular Hotfix/Maintenance release

cd /opt/
wget https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/<Gitbranch>/TVOAppliance/hf_upgrade.sh
chmod +x hf_upgrade.sh

b. Edit downloaded hf_upgrade.sh and set the BRANCH_NAME as triliodata-hotfix-4-2 for any Hotfix/Maintenance release and OFFLINE_PKG_NAME must be set to particular hotfix package name and then run the script.

You would find the offline package name for particular release at http://repos.trilio.io:8283/triliodata-hotfix-4-2/offlinePkgs/

For example, if you wish to upgarde to 4.2.7 release then BRANCH_NAME would be triliodata-hotfix-4-2 and OFFLINE_PKG_NAME must be set to 4.2.7-offlinePkgs.tar.gz in the script.

# Download the upgraded packages
./hf_upgrade.sh --downloadonly

2. Copy the downloaded `4.2-offlinePkgs.tar.gz` package and `hf_upgrade.sh` script to all the Trilio nodes

```
scp <tvo_packages_download_path>/* root@<TrilioVault_node_IP>:/<path_to_upgrade_package>/
```

3. Now access the appliance VMs and install the copied offline packages on each of the Trilio appliances by running the below-mentioned command:

```
cd <path_to_upgrade_package>/
./hf_upgrade.sh --installonly

```

4. Once the installation is done, run the following command to enable the wlm-cron service

```
pcs resource enable wlm-cron
```

Post upgrade steps

Verify all wlm services on all Trilio nodes

systemctl status tvault-config wlm-workloads wlm-api wlm-scheduler
pcs status (on primary node)
systemctl status wlm-cron (on primary node)
systemctl status tvault-object-store (only if Trilio configured with S3 backend storage)

Make sure all wlm services are up and running from python version 3.8.x
Check the mount point using “df -h” command
Grafana dashboard shows the correct wlm service status on all T4O nodes.
Enable Global Job Scheduler

Additional check for wlm-cron on the primary node

ps -ef | grep workloadmgr-cron | grep -v grep
# Above command should show only 2 processes running; sample below

[root@tvm6 ~]# ps -ef | grep workloadmgr-cron | grep -v grep
nova      8841     1  2 Jul28 ?        00:40:44 /home/stack/myansible/bin/python3 /home/stack/myansible/bin/workloadmgr-cron --config-file=/etc/workloadmgr/workloadmgr.conf
nova      8898  8841  0 Jul28 ?        00:07:03 /home/stack/myansible/bin/python3 /home/stack/myansible/bin/workloadmgr-cron --config-file=/etc/workloadmgr/workloadmgr.conf

[RHOSP, TripleO and Kolla only] Verify nova UID/GID for nova user on the Appliance

Red Hat OpenStack, TripleO, and Kolla Ansible Openstack are using the nova UID/GID of 42436 inside their containers instead of 162:162 which is the standard in other Openstack environments.

Please verify that the nova UID/GID on the Trilio Appliance is still 42436, to do so follow the below document provided here:

Update nova UID/GID on the appliance

Upgrade Trilio Appliance

The Trilio appliance of T4O 4.2 is running a different Kernel than the Trilio appliance for T4O 4.1 or older.

Generic Prerequisites

The prerequisites should already be fulfilled from upgrading the Trilio components on the Controller and Compute nodes.

Please ensure to complete the upgrade of all the Trilio components on the Openstack controller & compute nodes before starting the rolling upgrade of Trilio.
The mentioned Gemfury repository should be accessible from Trilio VM.
Either 4.2 GA OR any hotfix patch against 4.2 should be already deployed for performing upgrades mentioned in the current document.
Please ensure the following points before starting the upgrade process:
- No snapshot OR restore to be running.
- Global job-scheduler should be disabled.
- wlm-cron should be disabled and any lingering process should be killed.

Deactivating the wlm-cron service

The following sets of commands will disable the wlm-cron service and verify that it is has been completly shut-down.

[root@TVM2 ~]# pcs resource disable wlm-cron
[root@TVM2 ~]# systemctl status wlm-cron
● wlm-cron.service - workload's scheduler cron service
   Loaded: loaded (/etc/systemd/system/wlm-cron.service; disabled; vendor preset                                                                                                                                                                                                                                                                                                                                                           : disabled)
   Active: inactive (dead)

Jun 11 08:27:06 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:06 - INFO - 1...t
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 140686268624368 Child 11389 ki...5
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - 1...5
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: Shutting down thread pool
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - S...l
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: Stopping the threads
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - S...s
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: All threads are stopped succes...y
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - A...y
Jun 11 08:27:09 TVM2 systemd[1]: Stopped workload's scheduler cron service.
Hint: Some lines were ellipsized, use -l to show in full.
[root@TVM2 ~]# pcs resource show wlm-cron
 Resource: wlm-cron (class=systemd type=wlm-cron)
  Meta Attrs: target-role=Stopped
  Operations: monitor interval=30s on-fail=restart timeout=300s (wlm-cron-monito                                                                                                                                                                                                                                                                                                                                                           r-interval-30s)
              start interval=0s on-fail=restart timeout=300s (wlm-cron-start-int                                                                                                                                                                                                                                                                                                                                                           erval-0s)
              stop interval=0s timeout=300s (wlm-cron-stop-interval-0s)
[root@TVM2 ~]# ps -ef | grep -i workloadmgr-cron
root     15379 14383  0 08:27 pts/0    00:00:00 grep --color=auto -i workloadmgr                                                                                                                                                                                                                                                                                                                                                           -cron

Steps to upgrade Trilio having Internet access

1. Download the `hf_upgrade.sh` script on all T4O nodes where the upgrade is to be done.

You can check the usage of this script here.

Either perform the Step 1.1 Upgrade to the latest Hotfix/Maintenance release or 1.2 Upgrade to the intermediate Hotfix/Maintenance release depending on your requirement.

1.1 Upgrade to the latest Hotfix/Maintenance release

cd /opt/ 
wget https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/TVO/4.2.8/TVOAppliance/hf_upgrade.sh
chmod +x hf_upgrade.sh

1.2 Upgrade to the intermediate Hotfix/Maintenance release

a. Download the hf_upgrade.sh of desired Hotfix/Maintenance release

You have to replace the Gitbranch in below command with the actual branch name of the Hotfix/Maintenance release you wish to upgrade.

You will get that git branch name from Release Notes page of that particular Hotfix/Maintenance release

cd /opt/ 
wget https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/<Gitbranch>/TVOAppliance/hf_upgrade.sh
chmod +x hf_upgrade.sh

b. Edit downloaded hf_upgrade.sh and set the BRANCH_NAME as triliodata-hotfix-4-2 for any Hotfix/Maintenance release and OFFLINE_PKG_NAME must be set to particular hotfix package name

You would find the offline package name for particular release at http://repos.trilio.io:8283/triliodata-hotfix-4-2/offlinePkgs/

For example, if you wish to upgarde to 4.2.7 release then BRANCH_NAME would be triliodata-hotfix-4-2 and OFFLINE_PKG_NAME must be set to 4.2.7-offlinePkgs.tar.gz in the script.

2. Run the following command to download and Install the upgraded packages

./hf_upgrade.sh --all  
OR 
./hf_upgrade.sh -a

3. Run the following command to enable the wlm-cron service

pcs resource enable wlm-cron

Steps to upgrade Trilio having No Internet access

Here the packages need to be downloaded on a separate host which has internet access. The script ./hf_upgrade.sh can be used with the below-mentioned option to download the required package

1. Download the `hf_upgrade.sh` script on a separate host which has internet access

Use this script to download the upgraded packages.

You can check the usage of this script here.

Either perform the Step 1.1 Upgrade to the latest Hotfix/Maintenance release or 1.2 Upgrade to the intermediate Hotfix/Maintenance release depending on your requirement.

1.1 Upgrade to the latest Hotfix/Maintenance release

cd /<tvo_packages_download_path>/ 
wget https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/TVO/4.2.8/TVOAppliance/hf_upgrade.sh

# Download the upgraded packages
./hf_upgrade.sh --downloadonly

1.2 Upgrade to the intermediate Hotfix/Maintenance release.

a. Download the hf_upgrade.sh of desired Hotfix/Maintenance release

You have to replace the Gitbranch in below command with the actual branch name of the Hotfix/Maintenance release you wish to upgrade.

You will get that git branch name from Release Notes page of that particular Hotfix/Maintenance release

cd /opt/
wget https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/<Gitbranch>/TVOAppliance/hf_upgrade.sh
chmod +x hf_upgrade.sh

You would find the offline package name for particular release at http://repos.trilio.io:8283/triliodata-hotfix-4-2/offlinePkgs/

For example, if you wish to upgarde to 4.2.7 release then BRANCH_NAME would be triliodata-hotfix-4-2 and OFFLINE_PKG_NAME must be set to 4.2.7-offlinePkgs.tar.gz in the script.

# Download the upgraded packages
./hf_upgrade.sh --downloadonly

2. Copy the downloaded `4.2-offlinePkgs.tar.gz` package and `hf_upgrade.sh` script to all the Trilio nodes

```
scp <tvo_packages_download_path>/* root@<TrilioVault_node_IP>:/<path_to_upgrade_package>/
```

3. Now access the appliance VMs and install the copied offline packages on each of the Trilio appliances by running the below-mentioned command:

```
cd <path_to_upgrade_package>/
./hf_upgrade.sh --installonly

```

4. Once the installation is done, run the following command to enable the wlm-cron service

```
pcs resource enable wlm-cron
```

Post upgrade steps

Verify all wlm services on all Trilio nodes

systemctl status tvault-config wlm-workloads wlm-api wlm-scheduler
pcs status (on primary node)
systemctl status wlm-cron (on primary node)
systemctl status tvault-object-store (only if Trilio configured with S3 backend storage)

Make sure all wlm services are up and running from python version 3.8.x
Check the mount point using “df -h” command
Grafana dashboard shows the correct wlm service status on all T4O nodes.
Enable Global Job Scheduler

Additional check for wlm-cron on the primary node

ps -ef | grep workloadmgr-cron | grep -v grep
# Above command should show only 2 processes running; sample below

[root@tvm6 ~]# ps -ef | grep workloadmgr-cron | grep -v grep
nova      8841     1  2 Jul28 ?        00:40:44 /home/stack/myansible/bin/python3 /home/stack/myansible/bin/workloadmgr-cron --config-file=/etc/workloadmgr/workloadmgr.conf
nova      8898  8841  0 Jul28 ?        00:07:03 /home/stack/myansible/bin/python3 /home/stack/myansible/bin/workloadmgr-cron --config-file=/etc/workloadmgr/workloadmgr.conf

[RHOSP, TripleO and Kolla only] Verify nova UID/GID for nova user on the Appliance

Red Hat OpenStack, TripleO, and Kolla Ansible Openstack are using the nova UID/GID of 42436 inside their containers instead of 162:162 which is the standard in other Openstack environments.

Please verify that the nova UID/GID on the Trilio Appliance is still 42436, to do so follow the below document provided here:

Update nova UID/GID on the appliance

Upgrade Trilio Appliance

Generic Prerequisites

Deactivating the wlm-cron service

Steps to upgrade Trilio having Internet access

1. Download the hf_upgrade.sh script on all T4O nodes where the upgrade is to be done.

1.1 Upgrade to the latest Hotfix/Maintenance release

1.2 Upgrade to the intermediate Hotfix/Maintenance release

2. Run the following command to download and Install the upgraded packages

3. Run the following command to enable the wlm-cron service

Steps to upgrade Trilio having No Internet access

1. Download the hf_upgrade.sh script on a separate host which has internet access

1.1 Upgrade to the latest Hotfix/Maintenance release

1.2 Upgrade to the intermediate Hotfix/Maintenance release.

2. Copy the downloaded 4.2-offlinePkgs.tar.gz package and hf_upgrade.sh script to all the Trilio nodes

3. Now access the appliance VMs and install the copied offline packages on each of the Trilio appliances by running the below-mentioned command:

4. Once the installation is done, run the following command to enable the wlm-cron service

Post upgrade steps

[RHOSP, TripleO and Kolla only] Verify nova UID/GID for nova user on the Appliance

Upgrade Trilio Appliance

Generic Prerequisites

Deactivating the wlm-cron service

Steps to upgrade Trilio having Internet access

1. Download the hf_upgrade.sh script on all T4O nodes where the upgrade is to be done.

1.1 Upgrade to the latest Hotfix/Maintenance release

1.2 Upgrade to the intermediate Hotfix/Maintenance release

2. Run the following command to download and Install the upgraded packages

3. Run the following command to enable the wlm-cron service

Steps to upgrade Trilio having No Internet access

1. Download the hf_upgrade.sh script on a separate host which has internet access

1.1 Upgrade to the latest Hotfix/Maintenance release

1.2 Upgrade to the intermediate Hotfix/Maintenance release.

2. Copy the downloaded 4.2-offlinePkgs.tar.gz package and hf_upgrade.sh script to all the Trilio nodes

3. Now access the appliance VMs and install the copied offline packages on each of the Trilio appliances by running the below-mentioned command:

4. Once the installation is done, run the following command to enable the wlm-cron service

Post upgrade steps

[RHOSP, TripleO and Kolla only] Verify nova UID/GID for nova user on the Appliance

1. Download the `hf_upgrade.sh` script on all T4O nodes where the upgrade is to be done.

1. Download the `hf_upgrade.sh` script on a separate host which has internet access

2. Copy the downloaded `4.2-offlinePkgs.tar.gz` package and `hf_upgrade.sh` script to all the Trilio nodes

1. Download the `hf_upgrade.sh` script on all T4O nodes where the upgrade is to be done.

1. Download the `hf_upgrade.sh` script on a separate host which has internet access

2. Copy the downloaded `4.2-offlinePkgs.tar.gz` package and `hf_upgrade.sh` script to all the Trilio nodes