Upgrade Trilio Appliance

The Trilio appliance of T4O 4.2 is running a different Kernel than the Trilio appliance for T4O 4.1 or older.

When upgrading from T4O 4.1 or older it is recommended to replace the Trilio appliance entirely and not do an in-place upgrade. This document provides both online/offline upgrades of Trilio Appliance from any of the older T4O 4.2-based releases to the latest T4O-4.3 release.

Generic Prerequisites

The prerequisites should already be fulfilled from upgrading the Trilio components on the Controller and Compute nodes.

Please ensure to complete the upgrade of all the Trilio components on the Openstack controller & compute nodes before starting the rolling upgrade of Trilio.
The mentioned Gemfury repository should be accessible from Trilio VM.
Either 4.2 GA OR any hotfix patch against 4.2 should be already deployed for performing upgrades mentioned in the current document.
Please ensure the following points before starting the upgrade process:
- No snapshot OR restore to be running.
- Global job-scheduler should be disabled.
- wlm-cron should be disabled and any lingering process should be killed.

Deactivating the wlm-cron service

The following sets of commands will disable the wlm-cron service and verify that it is has been completly shut-down.

[root@TVM2 ~]# pcs resource disable wlm-cron
[root@TVM2 ~]# systemctl status wlm-cron
● wlm-cron.service - workload's scheduler cron service
   Loaded: loaded (/etc/systemd/system/wlm-cron.service; disabled; vendor preset                                                                                                                                                                                                                                                                                                                                                           : disabled)
   Active: inactive (dead)

Jun 11 08:27:06 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:06 - INFO - 1...t
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 140686268624368 Child 11389 ki...5
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - 1...5
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: Shutting down thread pool
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - S...l
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: Stopping the threads
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - S...s
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: All threads are stopped succes...y
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - A...y
Jun 11 08:27:09 TVM2 systemd[1]: Stopped workload's scheduler cron service.
Hint: Some lines were ellipsized, use -l to show in full.
[root@TVM2 ~]# pcs resource show wlm-cron
 Resource: wlm-cron (class=systemd type=wlm-cron)
  Meta Attrs: target-role=Stopped
  Operations: monitor interval=30s on-fail=restart timeout=300s (wlm-cron-monito                                                                                                                                                                                                                                                                                                                                                           r-interval-30s)
              start interval=0s on-fail=restart timeout=300s (wlm-cron-start-int                                                                                                                                                                                                                                                                                                                                                           erval-0s)
              stop interval=0s timeout=300s (wlm-cron-stop-interval-0s)
[root@TVM2 ~]# ps -ef | grep -i workloadmgr-cron
root     15379 14383  0 08:27 pts/0    00:00:00 grep --color=auto -i workloadmgr                                                                                                                                                                                                                                                                                                                                                           -cron

Steps to upgrade Trilio having Internet access

Download the hf_upgrade.sh script on all T4O nodes where the upgrade is to be done.

cd /opt/ 
wget https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/4.3.2/TVOAppliance/hf_upgrade.sh
chmod +x hf_upgrade.sh

You can check the usage of this script here.

Run the following command to download and Install the upgraded packages

./hf_upgrade.sh --all  
OR 
./hf_upgrade.sh -a

Run the following command to enable the wlm-cron service

pcs resource enable wlm-cron

Steps to upgrade Trilio having No Internet access

Here the packages need to be downloaded on a separate host which has internet access. The script ./hf_upgrade.sh can be used with the below-mentioned option to download the required package

Download the hf_upgrade.sh script and use the script to download the upgraded packages. You can check the usage of this script here.\

cd /<tvo_packages_download_path>/ 
wget https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/4.3.2/TVOAppliance/hf_upgrade.sh
chmod +x hf_upgrade.sh

# Download the upgraded packages
./hf_upgrade.sh --downloadonly

Copy the downloaded 4.3-offlinePkgs.tar.gz package and hf_upgrade.sh script to all the Trilio nodes\
```
scp <tvo_packages_download_path>/* root@<TrilioVault_node_IP>:/<path_to_upgrade_package>/
```
Now access the appliance VMs and install the copied offline packages on each of the Trilio appliances by running the below-mentioned command:\
```
cd <path_to_upgrade_package>/
./hf_upgrade.sh --installonly
```
Once the installation is done, run the following command to enable the wlm-cron service\
```
pcs resource enable wlm-cron
```

Post upgrade steps

Verify all wlm services on all Trilio nodes

systemctl status tvault-config wlm-workloads wlm-api wlm-scheduler
pcs status (on primary node)
systemctl status wlm-cron (on primary node)
systemctl status tvault-object-store (only if Trilio configured with S3 backend storage)

Make sure all wlm services are up and running from python version 3.8.x
Check the mount point using “df -h” command
Grafana dashboard shows the correct wlm service status on all T4O nodes.
Enable Global Job Scheduler

Additional check for wlm-cron on the primary node

ps -ef | grep workloadmgr-cron | grep -v grep
# Above command should show only 2 processes running; sample below

[root@tvm6 ~]# ps -ef | grep workloadmgr-cron | grep -v grep
nova      8841     1  2 Jul28 ?        00:40:44 /home/stack/myansible/bin/python3 /home/stack/myansible/bin/workloadmgr-cron --config-file=/etc/workloadmgr/workloadmgr.conf
nova      8898  8841  0 Jul28 ?        00:07:03 /home/stack/myansible/bin/python3 /home/stack/myansible/bin/workloadmgr-cron --config-file=/etc/workloadmgr/workloadmgr.conf

[RHOSP, TripleO and Kolla only] Verify nova UID/GID for nova user on the Appliance

Red Hat OpenStack, TripleO, and Kolla Ansible Openstack are using the nova UID/GID of 42436 inside their containers instead of 162:162 which is the standard in other Openstack environments.

Please verify that the nova UID/GID on the Trilio Appliance is still 42436, to do so follow the below document provided here:

Update nova UID/GID on the appliance

PreviousUpgrading on TripleO Train [CentOS7]NextUpgrade OpenStack

Last updated 7 months ago