Upgrading on Canonical OpenStack

Learn about upgrading Trilio on Canonical OpenStack

Trilio supports the upgrade of charms and Trilio components from older releases (4.0 onwards) to the latest release. More information about the latest 4.2 Trilio charms for the Trilio-4.2 release can be found here.

The following charms exist:

• trilio-wlm ➡️ Installs and manages Trilio Controller services.

• trilio-dm-api ➡️Installs and manages the Trilio Datamover API service.

• trilio-data-mover ➡️ Installs and manages the Trilio Datamover service.

• trilio-horizon-plugin ➡️ Installs and manages the Trilio Horizon Plugin.

The documentation of the charms can be found here:

TrilioVault data protection — charm-guide 0.0.1.dev819 documentationdocs.openstack.org

The following steps have been tested and verified within Trilio environments. There have been cases where these steps updated all packages inside the LXC containers, leading to failures in basic OpenStack services.

It is recommended to run each of these steps in dry-run first.

When any other packages but Trilio packages are getting updated, stop the upgrade procedure and contact your Trilio customer success manager.

Upgrading the Trilio Charms

More detailed information about the latest 4.2 Trilio charms for the Trilio-4.2 release can be found here.

Upgrading the Trilio Charms up to OpenStack Wallaby release

Follow the steps mentioned in this document to upgrade the charms to the latest 4.2 charms before upgrading the Trilio packages.

Upgrading the Trilio Charms to OpenStack Yoga release onwards

Follow the steps mentioned below to upgrade the charms to the latest 4.2 charms before upgrading the Trilio packages.

General Prerequisites

1. No snapshot OR restore are to be running during this process.

2. Global job scheduler should be disabled

1. Upgrade juju charms

juju [-m <model>] upgrade-charm trilio-wlm --switch trilio-charmers-trilio-wlm-focal --channel latest/edge

juju [-m <model>] upgrade-charm trilio-horizon-plugin --switch trilio-charmers-trilio-horizon-plugin-focal --channel latest/edge

juju [-m <model>] upgrade-charm trilio-dm-api --switch trilio-charmers-trilio-dm-api-focal --channel latest/edge

juju [-m <model>] upgrade-charm trilio-data-mover --switch trilio-charmers-trilio-data-mover-focal --channel latest/edge

2. Wait till all trilio units are active/idle then Restart all the trilio services

juju [-m <model>] exec --application trilio-dm-api "sudo systemctl restart tvault-datamover-api"

juju [-m <model>] exec --application trilio-data-mover "sudo systemctl restart tvault-contego"

juju [-m <model>] exec --application trilio-horizon-plugin "sudo systemctl restart apache2"

3. Select Single Node or HA steps as appropriate to restart services

For setup with single node wlm unit

juju [-m <model>] exec --application trilio-wlm "sudo systemctl restart wlm-api wlm-scheduler wlm-workloads wlm-cron"

For setups with 3 node HA enabled wlm

juju [-m <model>] exec --application trilio-wlm "systemctl restart wlm-api wlm-scheduler wlm-workloads"

juju [-m <model>] exec --unit trilio-wlm/leader "sudo crm resource restart res_trilio_wlm_wlm_cron"

4. Ensure that wlm-cron is only running on a single Unit in the output of this command

juju [-m <model>] exec --application trilio-wlm "sudo systemctl status wlm-cron"

If wlm-cron is running in more than one unit, or nowhere at all, this can be fixed by following the below steps:

juju [-m <model>] exec --unit trilio-wlm/leader "sudo crm resource stop res_trilio_wlm_wlm_cron" juju [-m <model>] exec --application trilio-wlm "sudo systemctl stop wlm-cron" juju [-m <model>] exec --application trilio-wlm "sudo ps -ef | grep workloadmgr-cron | grep -v grep"

No workloadmgr-cron process should be running. If it's still running somewhere, login to that unit and stop the service manually with 'sudo systemctl stop wlm-cron'

juju [-m <model>] exec --unit trilio-wlm/leader "sudo crm resource start res_trilio_wlm_wlm_cron" juju [-m <model>] exec --application trilio-wlm "sudo systemctl status wlm-cron"

wlm-cron service should only be running in a single Juju Unit and not running in the other two.

5. Check upgraded juju charm & trilio services

Check the trilio unit's status in juju status [-m <model>] | grep trilio output.

All the trilio units will be reporting the new juju charm version.

trilio-data-mover 4.2.64.26 active 3 trilio-charmers-trilio-data-mover-jammy latest/edge 4 no Unit is ready trilio-dm-api 4.2.64.2 active 1 trilio-charmers-trilio-dm-api-jammy latest/edge 2 no Unit is ready trilio-horizon-plugin 4.2.64.5 active 1 trilio-charmers-trilio-horizon-plugin-jammy latest/edge 4 no Unit is ready trilio-wlm 4.2.64.20 active 1 trilio-charmers-trilio-wlm-jammy latest/edge 3 no Unit is ready

6. Check that all Trilio services are running

juju [-m <model>] exec --application trilio-dm-api "sudo systemctl status tvault-datamover-api"

juju [-m <model>] exec --application trilio-data-mover "sudo systemctl status tvault-contego"

juju [-m <model>] exec --application trilio-horizon-plugin "sudo systemctl status apache2"

juju [-m <model>] exec --application trilio-wlm "sudo systemctl status wlm-api wlm-scheduler wlm-workloads wlm-cron"

Next move on the Updating the Trilio packages.

Upgrading the Trilio packages

Trilio is releasing hotfixes, which require updating the packages inside the containers. These hotfixes can not be installed using the Juju charms as they don't require an update to the charms.

Generic Prerequisites

1. Trilio packages can be upgraded after deployment. Trilio supports upgrade to the latest 4.2 releases from as old as the Trilio 4.0 release.

2. No snapshot OR restore are to be running during this process.

3. Global job scheduler should be disabled.

4. wlm-cron should be disabled ( Following sets of commands are to be run on MAAS node).

5. If trilio-wlm is HA enabled, set the cluster configuration to maintenance mode ( this command will fail for single node deployment).

juju exec [-m <model>] --unit trilio-wlm/leader "sudo crm configure property maintenance-mode=true"

1. Stop the wlm-cron service

juju exec [-m <model>] --application trilio-wlm "sudo systemctl stop wlm-cron"

1. Ensure that no stale wlm-cron processes exist

juju exec [-m <model>] --application trilio-wlm "sudo ps -ef | grep [w]orkloadmgr-cron"

If any stale process are found, they should be stopped manually.

Upgrade package on trilio units

The deployed Trilio version is controlled by the triliovault-pkg-source charm configuration option.

The gemfury repository should be accessible from all Trilio units. For each trilio charm, it should be pointing to the following Gemfury repository:

deb [trusted=yes] https://apt.fury.io/trilio-4-3/ /

This can be checked via juju [-m ] config trilio-wlm triliovault-pkg-source command output.

The preferred, recommended, and tested method to update the packages is through the Juju command line.

Run the below commands from MAAS node

On Ubuntu Bionic environments

juju exec [-m <model>] --application trilio-wlm 'sudo apt-get update && sudo apt-get install -y --only-upgrade -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" workloadmgr python3-workloadmgrclient python3-contegoclient s3-fuse-plugin'
juju exec [-m <model>] --application trilio-horizon-plugin 'sudo apt-get update && sudo apt-get install -y --only-upgrade -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" tvault-horizon-plugin python-workloadmgrclient'
juju exec [-m <model>] --application trilio-dm-api 'sudo apt-get update && sudo apt-get install -y --only-upgrade -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" python3-dmapi'
juju exec [-m <model>] --application trilio-data-mover 'sudo apt-get update && sudo apt-get install -y --only-upgrade -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" tvault-contego s3-fuse-plugin'

On other (not Ubuntu Bionic) environments

juju exec [-m <model>] --application trilio-wlm 'sudo apt-get update && sudo apt-get install -y --only-upgrade -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" workloadmgr python3-workloadmgrclient python3-contegoclient python3-s3-fuse-plugin'
juju exec [-m <model>] --application trilio-horizon-plugin 'sudo apt-get update && sudo apt-get install -y --only-upgrade -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" python3-tvault-horizon-plugin python3-workloadmgrclient'
juju exec [-m <model>] --application trilio-dm-api 'sudo apt-get update && sudo apt-get install -y --only-upgrade -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" python3-dmapi'
juju exec [-m <model>] --application trilio-data-mover 'sudo apt-get update && sudo apt-get install -y --only-upgrade -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" python3-tvault-contego python3-s3-fuse-plugin'

Check the trilio unit's status in juju status [-m ] | grep trilio output. All the trilio units will be with the new packages.

trilio-data-mover      <package version>  active      3  trilio-data-mover      jujucharms    8  ubuntu
trilio-dm-api          <package version>  active      1  trilio-dm-api          jujucharms    5  ubuntu
trilio-horizon-plugin  <package version>  active      1  trilio-horizon-plugin  jujucharms    4  ubuntu
trilio-wlm             <package version>  active      3  trilio-wlm             jujucharms    7  ubuntu

Update the DB schema

Run the below command to update the schema

juju exec [-m <model>] --unit trilio-wlm/leader "alembic -c /etc/workloadmgr/alembic.ini upgrade heads"

Check the schema head with below command. It should point to latest schema head.

juju exec [-m <model>] --unit trilio-wlm/leader "alembic -c /etc/workloadmgr/alembic.ini current"

Enable mount-bind for NFS

T4O 4.2 is changing how the mount point is calculated. It is required to setup a mount bind to make T4O 4.1 or older backups available.

Please refer to this page for detailed steps to set up the mount bind.

Post-Upgrade steps

1. If the trilio-wlm nodes are HA enabled:

   1. Make sure the wlm-cron services are down after the pkg upgrade. Run the following command for the same:

juju exec [-m <model>] --application trilio-wlm "sudo systemctl stop wlm-cron"

1. Unset the cluster maintenance mode

juju exec [-m <model>] --unit trilio-wlm/leader "sudo crm configure property maintenance-mode=false"

1. Make sure the wlm-cron service up and running on any one node.

juju exec [-m <model>] --application trilio-wlm "sudo systemctl status wlm-cron"

1. Set the Global Job Scheduler to the original state.

Troubleshooting

If any trilio unit gets into an error state with the message :

hook failed: "update-status"

One of the reasons could be the package installation did not finish correctly. One way to verify that is by logging into that unit and following the below steps;

Login to trilio unit and run "sudo dpkg --configure -a"
It will ask for user input, hit enter and log out from the unit.
From mass node run command "juju resolve <trilio unit name>"