Upgrading on Canonical Openstack

Upgrading Trilio 4.0 to 4.1 using JuJu Charms

For the major upgrade from 4.0 to 4.1 use the JuJu charms upgrade path.

The charms will always install the latest version available of T4O 4.1. This will only work when upgrading from 4.0 to 4.1.

The following charms exist:

• trilio-wlm ➡️ Installs and manages Trilio Controller services.

• trilio-dm-api ➡️Installs and manages the Trilio Datamover API service.

• trilio-data-mover ➡️ Installs and manages the Trilio Datamover service.

• trilio-horizon-plugin ➡️ Installs and manages the Trilio Horizon Plugin.

The documentation of the charms can be found here:

TrilioVault data protection — charm-guide 0.0.1.dev821 documentationdocs.openstack.org

The following steps have been tested and verified within Trilio environments. There have been cases where these steps updated all packages inside the LXC containers, leading to failures in basic OpenStack services.

It is recommended to run each of these steps in dry-run first.

When any other packages but Trilio packages are getting updated, stop the upgrade procedure and contact your Trilio customer success manager.

Upgrading the Trilio packages

Trilio is releasing hotfixes, which require updating the packages inside the containers. These hotfixes can not be installed using the Juju charms as they don't require an update to the charms.

Generic Pre-requisites

1. Either 4.1 GA OR any hotfix patch against 4.1 should be already deployed for performing upgrades mentioned in the current document.

2. No snapshot OR restore to be running.

3. Global job scheduler should be disabled.

4. wlm-cron should be disabled ( Following commands are to be run on MAAS node)

   1. If trilio-wlm is HA enabled, set the cluster configuration to maintenance mode ( this command will fail for single node deployment) juju exec [-m <model>] --unit trilio-wlm/leader "sudo crm configure property maintenance-mode=true"

   2. Stop wlm-cron service juju exec [-m <model>] --application trilio-wlm "sudo systemctl stop wlm-cron"

   3. Ensure that no stale wlm-cron processes are there juju exec [-m <model>] --application trilio-wlm "sudo ps -ef | grep [w]orkloadmgr-cron"

      If any stale process is found, that needs to be killed manually.

5. The mentioned gemfury repository should be accessible from trilio units.

Upgrade package on trilio units

The deployed Trilio version is controlled by the triliovault-pkg-source charm configuration option. For each trilio charm it should be pointing to below gemfury repository. This can be checked via juju [-m ] config triliovault-pkg-source command output.

deb [trusted=yes] https://apt.fury.io/triliodata-4-1/ /

This is the preferred, recommended and tested method to update the packages is through the Juju command line.

Run below commands form MASS node

On Ubuntu Bionic environments

juju exec [-m <model>] --application trilio-wlm 'sudo apt-get update && sudo apt-get install -y --only-upgrade -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" workloadmgr python3-workloadmgrclient python3-contegoclient s3-fuse-plugin'
juju exec [-m <model>] --application trilio-horizon-plugin 'sudo apt-get update && sudo apt-get install -y --only-upgrade -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" tvault-horizon-plugin python-workloadmgrclient'
juju exec [-m <model>] --application trilio-dm-api 'sudo apt-get update && sudo apt-get install -y --only-upgrade -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" python3-dmapi'
juju exec [-m <model>] --application trilio-data-mover 'sudo apt-get update && sudo apt-get install -y --only-upgrade -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" tvault-contego s3-fuse-plugin'

On other (not Ubuntu Bionic) environments

juju exec [-m <model>] --application trilio-wlm 'sudo apt-get update && sudo apt-get install -y --only-upgrade -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" workloadmgr python3-workloadmgrclient python3-contegoclient python3-s3-fuse-plugin'
juju exec [-m <model>] --application trilio-horizon-plugin 'sudo apt-get update && sudo apt-get install -y --only-upgrade -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" python3-tvault-horizon-plugin python3-workloadmgrclient'
juju exec [-m <model>] --application trilio-dm-api 'sudo apt-get update && sudo apt-get install -y --only-upgrade -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" python3-dmapi'
juju exec [-m <model>] --application trilio-data-mover 'sudo apt-get update && sudo apt-get install -y --only-upgrade -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" python3-tvault-contego python3-s3-fuse-plugin'

Check the trilio units status in juju status [-m ] | grep trilio output. All the trilio units will be with new package.

trilio-data-mover      <package version>  active      3  trilio-data-mover      jujucharms    8  ubuntu
trilio-dm-api          <package version>  active      1  trilio-dm-api          jujucharms    5  ubuntu
trilio-horizon-plugin  <package version>  active      1  trilio-horizon-plugin  jujucharms    4  ubuntu
trilio-wlm             <package version>  active      3  trilio-wlm             jujucharms    7  ubuntu

Update the DB schema

Run the below command to update the schema

juju exec [-m <model>] --unit trilio-wlm/leader "alembic -c /etc/workloadmgr/alembic.ini upgrade heads"

Check the schema head with below command. It should point to latest schema head.

juju exec [-m <model>] --unit trilio-wlm/leader "alembic -c /etc/workloadmgr/alembic.ini current"

Restart the apache2 service

Run below command to restart the apache2 service on horizon container

juju exec [-m <model>] -application trilio-horizon-plugin "systemctl restart apache2"

Post-Upgrade steps

1. If the trilio-wlm nodes are HA enabled:

   1. Make sure the wlm-cron services are down after the pkg upgrade. Run the following command for the same:juju exec [-m <model>] --application trilio-wlm "sudo systemctl stop wlm-cron"

   2. Unset the cluster maintenance modejuju exec [-m <model>] --unit trilio-wlm/leader "sudo crm configure property maintenance-mode=false"

2. Make sure the wlm-cron service up and running on any one node.juju exec [-m <model>] --application trilio-wlm "sudo systemctl status wlm-cron"

3. Set the Global Job Scheduler to the original state.

Troubleshooting

If any trilio unit get into error state with message

hook failed: "update-status"

Follow below steps

Login to trilio unit and run "sudo dpkg --configure -a"
It will ask for user input, hit enter and log out from the unit.
From mass node run command "juju resolve <trilio unit name>"