Upgrade Trilio Appliance

Generic Pre-requisites

• Please ensure to complete the upgrade of all the TVault components on Openstack controller & compute nodes before starting the rolling upgrade of TVM.

• The mentioned gemfury repository should be accessible from TVault VM.

• Please ensure the following points before starting the upgrade process:

  • No snapshot OR restore to be running.

  • Global job-scheduler should be disabled.

• Take a backup of the conf files on all TVM nodes.

> tar -czvf tvault_backup.tar.gz /etc/tvault /etc/tvault-config /etc/workloadmgr
> cp tvault_backup.tar.gz /root/ 

• Activate the virtual environment on all TVM nodes.

> source /home/stack/myansible/bin/activate

• Export new TVault version PYPI url:

  > export TVAULT_PYPI=https://pypi.fury.io/triliodata-4-0/

Upgrade s3fuse/tvault-object-store

• Run the following command on all TVM nodes to upgrade s3fuse and its dependent packages.

  > pip install --extra-index-url $TVAULT_PYPI s3fuse --upgrade --no-cache-dir

Upgrade tvault-configurator

• Run the following command on all TVM nodes to upgrade tvault-configurator and its dependent packages.

  > pip install --extra-index-url $TVAULT_PYPI tvault-configurator --upgrade --no-cache-dir

Upgrade workloadmgr

• Run the upgrade command on all TVM nodes to upgrade workloadmgr and its dependent packages (workloadmgrclient, contegoclient, etc)

  > pip install --extra-index-url $TVAULT_PYPI workloadmgr --upgrade --no-cache-dir

Updating Config Parameters of specific services

• Update wlm-cron service entries

  • If Reconfigure is NOT planned, please perform following steps on all TVM nodes, else skip.

    • Update the following two parameters in wlm-cron.service systemd file (/etc/systemd/system/wlm-cron.service) :

      KillMode=control-group
      Restart=on-failure

    • And once done, use the following command to reload the service file:

      systemctl daemon-reload

• Maria DB changes

  • If Reconfigure is planned

    • Remove Galera clustered-flag from all TVM nodes & proceed with reconfigure

      rm -rf /etc/galera_cluster_configured

  • If Reconfigure is NOT planned

    • Increase the max SQL connections limit by doing the following steps:

      • Edit /etc/my.cnf.d/server.cnf file on each TVM node

      • Add the parameter max_connections=5000 under [mysqld] section

      • Stop and Start MariaDB service on each node one by one

        systemctl stop mariadb
        systemctl start mariadb

Post Upgrade Steps

• Restore the backed-up config files

  > cd /root 
  > tar -xzvf tvault_backup.tar.gz -C /

• Restart following services on all node(s) using respective commands

  systemctl restart wlm-api 
  systemctl restart wlm-workloads 
  systemctl restart tvault-config
  systemctl restart tvault-object-store (Required ONLY if TVault configured with S3 backend storage)

• Enable Global Job Scheduler

• Restart pcs resources only on the primary node

  pcs resource restart wlm-cron
  pcs resource restart wlm-scheduler

• Verify the status of the services

  • Note: tvault-object-store will run only if TVault configured with S3 backend storage

    On All nodes
    systemctl status wlm-api wlm-workloads tvault-config tvault-object-store | grep -E 'Active|loaded'
    On primary node
    pcs status
    systemctl status wlm-cron wlm-scheduler

• Additional check for wlm-cron on primary node

  ps -ef | grep workloadmgr-cron | grep -v grep
  Above command should show only 2 processes running; sample below
  
  [root@tvm6 ~]# ps -ef | grep workloadmgr-cron | grep -v grep
  nova      8841     1  2 Jul28 ?        00:40:44 /home/stack/myansible/bin/python3 /home/stack/myansible/bin/workloadmgr-cron --config-file=/etc/workloadmgr/workloadmgr.conf
  nova      8898  8841  0 Jul28 ?        00:07:03 /home/stack/myansible/bin/python3 /home/stack/myansible/bin/workloadmgr-cron --config-file=/etc/workloadmgr/workloadmgr.conf

• Check the mount point using “df -h” command