Upgrading on Ansible OpenStack

Upgrading from Trilio 4.0 to Trilio 4.1

Due to the new installation method of Trilio for Kolla OpenStack, it is required to reinstall the Trilio components running on the Kolla OpenStack nodes when upgrading from Trilio 4.0.

The Trilio appliance can be upgraded as documented here.

Upgrading from Trilio 4.1 to a higher version

Trilio 4.1 can be upgraded without reinstallation to a higher version of T4O if available.

Pre-requisites

Please ensure the following points are met before starting the upgrade process:

• No Snapshot or Restore is running

• Global job scheduler is disabled

• wlm-cron is disabled on the Trilio Appliance

• Access to the gemfury repository to fetch new packages

Deactivating the wlm-cron service

The following sets of commands will disable the wlm-cron service and verify that it is has been completly shut down.

[root@TVM2 ~]# pcs resource disable wlm-cron
[root@TVM2 ~]# systemctl status wlm-cron
● wlm-cron.service - workload's scheduler cron service
   Loaded: loaded (/etc/systemd/system/wlm-cron.service; disabled; vendor preset                                                                                                                                                                                                                                                                                                                                                           : disabled)
   Active: inactive (dead)

Jun 11 08:27:06 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:06 - INFO - 1...t
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 140686268624368 Child 11389 ki...5
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - 1...5
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: Shutting down thread pool
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - S...l
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: Stopping the threads
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - S...s
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: All threads are stopped succes...y
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - A...y
Jun 11 08:27:09 TVM2 systemd[1]: Stopped workload's scheduler cron service.
Hint: Some lines were ellipsized, use -l to show in full.
[root@TVM2 ~]# pcs resource show wlm-cron
 Resource: wlm-cron (class=systemd type=wlm-cron)
  Meta Attrs: target-role=Stopped
  Operations: monitor interval=30s on-fail=restart timeout=300s (wlm-cron-monito                                                                                                                                                                                                                                                                                                                                                           r-interval-30s)
              start interval=0s on-fail=restart timeout=300s (wlm-cron-start-int                                                                                                                                                                                                                                                                                                                                                           erval-0s)
              stop interval=0s timeout=300s (wlm-cron-stop-interval-0s)
[root@TVM2 ~]# ps -ef | grep -i workloadmgr-cron
root     15379 14383  0 08:27 pts/0    00:00:00 grep --color=auto -i workloadmgr                                                                                                                                                                                                                                                                                                                                                           -cron

Update the repositories

Deb-based (Ubuntu)

Add the Gemfury repository on each dmapi, horizon containers & compute nodes.

Create file /etc/apt/sources.list.d/fury.list and add the below line to it.

deb [trusted=yes] https://apt.fury.io/triliodata-4-1/ /

The following commands can be used to verify the connection to the gemfury repository and to check for available packages.

apt-get update
apt list --upgradable

RPM-based (CentOS)

Add trilio repo on each dmapi, horizon containers & compute nodes.

Modify the file /etc/yum.repos.d/trilio.repo and add below line in it.

[triliovault-4-1]
name=triliovault-4-1
baseurl=http://trilio:[e-mail]:8283/triliodata-4-1/yum/
gpgcheck=0
enabled=1

The following commands can be used to verify the connection to the Trilio rpm server and to check for available packages.

yum repolist
yum check-upgrade

Upgrade tvault-datamover-api package

The following steps represent the best practice procedure to upgrade the dmapi service.

1. Login to dmapi container

2. Take a backup of the dmapi configuration in /etc/dmapi/

3. use apt list --upgradeable to identify the package used for the dmapi service

4. Update the dmapi package

5. restore the backed-up config files into /etc/dmapi/

6. Restart the dmapi container

7. Check the status of the dmapi service

These steps are done with the following commands. This example is assuming that the more common python3 packages are used.

Deb-based (Ubuntu)

lxc-attach -n controller_dmapi
tar -czvf dmapi_config.tar.gz /etc/dmapi
apt list --upgradable
apt install python3-dmapi --upgrade
tar -xzvf dmapi_config.tar.gz -C /
systemctl restart tvault-datamover-api
systemctl status tvault-datamover-api

RPM-based (CentOS)

lxc-attach -n controller_dmapi
tar -czvf dmapi_config.tar.gz /etc/dmapi
yum list installed | grep dmapi
yum check-update python3-dmapi
yum upgrade python3-dmapi
tar -xzvf dmapi_config.tar.gz -C /
systemctl restart tvault-datamover-api
systemctl status tvault-datamover-api

Upgrade Horizon plugin

The following steps represent the best practice procedure to update the Horizon plugin.

1. Login to Horizon Container

2. use apt list --upgradeable to identify the package the Trilio packages for the workloadmgrclient, contegoclient and Horizon plugin

3. Install the tvault-horizon-plugin package in the required python version

4. install the workloadmgrclient package

5. install the contegoclient

6. Restart the Horizon webserver

7. check the installed version of the workloadmgrclient

These steps are done with the following commands. This example is assuming that the more common python3 packages are used.

Deb-based (Ubuntu)

lxc-attach -n controller_horizon_container-ead7cc60
apt list --upgradable
apt install python3-tvault-horizon-plugin --upgrade
apt install python3-workloadmgrclient --upgrade
apt install python3-contegoclient --upgrade
systemctl restart apache2
workloadmgr --version

RPM-based (CentOS)

lxc-attach -n controller_horizon_container-ead7cc60
yum list installed | grep trilio
yum upgrade python3-contegoclient-el8 python3-tvault-horizon-plugin-el8 python3-workloadmgrclient-el8
systemctl restart httpd
workloadmgr --version

Upgrade the tvault-contego service

The following steps represent the best practice procedure to update the tvault-contego service on the compute nodes.

1. Login into the compute node

2. Take a backup of the config files in

   1. (NFS and S3) /etc/tvault-contego/

   2. (S3 only) /etc/tvault-object-store

3. use apt list --upgradeable to identify the tvault-contego package used

4. Unmount backup storage

5. upgrade the tvault-contego package in the required python version

6. (S3 only) upgrade the s3-fuse-plugin package

7. restore the config files into /etc/tvault-contego/

8. (S3 only) Restart the tvault-object-store service

9. Restart the tvault-contego service

10. check the status

These steps are done with the following commands. This example is assuming that the more common python3 packages are used.

Deb-based (Ubuntu)

tar -czvf  contego_config.tar.gz /etc/tvault-contego/ 
apt list --upgradable
(NFS only) umount /var/triliovault-mounts/<base64-hash>
(S3 only) umount /var/triliovault-mounts
apt install python3-tvault-contego --upgrade
apt install python3-s3-fuse-plugin --upgrade
tar -xzvf  contego_config.tar.gz -C /
systemctl restart tvault-object-store
systemctl restart tvault-contego
systemctl status tvault-contego
df -h

RPM-based (CentOS)

tar -czvf  contego_config.tar.gz /etc/tvault-contego/ 
yum list installed | grep tvault
yum upgrade python3-tvault-contego
yum upgrade python3-s3fuse-plugin
tar -xzvf  contego_config.tar.gz -C /
systemctl restart tvault-object-store
systemctl restart tvault-contego
systemctl status tvault-contego
df -h

Advance settings/configuration

Customize HAproxy cfg parameters for Trilio datamover api service

Following are the haproxy cfg parameters recommended for optimal performance of dmapi service. File location on controller /etc/haproxy/haproxy.cfg

retries 5
timeout http-request 10m
timeout queue 10m
timeout connect 10m
timeout client 10m
timeout server 10m
timeout check 10m
balance roundrobin
maxconn 50000

Parameters timeout client, timeout server and balance for DMAPI service

If values were already updated during any of the previous releases, further steps can be skipped.

Remove below content, if present in the file/etc/openstack_deploy/user_variables.ymlon ansible host.

haproxy_extra_services:
  - service:
      haproxy_service_name: datamover_service
      haproxy_backend_nodes: "{{ groups['dmapi_all'] | default([]) }}"
      haproxy_ssl: "{{ haproxy_ssl }}"
      haproxy_port: 8784
      haproxy_balance_type: http
      haproxy_backend_options:
        - "httpchk GET / HTTP/1.0\\r\\nUser-agent:\\ osa-haproxy-healthcheck"

Add the below lines at end of the file /etc/openstack_deploy/user_variables.yml on the ansible host.

# Datamover haproxy setting
haproxy_extra_services:
  - service:
      haproxy_service_name: datamover_service
      haproxy_backend_nodes: "{{ groups['dmapi_all'] | default([]) }}"
      haproxy_ssl: "{{ haproxy_ssl }}"
      haproxy_port: 8784
      haproxy_balance_type: http
      haproxy_balance_alg: roundrobin
      haproxy_timeout_client: 10m
      haproxy_timeout_server: 10m
      haproxy_backend_options:
        - "httpchk GET / HTTP/1.0\\r\\nUser-agent:\\ osa-haproxy-healthcheck"

Update Haproxy configuration using the below command on ansible host.

cd /opt/openstack-ansible/playbooks
openstack-ansible haproxy-install.yml