# Upgrading on Kolla OpenStack ## Upgrading from Trilio 4.0 to Trilio 4.1 Due to the new installation method of Trilio for Kolla OpenStack, it is required to reinstall the Trilio components running on the Kolla Openstack nodes when upgrading from Trilio 4.0. The Trilio appliance can be upgraded as documented. ## Upgrading from Trilio 4.1 to a higher version Trilio 4.1 can be upgraded without reinstallation to a higher version of T4O if available. {% hint style="info" %} Refer to the below-mentioned acceptable values for the placeholders in this document as per the Openstack environment:\ **kolla\_base\_distro :** ubuntu / centos\ **triliovault\_tag :** 4.1.94-hotfix-13-ussuri / 4.1.94-hotfix-12-victoria {% endhint %} ### Pre-requisites Please ensure the following points are met before starting the upgrade process: * Either 4.1 GA OR any hotfix patch against 4.1 should be already deployed * No Snapshot OR Restore is running * Global job scheduler should be disabled * wlm-cron is disabled on the primary Trilio Appliance * Access to the gemfury repository to fetch new packages #### Deactivating the wlm-cron service The following sets of commands will disable the wlm-cron service and verify that it is has been completly shut-down. ``` [root@TVM2 ~]# pcs resource disable wlm-cron [root@TVM2 ~]# systemctl status wlm-cron ● wlm-cron.service - workload's scheduler cron service Loaded: loaded (/etc/systemd/system/wlm-cron.service; disabled; vendor preset : disabled) Active: inactive (dead) Jun 11 08:27:06 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:06 - INFO - 1...t Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 140686268624368 Child 11389 ki...5 Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - 1...5 Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: Shutting down thread pool Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - S...l Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: Stopping the threads Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - S...s Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: All threads are stopped succes...y Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - A...y Jun 11 08:27:09 TVM2 systemd[1]: Stopped workload's scheduler cron service. Hint: Some lines were ellipsized, use -l to show in full. [root@TVM2 ~]# pcs resource show wlm-cron Resource: wlm-cron (class=systemd type=wlm-cron) Meta Attrs: target-role=Stopped Operations: monitor interval=30s on-fail=restart timeout=300s (wlm-cron-monito r-interval-30s) start interval=0s on-fail=restart timeout=300s (wlm-cron-start-int erval-0s) stop interval=0s timeout=300s (wlm-cron-stop-interval-0s) [root@TVM2 ~]# ps -ef | grep -i workloadmgr-cron root 15379 14383 0 08:27 pts/0 00:00:00 grep --color=auto -i workloadmgr -cron ``` ### Clone latest configuration scripts Before the latest configuration script is loaded it is recommended to take a backup of the existing config scripts' folder & Trilio ansible roles.\ The following command can be used for this purpose: ``` mv triliovault-cfg-scripts triliovault-cfg-scripts_4.1old mv /usr/local/share/kolla-ansible/ansible/roles/triliovault /opt/triliovault_4.1old ``` Clone the latest configuration scripts of the required branch and access the deployment script directory for Kolla Ansible Openstack.\ Available branches to upgrade T4O 4.1 are: ``` git clone -b hotfix-13-TVO/4.1 https://github.com/trilioData/triliovault-cfg-scripts.git cd triliovault-cfg-scripts/kolla-ansible/ ``` Copy the downloaded Trilio ansible role into the Kolla-Ansible roles directory. ``` cp -R ansible/roles/triliovault /usr/local/share/kolla-ansible/ansible/roles/ ``` ### Append Trilio variables #### Clean old Trilio variables and append new Trilio variables {% hint style="warning" %} This step is not always required. It is recommended to compare`triliovault_globals.yml`with the Trilio entries in the`/etc/kolla/globals.yml`file. In case of no changes, this step can be skipped. {% endhint %} This is required, in case of some variable names changed, some new variables have been added, or old variables removed in the latest `triliovault_globals.yml` they need to be updated in `/etc/kolla/globals.yml` file. ``` #copy the backed-up original globals.yml which is not having triliovalut variables iniside current globals.yml cp /opt/globals.yml /etc/kolla/globals.yml #Append Trilio global variables to globals.yml cat ansible/triliovault_globals.yml >> /etc/kolla/globals.yml ``` #### Clean old Trilio passwords and append new Trilio password variables {% hint style="warning" %} This step is not always required. It is recommended to compare`triliovault_passwords.yml`with the Trilio entries in the`/etc/kolla/passwords.yml`file. In case of no changes, this step can be skipped. {% endhint %} This step is required, when some password variable names have been added, changed, or removed in the latest triliovault\_passwords.yml. In this case, the /etc/kolla/passwords.yml needs to be updated. ``` Take backup of current password file cp /etc/kolla/password.yml /opt/password-.yml #Reset the passwords file to default one by reverting the backed-up original password.yml. This backup would have been taken during previous install/upgrade. cp /opt/password.yml /etc/kolla/password.yml #Append Trilio password variables to passwords.yml cat ansible/triliovault_passwords.yml >> /etc/kolla/passwords.yml #File /etc/kolla/passwords.yml to be edited to set passwords. #To set the passwords, it's recommended to use the same passwords as done during previous T4O deployment, as present in the password file backup (/opt/password-.yml). #Any additional passwords (in triliovault_passwords.yml), should be set by the user in /etc/kolla/passwords.yml. ``` #### Append triliovault\_site.yml content to kolla ansible's site.yml {% hint style="warning" %} This step is not always required. It is recommended to compare`triliovault_site.yml`with the Trilio entries in the`/usr/local/share/kolla-ansible/ansible/site.yml`file. In case of no changes, this step can be skipped. {% endhint %} This is required because, in case of some variable names changed, some new variables have been added, or old variables removed in the latest `triliovault_site.yml` they need to be updated in `/usr/local/share/kolla-ansible/ansible/site.yml` file. ``` #Take backup of current site.yml file cp /usr/local/share/kolla-ansible/ansible/site.yml /opt/site-.yml #Reset the site.yml to default one by reverting the backed-up original site.yml inside current site.yml. This backup would have been taken during previous install/upgrade. cp /opt/site.yml /usr/local/share/kolla-ansible/ansible/site.yml #Append Trilio code to site.yml cat ansible/triliovault_site.yml >> /usr/local/share/kolla-ansible/ansible/site.yml ``` #### Append triliovault\_inventory.txt to the kolla-ansible inventory file {% hint style="warning" %} This step is not always required. It is recommended to compare`triliovault_inventory.yml` ith the Trilio entries in the`/root/multinode`file. In case of no changes, this step can be skipped. {% endhint %} By default, the triliovault-datamover-api service gets installed on ‘control' hosts and the trilio-datamover service gets installed on 'compute’ hosts. You can edit the T4O groups in the inventory file as per your cloud architecture. T4O group names are ‘triliovault-datamover-api’ and ‘triliovault-datamover’ ``` For example: If your inventory file name path '/root/multinode' then use following #cleanup old T4O groups from /root/multinode and copy latest triliovault inventory file cat ansible/triliovault_inventory.txt >> /root/multinode ``` ### Edit globals.yml to set T4O parameters Edit '/etc/kolla/globals.yml' file to fill triliovault backup target and build details. You will find the triliovault related parameters at the end of globals.yml file. User needs to fill in details like triliovault build version, backup target type, backup target details, etc. Following is the list of parameters that the user needs to edit. | Parameter | Defaults/choices | comments | | -------------------------------------------- | -------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | triliovault\_tag |

\

| Trilio Build Version | | horizon\_image\_\_\_full | commented out | Uncomment to install Trilio Horizon Container instead of previous installed container | | triliovault\_docker\_\_\_username | triliodocker | | | triliovault\_docker\_\_\_password | triliopassword | | | triliovault\_docker\_\_\_registry | Default: docker.io |

If users want to use a different container registry for the triliovault containers, then the user can edit this value.
In that case, the user first needs to manually pull triliovault containers from docker.io and push them to the other registry.

| | triliovault\_backup\_\_\_target |

nfs

amazon\_s3

ceph\_s3

|

'nfs': If the backup target is NFS

'amazon\_s3': If the backup target is Amazon S3

'ceph\_s3': If the backup target type is S3 but not amazon S3.

| | dmapi\_workers | Default: 16 | If dmapi\_workers field is not present in config file. The Default value will be equals to number of cores present on the node | | triliovault\_nfs\_\_\_shares | |

Only with nfs for triliovault\_backup\_target

User needs to provide NFS share path, e.g.: 192.168.122.101:/opt/tvault

| | triliovault\_nfs\_\_\_options | Default: nolock, soft, timeo=180, intr, lookupcache=none |

Only with nfs for triliovault\_backup\_target

Keep default values if unclear

| | triliovault\_s3\_\_\_access\_key | |

Only with amazon\_s3 or cephs3 for triliovault\_backuptarget

Provide S3 access key

| | triliovault\_s3\_\_\_secret\_key | |

Only with amazon\_s3 or cephs3 for triliovault\_backuptarget

Provide S3 secret key

| | triliovault\_s3\_\_\_region\_name | Default: us-east-1 |

Only with amazon\_s3 or cephs3 for triliovault\_backuptarget

Provide S3 region or keep default if no region required

| | triliovault\_s3\_\_\_bucket\_name | |

Only with amazon\_s3 or cephs3 for triliovault\_backuptarget

Provide S3 bucket

| | triliovault\_s3\_\_\_endpoint\_url | |

Only with cephs3 for triliovault\_backuptarget

Provide S3 endpoint URL

| | triliovault\_s3\_\_\_ssl\_enabled |

True

False

|

Only with ceph\_s3 for triliovault\_backup\_target

Set to true if endpoint is on HTTPS

| | triliovault\_s3\_\_*ssl\_cert*\_\_file\_name | s3-cert-pem |

Only with ceph\_s3 for triliovault\_backup\_target and

if SSL is enabled on S3 endpoint URL and SSL certificates are self-signed OR issued by a private authority

user needs to copy the 'ceph s3 ca chain file' to "/etc/kolla/config/triliovault/"
directory on ansible server.
Create this directory if it does not exist already.

| | triliovault\_copy\_\_*ceph\_s3*\_\_ssl\_cert |

True

False

|

Set to true if:


ceph\_s3 for triliovault\_backup\_target and

if SSL is enabled on S3 endpoint URL and SSL certificates are self-signed OR issued by a private authority

| ### Enable T4O Snapshot mount feature {% hint style="info" %} This step is already part of the 4.1 GA installation procedure and should only be verified. {% endhint %} To enable Trilio's Snapshot mount feature it is necessary to make the Trilio Backup target available to the nova-compute and nova-libvirt containers. Edit `/usr/local/share/kolla-ansible/ansible/roles/nova-cell/defaults/main.yml` and find `nova_libvirt_default_volumes` variable. Append the Trilio mount bind `/var/trilio:/var/trilio:shared` to the list of already existing volumes. For a default Kolla installation, will the variable look as follows afterward: ``` nova_libvirt_default_volumes: - "{{ node_config_directory }}/nova-libvirt/:{{ container_config_directory }}/:ro" - "/etc/localtime:/etc/localtime:ro" - "{{ '/etc/timezone:/etc/timezone:ro' if ansible_os_family == 'Debian' else '' }}" - "/lib/modules:/lib/modules:ro" - "/run/:/run/:shared" - "/dev:/dev" - "/sys/fs/cgroup:/sys/fs/cgroup" - "kolla_logs:/var/log/kolla/" - "libvirtd:/var/lib/libvirt" - "{{ nova_instance_datadir_volume }}:/var/lib/nova/" - " {% if enable_shared_var_lib_nova_mnt | bool %}/var/lib/nova/mnt:/var/lib/nova/mnt:shared{% endif %} " - "nova_libvirt_qemu:/etc/libvirt/qemu" - "{{ kolla_dev_repos_directory ~ '/nova/nova:/var/lib/kolla/venv/lib/python' ~ distro_python_version ~ '/site-packages/nova' if nova_dev_mode | bool else '' } - "/var/trilio:/var/trilio:shared" ``` Next, find the variable `nova_compute_default_volumes` in the same file and append the mount bind `/var/trilio:/var/trilio:shared` to the list. After the change will the variable look for a default Kolla installation as follows: ``` nova_compute_default_volumes: - "{{ node_config_directory }}/nova-compute/:{{ container_config_directory }}/:ro" - "/etc/localtime:/etc/localtime:ro" - "{{ '/etc/timezone:/etc/timezone:ro' if ansible_os_family == 'Debian' else '' }}" - "/lib/modules:/lib/modules:ro" - "/run:/run:shared" - "/dev:/dev" - "kolla_logs:/var/log/kolla/" - " {% if enable_iscsid | bool %}iscsi_info:/etc/iscsi{% endif %}" - "libvirtd:/var/lib/libvirt" - "{{ nova_instance_datadir_volume }}:/var/lib/nova/" - "{% if enable_shared_var_lib_nova_mnt | bool %}/var/lib/nova/mnt:/var/lib/nova/mnt:shared{% endif %}" - "{{ kolla_dev_repos_directory ~ '/nova/nova:/var/lib/kolla/venv/lib/python' ~ distro_python_version ~ '/site-packages/nova' if nova_dev_mode | bool else '' }}" - "/var/trilio:/var/trilio:shared" ``` In case of using Ironic compute nodes one more entry need to be adjusted in the same file.\ Find the variable `nova_compute_ironic_default_volumes` and append trilio mount `/var/trilio:/var/trilio:shared` to the list. After the changes the variable will looks like the following: ``` nova_compute_ironic_default_volumes: - "{{ node_config_directory }}/nova-compute-ironic/:{{ container_config_directory }}/:ro" - "/etc/localtime:/etc/localtime:ro" - "{{ '/etc/timezone:/etc/timezone:ro' if ansible_os_family == 'Debian' else '' }}" - "kolla_logs:/var/log/kolla/" - "{{ kolla_dev_repos_directory ~ '/nova/nova:/var/lib/kolla/venv/lib/python' ~ distro_python_version ~ '/site-packages/nova' if nova_dev_mode | bool else '' }}" - "/var/trilio:/var/trilio:shared" ``` ### Pull containers in case of private repository In case, the user doesn’t want to use the docker hub registry for triliovault containers during cloud deployment, then the user can pull triliovault images before starting cloud deployment and push them to other preferred registries. Following are the triliovault container image URLs. Replace kolla\_base\_distro and triliovault\_tag variables with their values ``` docker.io/trilio/-binary-trilio-datamover-api: docker.io/trilio/-binary-trilio-datamover: docker.io/trilio/-binary-trilio-horizon-plugin: ``` ### Pull T4O container images Run the below command from the directory with the multinode file tull pull the required images. ``` kolla-ansible -i multinode pull --tags triliovault ``` ### Run Kolla-Ansible upgrade command Run the below command from the directory with the multinode file to start the upgrade process. ``` kolla-ansible -i multinode upgrade ``` ### Verify Trilio deployment Verify on the nodes that are supposed to run the Trilio containers, that those are available and healthy. ``` [root@controller ~]# docker ps | grep triliovault_datamover_api f00781997bc3 trilio/centos-binary-trilio-datamover-api: "dumb-init --single-…" 2 minutes ago Up 2 minutes triliovault_datamover_api [root@compute ~]# docker ps | grep triliovault_datamover 4831db5d215 trilio/centos-binary-trilio-datamover: "dumb-init --single-…" 5 minutes ago Up 4 minutes triliovault_datamover [root@controller ~]# docker ps | grep horizon f3647e0fff27 trilio/centos-binary-trilio-horizon-plugin: "dumb-init --single-…" 8 minutes ago Up 8 minutes horizon ``` ### Advance settings/configuration for Trilio services #### Customize HAproxy configuration parameters for Trilio datamover api service Following are the default haproxy conf parameters set against triliovault datamover api service. ``` retries 5 timeout http-request 10m timeout queue 10m timeout connect 10m timeout client 10m timeout server 10m timeout check 10m balance roundrobin maxconn 50000 ``` These values work best for triliovault dmapi service. It’s not recommended to change these parameter values. However, in some exceptional cases, If needed to change any of the above parameter values then same can be done on kolla-ansible server in the following file. ``` /usr/local/share/kolla-ansible/ansible/roles/triliovault/defaults/main.yml ``` After editing, run kolla-ansible deploy command again to push these changes to openstack cloud. Post kolla-ansible deploy, to verify the changes, please check following file, available on all controller/haproxy nodes. ``` /etc/kolla/haproxy/services.d/triliovault-datamover-api.cfg ```