1 of 1

Upgrading on Kolla OpenStack

Trilio supports the upgrade of Trilio-Openstack components from any of the older releases (4.1 onwards) to the latest 4.2 hotfix releases without ripping up the older deployments.

Refer to the below-mentioned acceptable values for the placeholders kolla_base_distro and triliovault_tag in this document as per the Openstack environment:

Openstack Version

triliovault_tag

kolla_base_distro

1] Pre-requisites

Please ensure the following points are met before starting the upgrade process:

Either 4.1 or 4.2 GA OR any hotfix patch against 4.1/4.2 should be already deployed
No Snapshot OR Restore is running
Global job scheduler should be disabled
wlm-cron is disabled on the primary Trilio Appliance
Access to the gemfury repository to fetch new packages

1.1] Deactivating the wlm-cron service

The following sets of commands will disable the wlm-cron service and verify that it has been completly shut-down.

[root@TVM2 ~]# pcs resource disable wlm-cron
[root@TVM2 ~]# systemctl status wlm-cron
● wlm-cron.service - workload's scheduler cron service
   Loaded: loaded (/etc/systemd/system/wlm-cron.service; disabled; vendor preset                                                                                                                                                                                                                                                                                                                                                           : disabled)
   Active: inactive (dead)

Jun 11 08:27:06 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:06 - INFO - 1...t
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 140686268624368 Child 11389 ki...5
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - 1...5
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: Shutting down thread pool
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - S...l
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: Stopping the threads
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - S...s
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: All threads are stopped succes...y
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - A...y
Jun 11 08:27:09 TVM2 systemd[1]: Stopped workload's scheduler cron service.
Hint: Some lines were ellipsized, use -l to show in full.
[root@TVM2 ~]# pcs resource show wlm-cron
 Resource: wlm-cron (class=systemd type=wlm-cron)
  Meta Attrs: target-role=Stopped
  Operations: monitor interval=30s on-fail=restart timeout=300s (wlm-cron-monito                                                                                                                                                                                                                                                                                                                                                           r-interval-30s)
              start interval=0s on-fail=restart timeout=300s (wlm-cron-start-int                                                                                                                                                                                                                                                                                                                                                           erval-0s)
              stop interval=0s timeout=300s (wlm-cron-stop-interval-0s)
[root@TVM2 ~]# ps -ef | grep -i workloadmgr-cron
root     15379 14383  0 08:27 pts/0    00:00:00 grep --color=auto -i workloadmgr                                                                                                                                                                                                                                                                                                                                                           -cron

2] Clone latest configuration scripts

Before the latest configuration script is loaded it is recommended to take a backup of the existing config scripts' folder & Trilio ansible roles. The following command can be used for this purpose:

mv triliovault-cfg-scripts triliovault-cfg-scripts_old
mv /usr/local/share/kolla-ansible/ansible/roles/triliovault /opt/triliovault_old

Clone the latest configuration scripts of the required branch and access the deployment script directory for Kolla Ansible Openstack.

git clone -b 4.3.2 https://github.com/trilioData/triliovault-cfg-scripts.git

cd triliovault-cfg-scripts/kolla-ansible/

Copy the downloaded Trilio ansible role into the Kolla-Ansible roles directory.

cp -R ansible/roles/triliovault /usr/local/share/kolla-ansible/ansible/roles/

3] Append Trilio variables

3.1] Clean old Trilio variables and append new Trilio variables

This step is not always required. It is recommended to comparetriliovault_globals.ymlwith the Trilio entries in the/etc/kolla/globals.ymlfile.

In case of no changes, this step can be skipped.

This is required, in case of some variable names changed, some new variables have been added, or old variables removed in the latest triliovault_globals.yml they need to be updated in /etc/kolla/globals.yml file.

#copy the backed-up original globals.yml which is not having triliovault variables iniside current globals.yml
cp /opt/globals.yml /etc/kolla/globals.yml

#Append Trilio global variables to globals.yml
cat ansible/triliovault_globals.yml >> /etc/kolla/globals.yml

3.2] Clean old Trilio passwords and append new Trilio password variables

This step is not always required. It is recommended to comparetriliovault_passwords.ymlwith the Trilio entries in the/etc/kolla/passwords.ymlfile.

In case of no changes, this step can be skipped.

This step is required, when some password variable names have been added, changed, or removed in the latest triliovault_passwords.yml. In this case, the /etc/kolla/passwords.yml needs to be updated.

Take backup of current password file
cp /etc/kolla/password.yml /opt/password-<CURRENT-RELEASE>.yml

#Reset the passwords file to default one by reverting the backed-up original password.yml. This backup would have been taken during previous install/upgrade.
cp /opt/password.yml /etc/kolla/password.yml

#Append Trilio password variables to passwords.yml 
cat ansible/triliovault_passwords.yml >> /etc/kolla/passwords.yml

#File /etc/kolla/passwords.yml to be edited to set passwords.
#To set the passwords, it's recommended to use the same passwords as done during previous T4O deployment, as present in the password file backup (/opt/password-<CURRENT-RELEASE>.yml). 
#Any additional passwords (in triliovault_passwords.yml), should be set by the user in /etc/kolla/passwords.yml.

3.3] Append triliovault_site.yml content to kolla ansible's site.yml

This step is not always required. It is recommended to comparetriliovault_site.ymlwith the Trilio entries in the/usr/local/share/kolla-ansible/ansible/site.ymlfile.

In case of no changes, this step can be skipped.

This is required because, in case of some variable names changed, some new variables have been added, or old variables removed in the latest triliovault_site.yml they need to be updated in /usr/local/share/kolla-ansible/ansible/site.yml file.

#Take backup of current site.yml file
cp /usr/local/share/kolla-ansible/ansible/site.yml /opt/site-<CURRENT-RELEASE>.yml

#Reset the site.yml to default one by reverting the backed-up original site.yml inside current site.yml. This backup would have been taken during previous install/upgrade.
cp /opt/site.yml /usr/local/share/kolla-ansible/ansible/site.yml

# If the OpenStack release is ‘yoga' append below Trilio code to site.yml  
cat ansible/triliovault_site_yoga.yml >> /usr/local/share/kolla-ansible/ansible/site.yml    

# If the OpenStack release is other than 'yoga' append below Trilio code to site.yml 
cat ansible/triliovault_site.yml >> /usr/local/share/kolla-ansible/ansible/site.yml

3.4] Append triliovault_inventory.txt to the kolla-ansible inventory file

This step is not always required. It is recommended to comparetriliovault_inventory.yml ith the Trilio entries in the/root/multinodefile.

In case of no changes, this step can be skipped.

By default, the triliovault-datamover-api service gets installed on ‘control' hosts and the trilio-datamover service gets installed on 'compute’ hosts. You can edit the T4O groups in the inventory file as per your cloud architecture.

T4O group names are ‘triliovault-datamover-api’ and ‘triliovault-datamover’

For example:
If your inventory file name path '/root/multinode' then use following
#cleanup old T4O groups from /root/multinode and copy latest triliovault inventory file
cat ansible/triliovault_inventory.txt >> /root/multinode

4] Configure multi-IP NFS as Trilio backup target

This step is only required when the multi-IP NFS feature is used to connect different datamovers to the same NFS volume through multiple IPs

On kolla-ansible server node, change directory

cd triliovault-cfg-scripts/common/

Edit file 'triliovault_nfs_map_input.yml' in the current directory and provide compute host and NFS share/ip map.

If IP addresses are used in the kolla-ansible inventory file then you should use same IP addresses in 'triliovault_nfs_map_input.yml' file too. If you used hostnames there then you need to use same hostnames here in nfs map input file.

Compute host names or IP addresses that you are using in nfs map input file here should match with kolla-ansible inventory file entries.

vi triliovault_nfs_map_input.yml

The triliovault_nfs_map_input.yml is explained here.

Update PyYAML on the kolla-ansible server node only

pip3 install -U pyyaml

Expand the map file to create a one-to-one mapping of compute and NFS share.

python ./generate_nfs_map.py

The result will be in file - 'triliovault_nfs_map_output.yml'

Validate output map file

Open file 'triliovault_nfs_map_output.yml

vi triliovault_nfs_map_output.yml

available in the current directory and validate that all compute nodes are covered with all necessary NFS shares.

Append this output map file to triliovault_globals.yml File Path: /home/stack/triliovault-cfg-scripts/kolla-ansible/ansible/triliovault_globals.yml

cat triliovault_nfs_map_output.yml >> ../kolla-ansible/ansible/triliovault_globals.yml

Ensure to set multi_ip_nfs_enabled in _`_triliovault_globals.yml` file to yes

A new parameter is added to triliovault_globals.yml , set this parameter value to 'yes' if backup target NFS supports multiple endpoints/IPs. File Path: '/home/stack/triliovault-cfg-scripts/kolla-ansible/ansible/triliovault_globals.yml’ multi_ip_nfs_enabled: 'yes'
Later append triliovault_globals.yaml file to /etc/kolla/globals.yml

5] Edit globals.yml to set T4O parameters

Edit /etc/kolla/globals.yml file to fill triliovault backup target and build details. You will find the triliovault related parameters at the end of globals.yml. The user needs to fill in details like triliovault build version, backup target type, backup target details, etc.

Following is the list of parameters that the user needs to edit.

6] Enable T4O Snapshot mount feature

This step is already part of the 4.2 GA installation procedure and should only be verified.

To enable Trilio's Snapshot mount feature it is necessary to make the Trilio Backup target available to the nova-compute and nova-libvirt containers.

Edit /usr/local/share/kolla-ansible/ansible/roles/nova-cell/defaults/main.yml and find nova_libvirt_default_volumes variable. Append the Trilio mount bind /var/trilio:/var/trilio:shared to the list of already existing volumes.

For a default Kolla installation, will the variable look as follows afterward:

nova_libvirt_default_volumes:
  - "{{ node_config_directory }}/nova-libvirt/:{{ container_config_directory }}/:ro"
  - "/etc/localtime:/etc/localtime:ro"
  - "{{ '/etc/timezone:/etc/timezone:ro' if ansible_os_family == 'Debian' else '' }}"
  - "/lib/modules:/lib/modules:ro"
  - "/run/:/run/:shared"
  - "/dev:/dev"
  - "/sys/fs/cgroup:/sys/fs/cgroup"
  - "kolla_logs:/var/log/kolla/"
  - "libvirtd:/var/lib/libvirt"
  - "{{ nova_instance_datadir_volume }}:/var/lib/nova/"
  - "
{% if enable_shared_var_lib_nova_mnt | bool %}/var/lib/nova/mnt:/var/lib/nova/mnt:shared{% endif %}




"
  - "nova_libvirt_qemu:/etc/libvirt/qemu"
  - "{{ kolla_dev_repos_directory ~ '/nova/nova:/var/lib/kolla/venv/lib/python' ~ distro_python_version ~ '/site-packages/nova' if nova_dev_mode | bool else '' }
  - "/var/trilio:/var/trilio:shared"

Next, find the variable nova_compute_default_volumes in the same file and append the mount bind /var/trilio:/var/trilio:shared to the list.

After the change will the variable look for a default Kolla installation as follows:

nova_compute_default_volumes:
  - "{{ node_config_directory }}/nova-compute/:{{ container_config_directory }}/:ro"
  - "/etc/localtime:/etc/localtime:ro"
  - "{{ '/etc/timezone:/etc/timezone:ro' if ansible_os_family == 'Debian' else '' }}"
  - "/lib/modules:/lib/modules:ro"
  - "/run:/run:shared"
  - "/dev:/dev"
  - "kolla_logs:/var/log/kolla/"
  - "
{% if enable_iscsid | bool %}iscsi_info:/etc/iscsi{% endif %}"
  - "libvirtd:/var/lib/libvirt"
  - "{{ nova_instance_datadir_volume }}:/var/lib/nova/"
  - "{% if enable_shared_var_lib_nova_mnt | bool %}/var/lib/nova/mnt:/var/lib/nova/mnt:shared{% endif %}


"
  - "{{ kolla_dev_repos_directory ~ '/nova/nova:/var/lib/kolla/venv/lib/python' ~ distro_python_version ~ '/site-packages/nova' if nova_dev_mode | bool else '' }}"
  - "/var/trilio:/var/trilio:shared"

In the case of using Ironic compute nodes one more entry needs to be adjusted in the same file. Find the variable nova_compute_ironic_default_volumes and append trilio mount /var/trilio:/var/trilio:shared to the list.

After the changes the variable will look like the following:

nova_compute_ironic_default_volumes:
  - "{{ node_config_directory }}/nova-compute-ironic/:{{ container_config_directory }}/:ro"
  - "/etc/localtime:/etc/localtime:ro"
  - "{{ '/etc/timezone:/etc/timezone:ro' if ansible_os_family == 'Debian' else '' }}"
  - "kolla_logs:/var/log/kolla/"
  - "{{ kolla_dev_repos_directory ~ '/nova/nova:/var/lib/kolla/venv/lib/python' ~ distro_python_version ~ '/site-packages/nova' if nova_dev_mode | bool else '' }}"
  - "/var/trilio:/var/trilio:shared"

7] Pull containers in case of private repository

In case, the user doesn’t want to use the docker hub registry for triliovault containers during cloud deployment, then the user can pull triliovault images before starting cloud deployment and push them to other preferred registries.

Following are the triliovault container image URLs for 4.3.2 releases. Replace kolla_base_distro and triliovault_tag variables with their values.

This {{ kolla_base_distro }} variable can be either 'centos' or 'ubuntu' depends on your base OpenStack distro This {{ triliovault_tag }} is mentioned at the start of this document.

Trilio supports the Source-based containers from the OpenStack Yoga release **** onwards.

Below are the Source-based OpenStack deployment images

docker.io/trilio/kolla-{{ kolla_base_distro }}-trilio-datamover:{{ triliovault_tag }}
docker.io/trilio/kolla-{{ kolla_base_distro }}-trilio-datamover-api:{{ triliovault_tag }}
docker.io/trilio/kolla-{{ kolla_base_distro }}-trilio-horizon-plugin:{{ triliovault_tag }}

## EXAMPLE from Kolla Ubuntu source based OpenStack
docker.io/trilio/kolla-ubuntu-trilio-datamover:{{ triliovault_tag }}
docker.io/trilio/kolla-ubuntu-trilio-datamover-api:{{ triliovault_tag }}
docker.io/trilio/kolla-ubuntu-trilio-horizon-plugin:{{ triliovault_tag }}

Below are the Binary-based OpenStack deployment images

docker.io/trilio/kolla-{{ kolla_base_distro }}-trilio-datamover:{{ triliovault_tag }}
docker.io/trilio/kolla-{{ kolla_base_distro }}-trilio-datamover-api:{{ triliovault_tag }}
docker.io/trilio/{{ kolla_base_distro }}-binary-trilio-horizon-plugin:{{ triliovault_tag }}

## EXAMPLE from Kolla Ubuntu binary based OpenStack
docker.io/trilio/kolla-ubuntu-trilio-datamover:{{ triliovault_tag }}
docker.io/trilio/kolla-ubuntu-trilio-datamover-api:{{ triliovault_tag }}
docker.io/trilio/ubuntu-binary-trilio-horizon-plugin:{{ triliovault_tag }}

8] Pull T4O container images

Activate the login into dockerhub for Trilio tagged containers.

Please get the Dockerhub login credentials from Trilio Sales/Support team

ansible -i multinode control -m shell -a "docker login -u <docker-login-username> -p <docker-login-password> docker.io"

Run the below command from the directory with the multinode file tull pull the required images.

kolla-ansible -i multinode pull --tags triliovault

9] Run Kolla-Ansible upgrade command

Run the below command from the directory with the multinode file to start the upgrade process.

kolla-ansible -i multinode upgrade

10] Verify Trilio deployment

Verify on the controller and compute nodes that the Trilio containers are in UP state.

Following is a sample output of commands from controller and compute nodes. triliovault_tag will have value as per the openstack release where deployment being done.

[controller] docker ps  | grep "trilio-"
a2a3593f76db   trilio/kolla-centos-trilio-datamover-api:<triliovault_tag>       "dumb-init --single-…"   23 hours ago    Up 23 hours    triliovault_datamover_api
5f573caa7b02   trilio/kolla-centos-trilio-horizon-plugin:<triliovault_tag>      "dumb-init --single-…"   23 hours ago    Up 23 hours              horizon

[compute] docker ps | grep "trilio-"
f6d443c2942c   trilio/kolla-centos-trilio-datamover:<triliovault_tag>          "dumb-init --single-…"   23 hours ago    Up 23 hours    triliovault_datamover

11] Advance settings/configuration for Trilio services

11.1] Customize HAproxy configuration parameters for Trilio datamover api service

Following are the default haproxy conf parameters set against triliovault datamover api service.

retries 5
timeout http-request 10m
timeout queue 10m
timeout connect 10m
timeout client 10m
timeout server 10m
timeout check 10m
balance roundrobin
maxconn 50000

These values work best for triliovault dmapi service. It’s not recommended to change these parameter values. However, in some exceptional cases, If needed to change any of the above parameter values then same can be done on kolla-ansible server in the following file.

/usr/local/share/kolla-ansible/ansible/roles/triliovault/defaults/main.yml

After editing, run kolla-ansible deploy command again to push these changes to openstack cloud.

Post kolla-ansible deploy, to verify the changes, please check following file, available on all controller/haproxy nodes.

/etc/kolla/haproxy/services.d/triliovault-datamover-api.cfg

12] Enable mound-bind for NFS

In T4O 4.2 and later releases, deriving of the mount point has been changed. It is necessary to set up the mount-bind to make T4O 4.1 or older backups available for T4O 4.2 and onwards releases.

Please follow this documentation to ensure that Backups taken from T4O 4.1 or older can be used with T4O 4.2 and onwards releases.

Upgrading on Kolla OpenStack

Trilio supports the upgrade of Trilio-Openstack components from any of the older releases (4.1 onwards) to the latest 4.2 hotfix releases without ripping up the older deployments.

Refer to the below-mentioned acceptable values for the placeholders kolla_base_distro and triliovault_tag in this document as per the Openstack environment:

Openstack Version

triliovault_tag

kolla_base_distro

1] Pre-requisites

Please ensure the following points are met before starting the upgrade process:

Either 4.1 or 4.2 GA OR any hotfix patch against 4.1/4.2 should be already deployed
No Snapshot OR Restore is running
Global job scheduler should be disabled
wlm-cron is disabled on the primary Trilio Appliance
Access to the gemfury repository to fetch new packages

1.1] Deactivating the wlm-cron service

The following sets of commands will disable the wlm-cron service and verify that it has been completly shut-down.

[root@TVM2 ~]# pcs resource disable wlm-cron
[root@TVM2 ~]# systemctl status wlm-cron
● wlm-cron.service - workload's scheduler cron service
   Loaded: loaded (/etc/systemd/system/wlm-cron.service; disabled; vendor preset                                                                                                                                                                                                                                                                                                                                                           : disabled)
   Active: inactive (dead)

Jun 11 08:27:06 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:06 - INFO - 1...t
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 140686268624368 Child 11389 ki...5
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - 1...5
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: Shutting down thread pool
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - S...l
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: Stopping the threads
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - S...s
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: All threads are stopped succes...y
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - A...y
Jun 11 08:27:09 TVM2 systemd[1]: Stopped workload's scheduler cron service.
Hint: Some lines were ellipsized, use -l to show in full.
[root@TVM2 ~]# pcs resource show wlm-cron
 Resource: wlm-cron (class=systemd type=wlm-cron)
  Meta Attrs: target-role=Stopped
  Operations: monitor interval=30s on-fail=restart timeout=300s (wlm-cron-monito                                                                                                                                                                                                                                                                                                                                                           r-interval-30s)
              start interval=0s on-fail=restart timeout=300s (wlm-cron-start-int                                                                                                                                                                                                                                                                                                                                                           erval-0s)
              stop interval=0s timeout=300s (wlm-cron-stop-interval-0s)
[root@TVM2 ~]# ps -ef | grep -i workloadmgr-cron
root     15379 14383  0 08:27 pts/0    00:00:00 grep --color=auto -i workloadmgr                                                                                                                                                                                                                                                                                                                                                           -cron

2] Clone latest configuration scripts

Before the latest configuration script is loaded it is recommended to take a backup of the existing config scripts' folder & Trilio ansible roles. The following command can be used for this purpose:

mv triliovault-cfg-scripts triliovault-cfg-scripts_old
mv /usr/local/share/kolla-ansible/ansible/roles/triliovault /opt/triliovault_old

Clone the latest configuration scripts of the required branch and access the deployment script directory for Kolla Ansible Openstack.

git clone -b 4.3.2 https://github.com/trilioData/triliovault-cfg-scripts.git

cd triliovault-cfg-scripts/kolla-ansible/

Copy the downloaded Trilio ansible role into the Kolla-Ansible roles directory.

cp -R ansible/roles/triliovault /usr/local/share/kolla-ansible/ansible/roles/

3] Append Trilio variables

3.1] Clean old Trilio variables and append new Trilio variables

This step is not always required. It is recommended to comparetriliovault_globals.ymlwith the Trilio entries in the/etc/kolla/globals.ymlfile.

In case of no changes, this step can be skipped.

#copy the backed-up original globals.yml which is not having triliovault variables iniside current globals.yml
cp /opt/globals.yml /etc/kolla/globals.yml

#Append Trilio global variables to globals.yml
cat ansible/triliovault_globals.yml >> /etc/kolla/globals.yml

3.2] Clean old Trilio passwords and append new Trilio password variables

This step is not always required. It is recommended to comparetriliovault_passwords.ymlwith the Trilio entries in the/etc/kolla/passwords.ymlfile.

In case of no changes, this step can be skipped.

This step is required, when some password variable names have been added, changed, or removed in the latest triliovault_passwords.yml. In this case, the /etc/kolla/passwords.yml needs to be updated.

Take backup of current password file
cp /etc/kolla/password.yml /opt/password-<CURRENT-RELEASE>.yml

#Reset the passwords file to default one by reverting the backed-up original password.yml. This backup would have been taken during previous install/upgrade.
cp /opt/password.yml /etc/kolla/password.yml

#Append Trilio password variables to passwords.yml 
cat ansible/triliovault_passwords.yml >> /etc/kolla/passwords.yml

#File /etc/kolla/passwords.yml to be edited to set passwords.
#To set the passwords, it's recommended to use the same passwords as done during previous T4O deployment, as present in the password file backup (/opt/password-<CURRENT-RELEASE>.yml). 
#Any additional passwords (in triliovault_passwords.yml), should be set by the user in /etc/kolla/passwords.yml.

3.3] Append triliovault_site.yml content to kolla ansible's site.yml

This step is not always required. It is recommended to comparetriliovault_site.ymlwith the Trilio entries in the/usr/local/share/kolla-ansible/ansible/site.ymlfile.

In case of no changes, this step can be skipped.

#Take backup of current site.yml file
cp /usr/local/share/kolla-ansible/ansible/site.yml /opt/site-<CURRENT-RELEASE>.yml

#Reset the site.yml to default one by reverting the backed-up original site.yml inside current site.yml. This backup would have been taken during previous install/upgrade.
cp /opt/site.yml /usr/local/share/kolla-ansible/ansible/site.yml

# If the OpenStack release is ‘yoga' append below Trilio code to site.yml  
cat ansible/triliovault_site_yoga.yml >> /usr/local/share/kolla-ansible/ansible/site.yml    

# If the OpenStack release is other than 'yoga' append below Trilio code to site.yml 
cat ansible/triliovault_site.yml >> /usr/local/share/kolla-ansible/ansible/site.yml

3.4] Append triliovault_inventory.txt to the kolla-ansible inventory file

This step is not always required. It is recommended to comparetriliovault_inventory.yml ith the Trilio entries in the/root/multinodefile.

In case of no changes, this step can be skipped.

T4O group names are ‘triliovault-datamover-api’ and ‘triliovault-datamover’

For example:
If your inventory file name path '/root/multinode' then use following
#cleanup old T4O groups from /root/multinode and copy latest triliovault inventory file
cat ansible/triliovault_inventory.txt >> /root/multinode

4] Configure multi-IP NFS as Trilio backup target

This step is only required when the multi-IP NFS feature is used to connect different datamovers to the same NFS volume through multiple IPs

On kolla-ansible server node, change directory

cd triliovault-cfg-scripts/common/

Edit file 'triliovault_nfs_map_input.yml' in the current directory and provide compute host and NFS share/ip map.

Compute host names or IP addresses that you are using in nfs map input file here should match with kolla-ansible inventory file entries.

vi triliovault_nfs_map_input.yml

The triliovault_nfs_map_input.yml is explained here.

Update PyYAML on the kolla-ansible server node only

pip3 install -U pyyaml

Expand the map file to create a one-to-one mapping of compute and NFS share.

python ./generate_nfs_map.py

The result will be in file - 'triliovault_nfs_map_output.yml'

Validate output map file

Open file 'triliovault_nfs_map_output.yml

vi triliovault_nfs_map_output.yml

available in the current directory and validate that all compute nodes are covered with all necessary NFS shares.

Append this output map file to triliovault_globals.yml File Path: /home/stack/triliovault-cfg-scripts/kolla-ansible/ansible/triliovault_globals.yml

cat triliovault_nfs_map_output.yml >> ../kolla-ansible/ansible/triliovault_globals.yml

Ensure to set multi_ip_nfs_enabled in _`_triliovault_globals.yml` file to yes

A new parameter is added to triliovault_globals.yml , set this parameter value to 'yes' if backup target NFS supports multiple endpoints/IPs. File Path: '/home/stack/triliovault-cfg-scripts/kolla-ansible/ansible/triliovault_globals.yml’ multi_ip_nfs_enabled: 'yes'
Later append triliovault_globals.yaml file to /etc/kolla/globals.yml

5] Edit globals.yml to set T4O parameters

Following is the list of parameters that the user needs to edit.

Parameter

Defaults/choices

comments

6] Enable T4O Snapshot mount feature

This step is already part of the 4.2 GA installation procedure and should only be verified.

To enable Trilio's Snapshot mount feature it is necessary to make the Trilio Backup target available to the nova-compute and nova-libvirt containers.

For a default Kolla installation, will the variable look as follows afterward:

nova_libvirt_default_volumes:
  - "{{ node_config_directory }}/nova-libvirt/:{{ container_config_directory }}/:ro"
  - "/etc/localtime:/etc/localtime:ro"
  - "{{ '/etc/timezone:/etc/timezone:ro' if ansible_os_family == 'Debian' else '' }}"
  - "/lib/modules:/lib/modules:ro"
  - "/run/:/run/:shared"
  - "/dev:/dev"
  - "/sys/fs/cgroup:/sys/fs/cgroup"
  - "kolla_logs:/var/log/kolla/"
  - "libvirtd:/var/lib/libvirt"
  - "{{ nova_instance_datadir_volume }}:/var/lib/nova/"
  - "
{% if enable_shared_var_lib_nova_mnt | bool %}/var/lib/nova/mnt:/var/lib/nova/mnt:shared{% endif %}




"
  - "nova_libvirt_qemu:/etc/libvirt/qemu"
  - "{{ kolla_dev_repos_directory ~ '/nova/nova:/var/lib/kolla/venv/lib/python' ~ distro_python_version ~ '/site-packages/nova' if nova_dev_mode | bool else '' }
  - "/var/trilio:/var/trilio:shared"

Next, find the variable nova_compute_default_volumes in the same file and append the mount bind /var/trilio:/var/trilio:shared to the list.

After the change will the variable look for a default Kolla installation as follows:

nova_compute_default_volumes:
  - "{{ node_config_directory }}/nova-compute/:{{ container_config_directory }}/:ro"
  - "/etc/localtime:/etc/localtime:ro"
  - "{{ '/etc/timezone:/etc/timezone:ro' if ansible_os_family == 'Debian' else '' }}"
  - "/lib/modules:/lib/modules:ro"
  - "/run:/run:shared"
  - "/dev:/dev"
  - "kolla_logs:/var/log/kolla/"
  - "
{% if enable_iscsid | bool %}iscsi_info:/etc/iscsi{% endif %}"
  - "libvirtd:/var/lib/libvirt"
  - "{{ nova_instance_datadir_volume }}:/var/lib/nova/"
  - "{% if enable_shared_var_lib_nova_mnt | bool %}/var/lib/nova/mnt:/var/lib/nova/mnt:shared{% endif %}


"
  - "{{ kolla_dev_repos_directory ~ '/nova/nova:/var/lib/kolla/venv/lib/python' ~ distro_python_version ~ '/site-packages/nova' if nova_dev_mode | bool else '' }}"
  - "/var/trilio:/var/trilio:shared"

After the changes the variable will look like the following:

nova_compute_ironic_default_volumes:
  - "{{ node_config_directory }}/nova-compute-ironic/:{{ container_config_directory }}/:ro"
  - "/etc/localtime:/etc/localtime:ro"
  - "{{ '/etc/timezone:/etc/timezone:ro' if ansible_os_family == 'Debian' else '' }}"
  - "kolla_logs:/var/log/kolla/"
  - "{{ kolla_dev_repos_directory ~ '/nova/nova:/var/lib/kolla/venv/lib/python' ~ distro_python_version ~ '/site-packages/nova' if nova_dev_mode | bool else '' }}"
  - "/var/trilio:/var/trilio:shared"

7] Pull containers in case of private repository

Following are the triliovault container image URLs for 4.3.2 releases. Replace kolla_base_distro and triliovault_tag variables with their values.

This {{ kolla_base_distro }} variable can be either 'centos' or 'ubuntu' depends on your base OpenStack distro This {{ triliovault_tag }} is mentioned at the start of this document.

Trilio supports the Source-based containers from the OpenStack Yoga release **** onwards.

Below are the Source-based OpenStack deployment images

docker.io/trilio/kolla-{{ kolla_base_distro }}-trilio-datamover:{{ triliovault_tag }}
docker.io/trilio/kolla-{{ kolla_base_distro }}-trilio-datamover-api:{{ triliovault_tag }}
docker.io/trilio/kolla-{{ kolla_base_distro }}-trilio-horizon-plugin:{{ triliovault_tag }}

## EXAMPLE from Kolla Ubuntu source based OpenStack
docker.io/trilio/kolla-ubuntu-trilio-datamover:{{ triliovault_tag }}
docker.io/trilio/kolla-ubuntu-trilio-datamover-api:{{ triliovault_tag }}
docker.io/trilio/kolla-ubuntu-trilio-horizon-plugin:{{ triliovault_tag }}

Below are the Binary-based OpenStack deployment images

docker.io/trilio/kolla-{{ kolla_base_distro }}-trilio-datamover:{{ triliovault_tag }}
docker.io/trilio/kolla-{{ kolla_base_distro }}-trilio-datamover-api:{{ triliovault_tag }}
docker.io/trilio/{{ kolla_base_distro }}-binary-trilio-horizon-plugin:{{ triliovault_tag }}

## EXAMPLE from Kolla Ubuntu binary based OpenStack
docker.io/trilio/kolla-ubuntu-trilio-datamover:{{ triliovault_tag }}
docker.io/trilio/kolla-ubuntu-trilio-datamover-api:{{ triliovault_tag }}
docker.io/trilio/ubuntu-binary-trilio-horizon-plugin:{{ triliovault_tag }}

8] Pull T4O container images

Activate the login into dockerhub for Trilio tagged containers.

Please get the Dockerhub login credentials from Trilio Sales/Support team

ansible -i multinode control -m shell -a "docker login -u <docker-login-username> -p <docker-login-password> docker.io"

Run the below command from the directory with the multinode file tull pull the required images.

kolla-ansible -i multinode pull --tags triliovault

9] Run Kolla-Ansible upgrade command

Run the below command from the directory with the multinode file to start the upgrade process.

kolla-ansible -i multinode upgrade

10] Verify Trilio deployment

Verify on the controller and compute nodes that the Trilio containers are in UP state.

Following is a sample output of commands from controller and compute nodes. triliovault_tag will have value as per the openstack release where deployment being done.

[controller] docker ps  | grep "trilio-"
a2a3593f76db   trilio/kolla-centos-trilio-datamover-api:<triliovault_tag>       "dumb-init --single-…"   23 hours ago    Up 23 hours    triliovault_datamover_api
5f573caa7b02   trilio/kolla-centos-trilio-horizon-plugin:<triliovault_tag>      "dumb-init --single-…"   23 hours ago    Up 23 hours              horizon

[compute] docker ps | grep "trilio-"
f6d443c2942c   trilio/kolla-centos-trilio-datamover:<triliovault_tag>          "dumb-init --single-…"   23 hours ago    Up 23 hours    triliovault_datamover

11] Advance settings/configuration for Trilio services

11.1] Customize HAproxy configuration parameters for Trilio datamover api service

Following are the default haproxy conf parameters set against triliovault datamover api service.

retries 5
timeout http-request 10m
timeout queue 10m
timeout connect 10m
timeout client 10m
timeout server 10m
timeout check 10m
balance roundrobin
maxconn 50000

/usr/local/share/kolla-ansible/ansible/roles/triliovault/defaults/main.yml

After editing, run kolla-ansible deploy command again to push these changes to openstack cloud.

Post kolla-ansible deploy, to verify the changes, please check following file, available on all controller/haproxy nodes.

/etc/kolla/haproxy/services.d/triliovault-datamover-api.cfg

12] Enable mound-bind for NFS

In T4O 4.2 and later releases, deriving of the mount point has been changed. It is necessary to set up the mount-bind to make T4O 4.1 or older backups available for T4O 4.2 and onwards releases.

Please follow this documentation to ensure that Backups taken from T4O 4.1 or older can be used with T4O 4.2 and onwards releases.