CentOS/RHEL based Openstack environments

Generic Pre-requisites

1. Please ensure following points before starting the upgrade process:

   1. No snapshot OR restore to be running.

   2. Global job scheduler should be disabled.

2. The mentioned gemfury repository should be accessible from TVault VM.

Rolling upgrade on Openstack Ansible

Repo Updates

Add trilio repo on each dmapi, horizon containers & compute node(s) to get updated packages.

Modify the file /etc/yum.repos.d/trilio.repo and add below line in it.

[trilio]
name=Trilio Repository
baseurl=http://trilio:[email protected]:8283/triliovault-4.0/yum/
enabled=1
gpgcheck=0

Use below commands to get list of updated packages available on repo's.

yum repolist
yum check-upgrade

1.2 Upgrade tvault-datamover-api package

• Login to dmapi container from controller node using below command.

  lxc-attach -n controller_dmapi

• Take a backup of following file on each dmapi container(s).

tar -czvf dmapi_config.tar.gz /etc/dmapi

• Add the trilio repo and upgrade the dmapi package using below command.Select the appropriate package depending on python version used.

You can check installed package using below command.

yum list installed | grep dmapi
yum check-update dmapi

• Upgrade the dmapi package using below command.

yum upgrade dmapi

• Restore the backed-up config files

tar -xzvf dmapi_config.tar.gz -C /

• Now restart and check the service tvault-datamover-api on the controller.

systemctl restart tvault-datamover-api
systemctl status tvault-datamover-api

Upgrade Horizon Plugin Package

• We need to upgrade below two packages to upgrade horizon plugin.

tvault-horizon-plugin workloadmgrclient

• Login to horizon container from controller node using below command.

  lxc-attach -n controller_horizon_container-3990523e

• Add the trilio repo and upgrade the tvault-horizon-plugin & workloadmgrclient packages using below command.Select the appropriate package depending on python version used.

yum list installed | grep trilio

• Depending on the output of above command upgrade appropriate packages.

yum upgrade tvault-horizon-plugin workloadmgrclient

• Restart the httpd service and verify the workloadmgrclient version using below commands.

systemctl restart httpd
workloadmgr --version

Upgrade Tvault-Contego Package

NFS Storage Backend

• Take a backup of following file on each compute node(s) for nfs storage backend.

tar -czvf  contego_config.tar.gz /etc/tvault-contego/ 

• Add the trilio repo and upgrade the tvault-contego package using below command.Select the appropriate package depending on python version used.

yum list installed | grep tvault

• Depending on the output of above command upgrade appropriate packages.

yum upgrade tvault-contego

• Restore the backed-up config files

 tar -xzvf  contego_config.tar.gz -C /

• Now restart the service tvault-contego. Verify the status of the service and check the mount point.

systemctl restart tvault-contego
systemctl status tvault-contego
df -h

S3 Storage Backend

• Take a backup of following file on each compute node(s) for s3 storage backend.

tar -czvf  contego_config.tar.gz /etc/tvault-contego/ /etc/tvault-object-store/

• Upgrade the tvault-contego and s3fuse-plugin packages using below command. Select the appropriate package depending on the python version used.

 yum upgrade tvault-contego python-s3fuse-plugin-cent7

Note*:- If you get prompt while installing, choose the default selection.

• Restore the backed-up config files

 tar -xzvf  contego_config.tar.gz -C /

• Now restart and check the service tvault-contego on compute node(s).

systemctl restart tvault-object-store
systemctl restart tvault-contego 
systemctl status tvault-object-store
systemctl status tvault-contego
df -h

Rolling upgrade on Kolla Openstack

Upgrade tvault-datamover-api package (trilio_datamover_api Container)

Container trilio_datamover_api needs to be redeployed. Follow below steps on all controller nodes :

• Backup Existing conf files/folders

Copy folders /etc/kolla/nova-api & /etc/kolla/trilio-datamover-api/ to some local location.
Eg. cp -r /etc/kolla/nova-api $HOME/4.0-BK/

• Stop and Remove existing container trilio_datamover_api

docker stop trilio_datamover_api
docker rm trilio_datamover_api

• Pull trilio_datamover_api container image

  ## Login to docker using credentials: triliodocker/triliopassword
  docker login docker.io
  
  ## Pull Trilio Datamover Api container using following command. You need to edit OS_PLATFORM, TVAULT_VERSION,
   OPENSTACK_RELEASE_NAME.
  
  <OS_PLATFORM> could be ubuntu/centos.
  <OPENSTACK_RELEASE_NAME> could would be train
  <TVAULT_VERSION>    : For exact GA build version check section 1. i) Plan for deployment
  
  docker pull docker.io/trilio/<OS_PLATFORM>-source-trilio-datamover-api:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME>
  
  Example command for train openstack on ubuntu platform with triliovault 4.0 release:
  docker pull docker.io/trilio/centos-source-trilio-datamover-api:4.0.115-train

• Run datamover api container.

  You need to edit OS_PLATFORM, TVAULT_VERSION, OPENSTACK_RELEASE_NAME. 
  <OS_PLATFORM> could be ubuntu/centos.
  <OPENSTACK_RELEASE_NAME> would be train
  <TVAULT_VERSION> : For exact TVAULT build version check section 1. i) Plan for deployment
  
  docker run --network host --name trilio_datamover_api -d --restart always -v /etc/kolla/trilio-datamover-api/nova.conf:/etc/nova/nova.conf \
  -v /etc/kolla/trilio-datamover-api/dmapi.conf:/etc/dmapi/dmapi.conf \
  -v /var/log/kolla/:/var/log/kolla/ \
  trilio/<OS_PLATFORM>-source-trilio-datamover-api:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME> /var/lib/kolla/venv/bin/python /usr/bin/dmapi-api 

• Verify deployment

  Make sure 'trilio_datamover_api' named container running well and not in 'restarting' state on all controller nodes. 
  'docker ps' command can be used for the same.
  
  # docker ps | grep trilio_datamover_api
  e603ea78094f  trilio/centos-source-trilio-datamover-api:4.0.106-train   "dumb-init --single-…"   5 minutes ago Up 5 minutes
  trilio_datamover_api

Upgrade Tvault-Contego (trilio_datamover Container)

Container trilio_datamover needs to be redeployed. Follow below steps on all Compute nodes :

• Backup Existing conf files/folders

Copy folder /etc/kolla/trilio-datamover to some local location.
Eg. cp -r /etc/kolla/trilio-datamover $HOME/4.0-BK/

• Stop & remove existing trilio_datamover container

docker stop trilio_datamover
docker rm trilio_datamover

• Pull Trilio Datamover container image using the following command:

  ## Login to docker using credentials: triliodocker/triliopassword
  docker login docker.io
  
  ## Pull Trilio Datamover Api container using following command. You need to edit OS_PLATFORM, TVAULT_VERSION,
  OPENSTACK_RELEASE_NAME.
  
  <OS_PLATFORM> could be ubuntu/centos.
  <OPENSTACK_RELEASE_NAME> would be train
  <TVAULT_VERSION> : For exact TVAULT build version check section 1. i) Plan for deployment
  
  docker pull docker.io/trilio/<OS_PLATFORM>-source-trilio-datamover:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME>
  
  Example command for train openstack on ubuntu platform with Trilio 4.0 release:
  docker pull docker.io/trilio/centos-source-trilio-datamover:4.0.106-train

• Run datamover container.

  -- In following docker run command you need to edit OS_PLATFORM, TVAULT_VERSION,
  OPENSTACK_RELEASE_NAME.
  
  <OS_PLATFORM> could be ubuntu/centos.
  <OPENSTACK_RELEASE_NAME> would be train
  <TVAULT_VERSION> : For exact TVAULT build version check section 1. i) Plan for deployment
  
  Note: If your cloud does not use 'ceph' storage for nova/cinder, remove '/etc/ceph' volume mount option 
  from below commands.
  
  ##If backup storage type is 'NFS'
  docker run --privileged --network host --name trilio_datamover -d \
  --restart always \
  -v /etc/kolla/trilio-datamover/tvault-contego.conf:/etc/tvault-contego/tvault-contego.conf \
  -v /etc/kolla/trilio-datamover/nova.conf:/etc/nova/nova.conf \
  -v /dev:/dev:rw \
  -v /etc/ceph:/etc/ceph:ro \
  -v nova_compute:/var/lib/nova/:rw \
  -v /var/log/kolla/:/var/log/kolla/ \
  -v iscsi_info:/etc/iscsi:rw -v /var/run/libvirt \
  trilio/<OS_PLATFORM>-source-trilio-datamover:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME> /opt/tvault/start_datamover_nfs

• Verify Deployment of trilio_datamover

Make sure 'trilio_datamover' named container running well and not in 'restarting' state on all compute nodes.
   'docker ps' command can be used for the same.
# docker ps | grep trilio_datamover
3e62813e5ade        trilio/centos-source-trilio-datamover:4.0.106-train   "dumb-init --single-…"   7 hours ago         Up 7 hours                              trilio_datamover

Upgrade Horizon Plugin Package

• Login to horizon container

docker exec -it horizon /bin/bash

• Add trilio repo on each controller(s) to get updated packages.

  • Create a file /etc/yum.repos.d/trilio.repo and add below line in it.

    [trilio]
    name=Trilio Repository
    baseurl=http://trilio:[email protected]:8283/triliovault-4.0/yum/
    enabled=1
    gpgcheck=0

• Use below commands to get list of updated packages available on repo's.

  yum repolist
  yum check-upgrade

• For Horizon plugin upgrade, following packages need to be upgraded.

tvault-horizon-plugin workloadmgrclient

• Select the appropriate package depending on python version used.

 yum list installed | grep trilio

• Depending on the output of above command upgrade appropriate packages

yum upgrade tvault-horizon-plugin workloadmgrclient

• Restart docker container (from controller node) and verify the workloadmgrclient version (inside horizon container).

docker restart horizon
docker exec -it horizon /bin/bash
workloadmgr --version

RedHat Director

Please refer https://triliodata.atlassian.net/wiki/spaces/TRIL/pages/2055864321/Upgrade+of+Trilio+components+on+RHOSP