Upgrading on RHOSO

1] Configuration change

If any config parameter changed in tvo-operator-inputs.yaml like db user password or service endpoints, you can apply the changes using following command.

cd ctlplane-scripts
./deploy_tvo_control_plane.sh

Above command will output ‘configured' or 'unchanged’ depending upon changes happened in tvo-operator-inputs.yaml.

2] Upgrade to new build

Please ensure the following requirements are met before starting the upgrade process:

• No Snapshot or Restore is running

• Global job scheduler is disabled

Please follow below steps to upgrade to new build on RHOSO18 setup.

Take a backup of existing triliovault-cfg-scripts and clone latest triliovault-cfg-scripts github repository.

mv triliovault-cfg-scripts triliovault-cfg-scripts-old
git clone -b {{ trilio_branch }} https://github.com/trilioData/triliovault-cfg-scripts.git

Upgrade Path 1: Upgrade T4O 6.1.7+ to 6.1.9

2.1] Upgrade Trilio for OpenStack Operator

Run operator deployment with new image tag as mentioned in step 2.1 of this documentation

2.2] Edit tvo-operator-inputs.yaml file

Follow step 2.2 of this documentation

2.3] Run deploy command

cd triliovault-cfg-scripts/redhat-director-scripts/rhosp18/ctlplane-scripts/
./deploy_tvo_control_plane.sh

2.4] Verify successful deployment of T4O control plane services.

oc -n trilio-openstack get pods

3] Upgrade Trilio Data Plane Services

Follow step 3.2 to 3.8 of this documentation

4] Upgrade Trilio Horizon Plugin

Follow step 4 of this documentation and update the trilio horizon plugin with new image tag.

5] Upgrade dynamically added Backup Target

Please refer to step 5 of this documentation

Upgrade Path 2: Upgrade T4O from 6.1.6 or Earlier to 6.1.9

2.1] Edit tvo-operator-inputs.yaml file

Follow step 2.2 of this documentation

2.2] Upgrade T4O to latest T4O (at least to 6.1.9) with old db approach.

Update the following database parameters in tvo-operator-inputs.yaml to configure the old DB approach.

cd triliovault-cfg-scripts/redhat-director-scripts/rhosp18/ctlplane-scripts/
vi tvo-operator-inputs.yaml

  database:
    common:
      use_trilio_galera: false
      root_user_name: "root"
      host: "openstack.openstack.svc"
      port: 3306

Save file changes.

2.3] Create TLS certificate secrets

Follow step 2.4 of this documentation

2.4] Run deploy command

cd triliovault-cfg-scripts/redhat-director-scripts/rhosp18/ctlplane-scripts/
./deploy_tvo_control_plane.sh

2.5] Upgrade Trilio for OpenStack Operator

⚠️ Important: After running the above script, immediately upgrade the operator to version 6.1.9.

chmod +x install_operator.sh
./install_operator.sh <TVO_OPERATOR_CONTAINER_IMAGE_URL>

2.6] Verify successful deployment of T4O control plane services.

[openstackdev@localhost]$ oc get pods
NAME                                                READY   STATUS      RESTARTS   AGE
job-triliovault-datamover-api-db-init-kwt7q         0/1     Completed   0          8m43s
job-triliovault-datamover-api-keystone-init-gggcm   0/1     Completed   0          9m30s
job-triliovault-datamover-api-rabbitmq-init-mtck7   0/1     Completed   0          9m16s
job-triliovault-wlm-cloud-trust-f6xt2               0/1     Completed   0          8m27s
job-triliovault-wlm-db-init-c2tpm                   0/1     Completed   0          8m38s
job-triliovault-wlm-keystone-init-mw8q8             0/1     Completed   0          9m12s
job-triliovault-wlm-rabbitmq-init-qwvgr             0/1     Completed   0          8m47s
trilio-rabbitmq-cluster-server-0                    1/1     Running     0          10m
trilio-rabbitmq-cluster-server-1                    1/1     Running     0          10m
trilio-rabbitmq-cluster-server-2                    1/1     Running     0          10m
triliovault-datamover-api-7754567c94-57cqk          1/1     Running     0          8m27s
triliovault-datamover-api-7754567c94-9q47t          1/1     Running     0          8m27s
triliovault-datamover-api-7754567c94-rk6qc          1/1     Running     0          8m27s
triliovault-object-store-bt1-s3-fk7qs               1/1     Running     0          8m28s
triliovault-object-store-bt1-s3-lhvv7               1/1     Running     0          8m28s
triliovault-object-store-bt1-s3-p85p7               1/1     Running     0          8m28s
triliovault-wlm-api-85469585c7-8bbm2                1/1     Running     0          8m27s
triliovault-wlm-api-85469585c7-bf8nx                1/1     Running     0          8m27s
triliovault-wlm-api-85469585c7-z8q64                1/1     Running     0          8m27s
triliovault-wlm-cron-796c4d4554-7tgfh               1/1     Running     0          8m27s
triliovault-wlm-scheduler-65c6d68c94-8599n          1/1     Running     0          8m27s
triliovault-wlm-scheduler-65c6d68c94-pqqqp          1/1     Running     0          8m27s
triliovault-wlm-scheduler-65c6d68c94-rj6lb          1/1     Running     0          8m27s
triliovault-wlm-workloads-6b7f65fc4f-97gvj          1/1     Running     0          8m27s
triliovault-wlm-workloads-6b7f65fc4f-9mt7n          1/1     Running     0          8m27s
triliovault-wlm-workloads-6b7f65fc4f-wrs8f          1/1     Running     0          8m27s

Follow the steps mentioned below only if you want to migrate from the old DB approach (openstack-galera) to the new DB approach (trilio-galera) . Otherwise, skip steps 2.8 to 2.13 to continue using the old openstack-galera DB approach.

• Old DB approach: Trilio uses the DB service used by the OpenStack.

• New DB approach: Trilio deploys and uses its own DB service.

2.7] Create the trilio-openstack-secret in the trilio-openstack namespace.

• For the new database approach, the user needs to add DbRootPassword in Base64-encoded form to the trilio-openstack-secret.yaml file.

Follow step 1.4 of this documentation

2.8] Take workloadmgr Database Dump

Export workloadmgr database using following command from OpenStack’s galera cluster to a file

Create a directory to store .sql file using following command:

mkdir ~/trilio/

Get openstack-galera root password using following command:

oc -n openstack get secret osp-secret -o jsonpath='{.data.DbRootPassword}' | base64 -d

First, go inside the OpenStack Galera pod using the following command:

oc -n openstack rsh openstack-galera-0

Create a MySQL dump using the following command in the openstack-galera pod.

mysqldump -u root -p\
  --single-transaction \
  --no-create-info \
  --no-create-db \
  --skip-triggers \
  --skip-routines \
  --skip-events \
  --replace \
  workloadmgr > /tmp/t4o-workloadmgr.sql

### Exit from pod

Copy the above .sql file to the following directory using the command below:

cd ~/trilio/
oc -n openstack cp openstack-galera-0:/tmp/t4o-workloadmgr.sql ./t4o-workloadmgr.sql

2.9] Upgrade T4O 6.1.9 deployment with new database approach

Update the tvo-operator-inputs.yaml file with the following changes to configure Trilio to use the new DB installation approach.

cd triliovault-cfg-scripts/redhat-director-scripts/rhosp18/ctlplane-scripts/
vi tvo-operator-inputs.yaml

  database:
    common:
      use_trilio_galera: true
      root_user_name: "root"
      host: "trilio-galera-cluster.trilio-openstack.svc"
      port: 3306

2.10] Run deploy command

cd triliovault-cfg-scripts/redhat-director-scripts/rhosp18/ctlplane-scripts/
./deploy_tvo_control_plane.sh

2.11] Verify successful deployment of T4O control plane services.

[openstackdev@localhost ctlplane-scripts]$ oc get pods
NAME                                                READY   STATUS      RESTARTS   AGE
job-triliovault-datamover-api-db-init-flcvl         0/1     Completed   0          7m58s
job-triliovault-datamover-api-keystone-init-rfzxb   0/1     Completed   0          8m34s
job-triliovault-datamover-api-rabbitmq-init-zff58   0/1     Completed   0          8m20s
job-triliovault-wlm-cloud-trust-r99fr               0/1     Completed   0          7m42s
job-triliovault-wlm-db-init-l84hl                   0/1     Completed   0          7m53s
job-triliovault-wlm-keystone-init-6tp6r             0/1     Completed   0          8m16s
job-triliovault-wlm-rabbitmq-init-9vwd8             0/1     Completed   0          8m2s
trilio-galera-cluster-galera-0                      1/1     Running     0          9m17s
trilio-galera-cluster-galera-1                      1/1     Running     0          9m17s
trilio-galera-cluster-galera-2                      1/1     Running     0          9m17s
trilio-galera-wait-job-l4klg                        0/1     Completed   0          9m17s
trilio-rabbitmq-cluster-server-0                    1/1     Running     0          10m
trilio-rabbitmq-cluster-server-1                    1/1     Running     0          10m
trilio-rabbitmq-cluster-server-2                    1/1     Running     0          10m
triliovault-datamover-api-7d6977869-7gtfr           1/1     Running     0          7m43s
triliovault-datamover-api-7d6977869-lnvjb           1/1     Running     0          7m43s
triliovault-datamover-api-7d6977869-zqhqx           1/1     Running     0          7m43s
triliovault-object-store-bt1-s3-27bxz               1/1     Running     0          7m43s
triliovault-object-store-bt1-s3-hfrhw               1/1     Running     0          7m43s
triliovault-object-store-bt1-s3-phr6n               1/1     Running     0          7m43s
triliovault-wlm-api-566d9778d-287xk                 1/1     Running     0          7m42s
triliovault-wlm-api-566d9778d-gzqq6                 1/1     Running     0          7m42s
triliovault-wlm-api-566d9778d-hsksj                 1/1     Running     0          7m42s
triliovault-wlm-cron-85b45d4447-97r84               1/1     Running     0          7m42s
triliovault-wlm-scheduler-7558f58dc9-2lfnt          1/1     Running     0          7m42s
triliovault-wlm-scheduler-7558f58dc9-8v6ts          1/1     Running     0          7m42s
triliovault-wlm-scheduler-7558f58dc9-pqwqk          1/1     Running     0          7m42s
triliovault-wlm-workloads-8468959778-6r2tp          1/1     Running     0          7m42s
triliovault-wlm-workloads-8468959778-c5pzz          1/1     Running     0          7m42s
triliovault-wlm-workloads-8468959778-dffcw          1/1     Running     0          7m42s

2.12] Import Old workloadmgr Database Data into New Galera

After successful installation, import the database dump from the SQL file created using the mysqldump command.

Copy the workloadmgr database dump file to the Trilio Galera pod.

cd ~/trilio/
oc -n trilio-openstack cp ./t4o-workloadmgr.sql trilio-galera-cluster-galera-0:/tmp/t4o-workloadmgr.sql

Get the Trilio Galera root password using the following command:

oc -n trilio-openstack get secret trilio-openstack-secret -o jsonpath='{.data.DbRootPassword}' | base64 -d

Import the workloadmgr database data from the .sql file.

First, go inside the Trilio Galera pod using the following command:

oc -n trilio-openstack rsh trilio-galera-cluster-galera-0

Check the size of the .sql file using the following command:

du -sh /tmp/t4o-workloadmgr.sql

Import the .sql file into the workloadmgr database using the following command:

mysql -u root -p workloadmgr < /tmp/t4o-workloadmgr.sql

2.13] Restart of wlm-cron

After importing the database dump, the wlm-cron pod must be restarted. Restarting is required for wlm-cron to pick up the newly imported, scheduler-enabled workloads.

In OpenShift, a pod restart is performed by deleting the pod. The controller will automatically recreate it.

oc -n trilio-openstack get pods
oc -n trilio-openstack delete pod <wlm-cron-pod>

3] Upgrade Trilio Data Plane Services

Follow step 3 to 3.8 of this documentation

4] Upgrade Trilio Horizon Plugin

Follow step 4 of this documentation and update the trilio horizon plugin with new image tag.

5] Upgrade dynamically added Backup Target

Note: This step is needed only when there is change in python3-s3fuse-plugin-el9 package.

Please follow below steps to upgrade s3 backup target pods to new build on RHOSO18 setup.

Copy the input files from old triliovault-cfg-scripts directory to latest directory.

cp triliovault-cfg-scripts-old/redhat-director-scripts/rhosp18/ctlplane-scripts/tvo-backup-target-cr-amazon-s3.yaml triliovault-cfg-scripts/redhat-director-scripts/rhosp18/ctlplane-scripts/
cp triliovault-cfg-scripts-old/redhat-director-scripts/rhosp18/ctlplane-scripts/tvo-backup-target-cr-other-s3.yaml triliovault-cfg-scripts/redhat-director-scripts/rhosp18/ctlplane-scripts/
cp triliovault-cfg-scripts-old/redhat-director-scripts/rhosp18/dataplane-scripts/<BACKUP-TARGET-NAME> triliovault-cfg-scripts/redhat-director-scripts/rhosp18/dataplane-scripts/

5.1] Upgrade dynamic backup target on control plane

⚠️ Important Upgrade Note for T4O 6.1.3 or Older

• If you are on T4O 6.1.3 or older and have dynamic BTTs (TVOBackupTarget CRs), run the script below on your bastion node before upgrading to patch Helm annotations and labels:

  cd triliovault-cfg-scripts/redhat-director-scripts/rhosp18/ctlplane-scripts/
  chmod +x update_helm_annotations.sh
  ./update_helm_annotations.sh

• The script updates all dynamic TVOBackupTarget resources with required Helm annotations and labels so they are compatible with the current release.

• If a user has created dynamic BTTs using T4O 6.1.3 or older, this step is mandatory before upgrading.

• Users on T4O 6.1.4 or newer with BTTs created on 6.1.4+ do not need to run this script as the fix is included in the operator.

Update the below parameter with new image tag in respective files tvo-backup-target-cr-amazon-s3.yaml and tvo-backup-target-cr-other-s3.yaml.

cd triliovault-cfg-scripts/redhat-director-scripts/rhosp18/ctlplane-scripts
vi tvo-backup-target-cr-amazon-s3.yaml
vi tvo-backup-target-cr-other-s3.yaml

  images:
    triliovault_object_store: "registry.connect.redhat.com/trilio/trilio-wlm:<NEW-BUILD-TAG>"

Now apply the changes.

oc -n trilio-openstack apply -f tvo-backup-target-cr-amazon-s3.yaml
oc -n trilio-openstack apply -f tvo-backup-target-cr-other-s3.yaml

Check the s3 backup target containers up and running with new image.

oc -n trilio-openstack get pods | grep <backup-target-name>
oc -n trilio-openstack describe pod <pod-name> | grep Image:

5.2] Upgrade dynamic backup target on data plane

Update the wlm image parameter with new image tag in respective files.

cd triliovault-cfg-scripts/redhat-director-scripts/rhosp18/dataplane-scripts
vi <BACKUP-TARGET-NAME>/cm-trilio-backup-target.yaml


  trilio_env.yml: |
    triliovault_wlm_image: "registry.connect.redhat.com/trilio/trilio-wlm:<NEW-BUILD-TAG>"

Now apply the changes.

oc -n openstack apply -f <BACKUP-TARGET-NAME>/cm-trilio-backup-target.yaml

Update the ansible runner image with new image tag.

vi <BACKUP-TARGET-NAME>/trilio-add-backup-target-service.yaml


  openStackAnsibleEERunnerImage: registry.connect.redhat.com/trilio/trilio-ansible-runner-rhoso:<NEW-BUILD-TAG>

Now apply the changes.

oc -n openstack apply -f <BACKUP-TARGET-NAME>/trilio-add-backup-target-service.yaml

Update the deployment name with unique name.

vi <BACKUP-TARGET-NAME>/trilio-add-backup-target-deployment.yaml


metadata:
  name: edpm-trilio-add-backup-target-s3-bt4

Now apply the changes.

oc -n openstack apply -f <BACKUP-TARGET-NAME>/trilio-add-backup-target-deployment.yaml

Verify the s3 backup target containers on compute nodes.

podman ps -a | grep <BACKUP-TARGET-NAME>