Upgrading on RHOSO

1] Configuration change

If any config parameter changed in tvo-operator-inputs.yaml like db user password or service endpoints, you can apply the changes using following command.

cd ctlplane-scripts
./deploy_tvo_control_plane.sh

Above command will output ‘configured' or 'unchanged’ depending upon changes happened in tvo-operator-inputs.yaml.

2] Upgrade to new build

Please ensure the following requirements are met before starting the upgrade process:

No Snapshot or Restore is running
Global job scheduler is disabled

Please follow below steps to upgrade to new build on RHOSO18 setup.

Take a backup of existing triliovault-cfg-scripts and clone latest triliovault-cfg-scripts github repository.

mv triliovault-cfg-scripts triliovault-cfg-scripts-old
git clone -b {{ trilio_branch }} https://github.com/trilioData/triliovault-cfg-scripts.git

2.1] Create the trilio-openstack-secret in the trilio-openstack namespace.

Follow step 1.4 of this documentation

2.2] Edit tvo-operator-inputs.yaml file

Follow step 2.2 of this documentation

2.3] Upgrade T4O to latest T4O (at least to 6.1.7) with old db approach.

Update the following database parameters in tvo-operator-inputs.yaml to configure the old DB approach.

cd triliovault-cfg-scripts/redhat-director-scripts/rhosp18/ctlplane-scripts/
vi tvo-operator-inputs.yaml

  database:
    common:
      use_trilio_galera: false
      root_user_name: "root"
      host: "openstack.openstack.svc"
      port: 3306

Save file changes.

2.4] Create TLS certificate secrets

Follow step 2.4 of this documentation

2.5] Run deploy command

cd triliovault-cfg-scripts/redhat-director-scripts/rhosp18/ctlplane-scripts/
./deploy_tvo_control_plane.sh

2.6] Upgrade Trilio for OpenStack Operator

⚠️ Important: After running the above script, immediately upgrade the operator to version 6.1.7.

chmod +x install_operator.sh
./install_operator.sh <TVO_OPERATOR_CONTAINER_IMAGE_URL>

2.7] Verify successful deployment of T4O control plane services.

[openstackdev@localhost]$ oc get pods
NAME                                                READY   STATUS      RESTARTS   AGE
job-triliovault-datamover-api-db-init-kwt7q         0/1     Completed   0          8m43s
job-triliovault-datamover-api-keystone-init-gggcm   0/1     Completed   0          9m30s
job-triliovault-datamover-api-rabbitmq-init-mtck7   0/1     Completed   0          9m16s
job-triliovault-wlm-cloud-trust-f6xt2               0/1     Completed   0          8m27s
job-triliovault-wlm-db-init-c2tpm                   0/1     Completed   0          8m38s
job-triliovault-wlm-keystone-init-mw8q8             0/1     Completed   0          9m12s
job-triliovault-wlm-rabbitmq-init-qwvgr             0/1     Completed   0          8m47s
trilio-rabbitmq-cluster-server-0                    1/1     Running     0          10m
trilio-rabbitmq-cluster-server-1                    1/1     Running     0          10m
trilio-rabbitmq-cluster-server-2                    1/1     Running     0          10m
triliovault-datamover-api-7754567c94-57cqk          1/1     Running     0          8m27s
triliovault-datamover-api-7754567c94-9q47t          1/1     Running     0          8m27s
triliovault-datamover-api-7754567c94-rk6qc          1/1     Running     0          8m27s
triliovault-object-store-bt1-s3-fk7qs               1/1     Running     0          8m28s
triliovault-object-store-bt1-s3-lhvv7               1/1     Running     0          8m28s
triliovault-object-store-bt1-s3-p85p7               1/1     Running     0          8m28s
triliovault-wlm-api-85469585c7-8bbm2                1/1     Running     0          8m27s
triliovault-wlm-api-85469585c7-bf8nx                1/1     Running     0          8m27s
triliovault-wlm-api-85469585c7-z8q64                1/1     Running     0          8m27s
triliovault-wlm-cron-796c4d4554-7tgfh               1/1     Running     0          8m27s
triliovault-wlm-scheduler-65c6d68c94-8599n          1/1     Running     0          8m27s
triliovault-wlm-scheduler-65c6d68c94-pqqqp          1/1     Running     0          8m27s
triliovault-wlm-scheduler-65c6d68c94-rj6lb          1/1     Running     0          8m27s
triliovault-wlm-workloads-6b7f65fc4f-97gvj          1/1     Running     0          8m27s
triliovault-wlm-workloads-6b7f65fc4f-9mt7n          1/1     Running     0          8m27s
triliovault-wlm-workloads-6b7f65fc4f-wrs8f          1/1     Running     0          8m27s

Follow the steps mentioned below only if you want to migrate from the old DB approach (openstack-galera) to the new DB approach (trilio-galera) . Otherwise, skip steps 2.8 to 2.13 to continue using the old openstack-galera DB approach.

Old DB approach: Trilio uses the DB service used by the OpenStack.
New DB approach: Trilio deploys and uses its own DB service.

2.8] Take workloadmgr Database Dump

Export workloadmgr database using following command from OpenStack’s galera cluster to a file

Create a directory to store .sql file using following command:

mkdir ~/trilio/

Get openstack-galera root password using following command:

oc -n openstack get secret osp-secret -o jsonpath='{.data.DbRootPassword}' | base64 -d

First, go inside the OpenStack Galera pod using the following command:

oc -n openstack rsh openstack-galera-0

Create a MySQL dump using the following command in the openstack-galera pod.

mysqldump -u root -p\
  --single-transaction \
  --no-create-info \
  --no-create-db \
  --skip-triggers \
  --skip-routines \
  --skip-events \
  --replace \
  workloadmgr > /tmp/t40-workloadmgr.sql

### Exit from pod

Copy the above .sql file to the following directory using the command below:

cd ~/trilio/
oc -n openstack cp openstack-galera-0:/tmp/t40-workloadmgr.sql ./t40-workloadmgr.sql

2.9] Upgrade T4O 6.1.7 deployment with new database approach

Update the tvo-operator-inputs.yaml file with the following changes to configure Trilio to use the new DB installation approach.

cd triliovault-cfg-scripts/redhat-director-scripts/rhosp18/ctlplane-scripts/
vi tvo-operator-inputs.yaml

  database:
    common:
      use_trilio_galera: true
      root_user_name: "root"
      host: "trilio-galera-cluster.trilio-openstack.svc"
      port: 3306

2.10] Run deploy command

cd triliovault-cfg-scripts/redhat-director-scripts/rhosp18/ctlplane-scripts/
./deploy_tvo_control_plane.sh

2.11] Verify successful deployment of T4O control plane services.

[openstackdev@localhost ctlplane-scripts]$ oc get pods
NAME                                                READY   STATUS      RESTARTS   AGE
job-triliovault-datamover-api-db-init-flcvl         0/1     Completed   0          7m58s
job-triliovault-datamover-api-keystone-init-rfzxb   0/1     Completed   0          8m34s
job-triliovault-datamover-api-rabbitmq-init-zff58   0/1     Completed   0          8m20s
job-triliovault-wlm-cloud-trust-r99fr               0/1     Completed   0          7m42s
job-triliovault-wlm-db-init-l84hl                   0/1     Completed   0          7m53s
job-triliovault-wlm-keystone-init-6tp6r             0/1     Completed   0          8m16s
job-triliovault-wlm-rabbitmq-init-9vwd8             0/1     Completed   0          8m2s
trilio-galera-cluster-galera-0                      1/1     Running     0          9m17s
trilio-galera-cluster-galera-1                      1/1     Running     0          9m17s
trilio-galera-cluster-galera-2                      1/1     Running     0          9m17s
trilio-galera-wait-job-l4klg                        0/1     Completed   0          9m17s
trilio-rabbitmq-cluster-server-0                    1/1     Running     0          10m
trilio-rabbitmq-cluster-server-1                    1/1     Running     0          10m
trilio-rabbitmq-cluster-server-2                    1/1     Running     0          10m
triliovault-datamover-api-7d6977869-7gtfr           1/1     Running     0          7m43s
triliovault-datamover-api-7d6977869-lnvjb           1/1     Running     0          7m43s
triliovault-datamover-api-7d6977869-zqhqx           1/1     Running     0          7m43s
triliovault-object-store-bt1-s3-27bxz               1/1     Running     0          7m43s
triliovault-object-store-bt1-s3-hfrhw               1/1     Running     0          7m43s
triliovault-object-store-bt1-s3-phr6n               1/1     Running     0          7m43s
triliovault-wlm-api-566d9778d-287xk                 1/1     Running     0          7m42s
triliovault-wlm-api-566d9778d-gzqq6                 1/1     Running     0          7m42s
triliovault-wlm-api-566d9778d-hsksj                 1/1     Running     0          7m42s
triliovault-wlm-cron-85b45d4447-97r84               1/1     Running     0          7m42s
triliovault-wlm-scheduler-7558f58dc9-2lfnt          1/1     Running     0          7m42s
triliovault-wlm-scheduler-7558f58dc9-8v6ts          1/1     Running     0          7m42s
triliovault-wlm-scheduler-7558f58dc9-pqwqk          1/1     Running     0          7m42s
triliovault-wlm-workloads-8468959778-6r2tp          1/1     Running     0          7m42s
triliovault-wlm-workloads-8468959778-c5pzz          1/1     Running     0          7m42s
triliovault-wlm-workloads-8468959778-dffcw          1/1     Running     0          7m42s

2.12] Import Old `workloadmgr` Database Data into New Galera

After successful installation, import the database dump from the SQL file created using the mysqldump command.

Copy the workloadmgr database dump file to the Trilio Galera pod.

cd ~/trilio/
oc -n trilio-openstack cp ./t40-workloadmgr.sql trilio-galera-cluster-galera-0:/tmp/t40-workloadmgr.sql

Get the Trilio Galera root password using the following command:

oc -n trilio-openstack get secret trilio-openstack-secret -o jsonpath='{.data.DbRootPassword}' | base64 -d

Import the workloadmgr database data from the .sql file.

First, go inside the Trilio Galera pod using the following command:

oc -n trilio-openstack rsh trilio-galera-cluster-galera-0

Check the size of the .sql file using the following command:

du -sh /tmp/t40-workloadmgr.sql

Import the .sql file into the workloadmgr database using the following command:

mysql -u root -p workloadmgr < /tmp/t40-workloadmgr.sql

2.13] Restart of wlm-cron

After importing the database dump, the wlm-cron pod must be restarted. Restarting is required for wlm-cron to pick up the newly imported, scheduler-enabled workloads.

In OpenShift, a pod restart is performed by deleting the pod. The controller will automatically recreate it.

oc -n trilio-openstack get pods
oc -n trilio-openstack delete pod <wlm-cron-pod>

3] Upgrade Trilio Data Plane Services

Follow step 3 to 3.8 of this documentation

4] Upgrade Trilio Horizon Plugin

Follow step 4 of this documentation and update the trilio horizon plugin with new image tag.

5] Upgrade dynamically added Backup Target

Note: This step is needed only when there is change in python3-s3fuse-plugin-el9 package.

Please follow below steps to upgrade s3 backup target pods to new build on RHOSO18 setup.

Copy the input files from old triliovault-cfg-scripts directory to latest directory.

cp triliovault-cfg-scripts-old/redhat-director-scripts/rhosp18/ctlplane-scripts/tvo-backup-target-cr-amazon-s3.yaml triliovault-cfg-scripts/redhat-director-scripts/rhosp18/ctlplane-scripts/
cp triliovault-cfg-scripts-old/redhat-director-scripts/rhosp18/ctlplane-scripts/tvo-backup-target-cr-other-s3.yaml triliovault-cfg-scripts/redhat-director-scripts/rhosp18/ctlplane-scripts/
cp triliovault-cfg-scripts-old/redhat-director-scripts/rhosp18/dataplane-scripts/<BACKUP-TARGET-NAME> triliovault-cfg-scripts/redhat-director-scripts/rhosp18/dataplane-scripts/

5.1] Upgrade dynamic backup target on control plane

⚠️ Important Upgrade Note for T4O 6.1.3 or Older

If you are on T4O 6.1.3 or older and have dynamic BTTs (TVOBackupTarget CRs), run the script below on your bastion node before upgrading to patch Helm annotations and labels:
```
cd triliovault-cfg-scripts/redhat-director-scripts/rhosp18/ctlplane-scripts/
chmod +x update_helm_annotations.sh
./update_helm_annotations.sh
```
The script updates all dynamic TVOBackupTarget resources with required Helm annotations and labels so they are compatible with the current release.
If a user has created dynamic BTTs using T4O 6.1.3 or older, this step is mandatory before upgrading.
Users on T4O 6.1.4 or newer with BTTs created on 6.1.4+ do not need to run this script as the fix is included in the operator.

Update the below parameter with new image tag in respective files tvo-backup-target-cr-amazon-s3.yaml and tvo-backup-target-cr-other-s3.yaml.

cd triliovault-cfg-scripts/redhat-director-scripts/rhosp18/ctlplane-scripts
vi tvo-backup-target-cr-amazon-s3.yaml
vi tvo-backup-target-cr-other-s3.yaml

  images:
    triliovault_object_store: "registry.connect.redhat.com/trilio/trilio-wlm:<NEW-BUILD-TAG>"

Now apply the changes.

oc -n trilio-openstack apply -f tvo-backup-target-cr-amazon-s3.yaml
oc -n trilio-openstack apply -f tvo-backup-target-cr-other-s3.yaml

Check the s3 backup target containers up and running with new image.

oc -n trilio-openstack get pods | grep <backup-target-name>
oc -n trilio-openstack describe pod <pod-name> | grep Image:

5.2] Upgrade dynamic backup target on data plane

Update the wlm image parameter with new image tag in respective files.

cd triliovault-cfg-scripts/redhat-director-scripts/rhosp18/dataplane-scripts
vi <BACKUP-TARGET-NAME>/cm-trilio-backup-target.yaml


  trilio_env.yml: |
    triliovault_wlm_image: "registry.connect.redhat.com/trilio/trilio-wlm:<NEW-BUILD-TAG>"

Now apply the changes.

oc -n openstack apply -f <BACKUP-TARGET-NAME>/cm-trilio-backup-target.yaml

Update the ansible runner image with new image tag.

vi <BACKUP-TARGET-NAME>/trilio-add-backup-target-service.yaml


  openStackAnsibleEERunnerImage: registry.connect.redhat.com/trilio/trilio-ansible-runner-rhoso:<NEW-BUILD-TAG>

Now apply the changes.

oc -n openstack apply -f <BACKUP-TARGET-NAME>/trilio-add-backup-target-service.yaml

Update the deployment name with unique name.

vi <BACKUP-TARGET-NAME>/trilio-add-backup-target-deployment.yaml


metadata:
  name: edpm-trilio-add-backup-target-s3-bt4

Now apply the changes.

oc -n openstack apply -f <BACKUP-TARGET-NAME>/trilio-add-backup-target-deployment.yaml

Verify the s3 backup target containers on compute nodes.

podman ps -a | grep <BACKUP-TARGET-NAME>

PreviousUpgrading on RHOSP NextUpgrading on OpenStack-Helm/MOSK

Last updated 4 days ago

Was this helpful?

hashtag1] Configuration change

hashtag2] Upgrade to new build

hashtag2.1] Create the trilio-openstack-secret in the trilio-openstack namespace.

hashtag2.2] Edit tvo-operator-inputs.yaml file

hashtag2.3] Upgrade T4O to latest T4O (at least to 6.1.7) with old db approach.

hashtag2.4] Create TLS certificate secrets

hashtag2.5] Run deploy command

hashtag2.6] Upgrade Trilio for OpenStack Operator

hashtag2.7] Verify successful deployment of T4O control plane services.

hashtag2.8] Take workloadmgr Database Dump

hashtag2.9] Upgrade T4O 6.1.7 deployment with new database approach

hashtag2.10] Run deploy command

hashtag2.11] Verify successful deployment of T4O control plane services.

hashtag2.12] Import Old workloadmgr Database Data into New Galera

hashtag2.13] Restart of wlm-cron

hashtag3] Upgrade Trilio Data Plane Services

hashtag4] Upgrade Trilio Horizon Plugin

hashtag5] Upgrade dynamically added Backup Target

hashtag5.1] Upgrade dynamic backup target on control plane

hashtag⚠️ Important Upgrade Note for T4O 6.1.3 or Older

hashtag5.2] Upgrade dynamic backup target on data plane

1] Configuration change

2] Upgrade to new build

2.1] Create the trilio-openstack-secret in the trilio-openstack namespace.

2.2] Edit tvo-operator-inputs.yaml file

2.3] Upgrade T4O to latest T4O (at least to 6.1.7) with old db approach.

2.4] Create TLS certificate secrets

2.5] Run deploy command

2.6] Upgrade Trilio for OpenStack Operator

2.7] Verify successful deployment of T4O control plane services.

2.8] Take workloadmgr Database Dump

2.9] Upgrade T4O 6.1.7 deployment with new database approach

2.10] Run deploy command

2.11] Verify successful deployment of T4O control plane services.

2.12] Import Old `workloadmgr` Database Data into New Galera

2.13] Restart of wlm-cron

3] Upgrade Trilio Data Plane Services

4] Upgrade Trilio Horizon Plugin

5] Upgrade dynamically added Backup Target

5.1] Upgrade dynamic backup target on control plane

⚠️ Important Upgrade Note for T4O 6.1.3 or Older

5.2] Upgrade dynamic backup target on data plane