# Upgrading on RHOSO ### 1] Configuration change If any config parameter changed in tvo-operator-inputs.yaml like db user password or service endpoints, you can apply the changes using following command. ``` cd ctlplane-scripts ./deploy_tvo_control_plane.sh ``` Above command will output ‘configured' or 'unchanged’ depending upon changes happened in tvo-operator-inputs.yaml. ### 2] Upgrade to new build Please ensure the following requirements are met before starting the upgrade process: * No Snapshot or Restore is running * Global job scheduler is disabled Please follow below steps to upgrade to new build on RHOSO18 setup. Take a backup of existing triliovault-cfg-scripts and clone latest triliovault-cfg-scripts github repository. ``` mv triliovault-cfg-scripts triliovault-cfg-scripts-old git clone -b {{ trilio_branch }} https://github.com/trilioData/triliovault-cfg-scripts.git ``` ### Upgrade Path 1: Upgrade T4O 6.1.7+ to 6.1.9 #### 2.1] Upgrade Trilio for OpenStack Operator Run operator deployment with new image tag as mentioned in [step 2.1 of this documentation](/openstack/deployment/installing-on-rhosp/trilio_installation_on_rhoso.md#install-operator-tvo-operator) #### 2.2] Edit tvo-operator-inputs.yaml file Follow [step 2.2 of this documentation](/openstack/deployment/installing-on-rhosp/trilio_installation_on_rhoso.md#edit-operator-inputs) #### 2.3] Run deploy command ``` cd triliovault-cfg-scripts/redhat-director-scripts/rhosp18/ctlplane-scripts/ ./deploy_tvo_control_plane.sh ``` #### 2.4] Verify successful deployment of T4O control plane services. ``` oc -n trilio-openstack get pods ``` ### 3] Upgrade Trilio Data Plane Services Follow [step 3.2 to 3.8 of this documentation](/openstack/deployment/installing-on-rhosp/trilio_installation_on_rhoso.md#review-trilio-data-plane-parameters) ### 4] Upgrade Trilio Horizon Plugin Follow [step 4 of this documentation](/openstack/deployment/installing-on-rhosp/trilio_installation_on_rhoso.md#id-4-install-trilio-horizon-plugin) and update the trilio horizon plugin with new image tag. ### 5] Upgrade dynamically added Backup Target Please refer to [step 5 of this documentation](#upgrade-dynamically-added-backup-target) ### Upgrade Path 2: Upgrade T4O from 6.1.6 or Earlier to 6.1.9 #### 2.1] Edit tvo-operator-inputs.yaml file Follow [step 2.2 of this documentation](/openstack/deployment/installing-on-rhosp/trilio_installation_on_rhoso.md#edit-operator-inputs) #### 2.2] Upgrade T4O to latest T4O (at least to 6.1.9) with old db approach. Update the following database parameters in tvo-operator-inputs.yaml to configure the old DB approach. ``` cd triliovault-cfg-scripts/redhat-director-scripts/rhosp18/ctlplane-scripts/ vi tvo-operator-inputs.yaml ``` ``` database: common: use_trilio_galera: false root_user_name: "root" host: "openstack.openstack.svc" port: 3306 ``` Save file changes. #### 2.3] Create TLS certificate secrets Follow [step 2.4 of this documentation](/openstack/deployment/installing-on-rhosp/trilio_installation_on_rhoso.md#create-tls-certificate-secrets) #### 2.4] Run deploy command ``` cd triliovault-cfg-scripts/redhat-director-scripts/rhosp18/ctlplane-scripts/ ./deploy_tvo_control_plane.sh ``` #### 2.5] Upgrade Trilio for OpenStack Operator ⚠️ **Important:** After running the above script, immediately upgrade the operator to version **6.1.9**. ``` chmod +x install_operator.sh ./install_operator.sh ``` #### 2.6] Verify successful deployment of T4O control plane services. ``` [openstackdev@localhost]$ oc get pods NAME READY STATUS RESTARTS AGE job-triliovault-datamover-api-db-init-kwt7q 0/1 Completed 0 8m43s job-triliovault-datamover-api-keystone-init-gggcm 0/1 Completed 0 9m30s job-triliovault-datamover-api-rabbitmq-init-mtck7 0/1 Completed 0 9m16s job-triliovault-wlm-cloud-trust-f6xt2 0/1 Completed 0 8m27s job-triliovault-wlm-db-init-c2tpm 0/1 Completed 0 8m38s job-triliovault-wlm-keystone-init-mw8q8 0/1 Completed 0 9m12s job-triliovault-wlm-rabbitmq-init-qwvgr 0/1 Completed 0 8m47s trilio-rabbitmq-cluster-server-0 1/1 Running 0 10m trilio-rabbitmq-cluster-server-1 1/1 Running 0 10m trilio-rabbitmq-cluster-server-2 1/1 Running 0 10m triliovault-datamover-api-7754567c94-57cqk 1/1 Running 0 8m27s triliovault-datamover-api-7754567c94-9q47t 1/1 Running 0 8m27s triliovault-datamover-api-7754567c94-rk6qc 1/1 Running 0 8m27s triliovault-object-store-bt1-s3-fk7qs 1/1 Running 0 8m28s triliovault-object-store-bt1-s3-lhvv7 1/1 Running 0 8m28s triliovault-object-store-bt1-s3-p85p7 1/1 Running 0 8m28s triliovault-wlm-api-85469585c7-8bbm2 1/1 Running 0 8m27s triliovault-wlm-api-85469585c7-bf8nx 1/1 Running 0 8m27s triliovault-wlm-api-85469585c7-z8q64 1/1 Running 0 8m27s triliovault-wlm-cron-796c4d4554-7tgfh 1/1 Running 0 8m27s triliovault-wlm-scheduler-65c6d68c94-8599n 1/1 Running 0 8m27s triliovault-wlm-scheduler-65c6d68c94-pqqqp 1/1 Running 0 8m27s triliovault-wlm-scheduler-65c6d68c94-rj6lb 1/1 Running 0 8m27s triliovault-wlm-workloads-6b7f65fc4f-97gvj 1/1 Running 0 8m27s triliovault-wlm-workloads-6b7f65fc4f-9mt7n 1/1 Running 0 8m27s triliovault-wlm-workloads-6b7f65fc4f-wrs8f 1/1 Running 0 8m27s ``` Follow the steps mentioned below only if you want to migrate from the old DB approach (`openstack-galera`) to the new DB approach (`trilio-galera`) . Otherwise, skip steps 2.8 to 2.13 to continue using the old `openstack-galera` DB approach. * Old DB approach: Trilio uses the DB service used by the OpenStack. * New DB approach: Trilio deploys and uses its own DB service. #### 2.7] Create the trilio-openstack-secret in the trilio-openstack namespace. * For the new database approach, the user needs to add DbRootPassword in Base64-encoded form to the trilio-openstack-secret.yaml file. Follow [step 1.4 of this documentation](/openstack/deployment/installing-on-rhosp/trilio_installation_on_rhoso.md#create-trilio-openstack-secret) #### 2.8] Take workloadmgr Database Dump Export `workloadmgr` database using following command from OpenStack’s galera cluster to a file Create a directory to store .sql file using following command: ``` mkdir ~/trilio/ ``` Get openstack-galera root password using following command: ``` oc -n openstack get secret osp-secret -o jsonpath='{.data.DbRootPassword}' | base64 -d ``` First, go inside the OpenStack Galera pod using the following command: ``` oc -n openstack rsh openstack-galera-0 ``` Create a MySQL dump using the following command in the openstack-galera pod. ``` mysqldump -u root -p\ --single-transaction \ --no-create-info \ --no-create-db \ --skip-triggers \ --skip-routines \ --skip-events \ --replace \ workloadmgr > /tmp/t4o-workloadmgr.sql ### Exit from pod ``` Copy the above `.sql` file to the following directory using the command below: ``` cd ~/trilio/ oc -n openstack cp openstack-galera-0:/tmp/t4o-workloadmgr.sql ./t4o-workloadmgr.sql ``` #### 2.9] Upgrade T4O 6.1.9 deployment with new database approach Update the tvo-operator-inputs.yaml file with the following changes to configure Trilio to use the new DB installation approach. ``` cd triliovault-cfg-scripts/redhat-director-scripts/rhosp18/ctlplane-scripts/ vi tvo-operator-inputs.yaml ``` ``` database: common: use_trilio_galera: true root_user_name: "root" host: "trilio-galera-cluster.trilio-openstack.svc" port: 3306 ``` #### 2.10] Run deploy command ``` cd triliovault-cfg-scripts/redhat-director-scripts/rhosp18/ctlplane-scripts/ ./deploy_tvo_control_plane.sh ``` #### 2.11] Verify successful deployment of T4O control plane services. ``` [openstackdev@localhost ctlplane-scripts]$ oc get pods NAME READY STATUS RESTARTS AGE job-triliovault-datamover-api-db-init-flcvl 0/1 Completed 0 7m58s job-triliovault-datamover-api-keystone-init-rfzxb 0/1 Completed 0 8m34s job-triliovault-datamover-api-rabbitmq-init-zff58 0/1 Completed 0 8m20s job-triliovault-wlm-cloud-trust-r99fr 0/1 Completed 0 7m42s job-triliovault-wlm-db-init-l84hl 0/1 Completed 0 7m53s job-triliovault-wlm-keystone-init-6tp6r 0/1 Completed 0 8m16s job-triliovault-wlm-rabbitmq-init-9vwd8 0/1 Completed 0 8m2s trilio-galera-cluster-galera-0 1/1 Running 0 9m17s trilio-galera-cluster-galera-1 1/1 Running 0 9m17s trilio-galera-cluster-galera-2 1/1 Running 0 9m17s trilio-galera-wait-job-l4klg 0/1 Completed 0 9m17s trilio-rabbitmq-cluster-server-0 1/1 Running 0 10m trilio-rabbitmq-cluster-server-1 1/1 Running 0 10m trilio-rabbitmq-cluster-server-2 1/1 Running 0 10m triliovault-datamover-api-7d6977869-7gtfr 1/1 Running 0 7m43s triliovault-datamover-api-7d6977869-lnvjb 1/1 Running 0 7m43s triliovault-datamover-api-7d6977869-zqhqx 1/1 Running 0 7m43s triliovault-object-store-bt1-s3-27bxz 1/1 Running 0 7m43s triliovault-object-store-bt1-s3-hfrhw 1/1 Running 0 7m43s triliovault-object-store-bt1-s3-phr6n 1/1 Running 0 7m43s triliovault-wlm-api-566d9778d-287xk 1/1 Running 0 7m42s triliovault-wlm-api-566d9778d-gzqq6 1/1 Running 0 7m42s triliovault-wlm-api-566d9778d-hsksj 1/1 Running 0 7m42s triliovault-wlm-cron-85b45d4447-97r84 1/1 Running 0 7m42s triliovault-wlm-scheduler-7558f58dc9-2lfnt 1/1 Running 0 7m42s triliovault-wlm-scheduler-7558f58dc9-8v6ts 1/1 Running 0 7m42s triliovault-wlm-scheduler-7558f58dc9-pqwqk 1/1 Running 0 7m42s triliovault-wlm-workloads-8468959778-6r2tp 1/1 Running 0 7m42s triliovault-wlm-workloads-8468959778-c5pzz 1/1 Running 0 7m42s triliovault-wlm-workloads-8468959778-dffcw 1/1 Running 0 7m42s ``` #### 2.12] Import Old `workloadmgr` Database Data into New Galera After successful installation, import the database dump from the SQL file created using the `mysqldump` command. Copy the `workloadmgr` database dump file to the Trilio Galera pod. ``` cd ~/trilio/ oc -n trilio-openstack cp ./t4o-workloadmgr.sql trilio-galera-cluster-galera-0:/tmp/t4o-workloadmgr.sql ``` Get the Trilio Galera root password using the following command: ``` oc -n trilio-openstack get secret trilio-openstack-secret -o jsonpath='{.data.DbRootPassword}' | base64 -d ``` Import the `workloadmgr` database data from the `.sql` file. First, go inside the Trilio Galera pod using the following command: ``` oc -n trilio-openstack rsh trilio-galera-cluster-galera-0 ``` Check the size of the `.sql` file using the following command: ``` du -sh /tmp/t4o-workloadmgr.sql ``` Import the `.sql` file into the `workloadmgr` database using the following command: ``` mysql -u root -p workloadmgr < /tmp/t4o-workloadmgr.sql ``` #### 2.13] Restart of wlm-cron After importing the database dump, the `wlm-cron` pod must be restarted. Restarting is required for `wlm-cron` to pick up the newly imported, scheduler-enabled workloads. In OpenShift, a pod restart is performed by deleting the pod. The controller will automatically recreate it. ``` oc -n trilio-openstack get pods oc -n trilio-openstack delete pod ``` ### 3] Upgrade Trilio Data Plane Services Follow [step 3 to 3.8 of this documentation](/openstack/deployment/installing-on-rhosp/trilio_installation_on_rhoso.md#install-trilio-data-plane-services) ### 4] Upgrade Trilio Horizon Plugin Follow [step 4 of this documentation](/openstack/deployment/installing-on-rhosp/trilio_installation_on_rhoso.md#id-4-install-trilio-horizon-plugin) and update the trilio horizon plugin with new image tag. ### 5] Upgrade dynamically added Backup Target Note: This step is needed only when there is change in python3-s3fuse-plugin-el9 package. Please follow below steps to upgrade s3 backup target pods to new build on RHOSO18 setup. Copy the input files from old triliovault-cfg-scripts directory to latest directory. ``` cp triliovault-cfg-scripts-old/redhat-director-scripts/rhosp18/ctlplane-scripts/tvo-backup-target-cr-amazon-s3.yaml triliovault-cfg-scripts/redhat-director-scripts/rhosp18/ctlplane-scripts/ cp triliovault-cfg-scripts-old/redhat-director-scripts/rhosp18/ctlplane-scripts/tvo-backup-target-cr-other-s3.yaml triliovault-cfg-scripts/redhat-director-scripts/rhosp18/ctlplane-scripts/ cp triliovault-cfg-scripts-old/redhat-director-scripts/rhosp18/dataplane-scripts/ triliovault-cfg-scripts/redhat-director-scripts/rhosp18/dataplane-scripts/ ``` #### 5.1] Upgrade dynamic backup target on control plane #### ⚠️ Important Upgrade Note for T4O 6.1.3 or Older * If you are on T4O 6.1.3 or older and have dynamic BTTs (`TVOBackupTarget` CRs), **run the script below on your bastion node before upgrading** to patch Helm annotations and labels: ``` cd triliovault-cfg-scripts/redhat-director-scripts/rhosp18/ctlplane-scripts/ chmod +x update_helm_annotations.sh ./update_helm_annotations.sh ``` * The script updates all dynamic TVOBackupTarget resources with required Helm annotations and labels so they are compatible with the current release. * If a user has created dynamic BTTs using T4O 6.1.3 or older, this step is mandatory before upgrading. * Users on T4O 6.1.4 or newer with BTTs created on 6.1.4+ do **not** need to run this script as the fix is included in the operator. Update the below parameter with new image tag in respective files tvo-backup-target-cr-amazon-s3.yaml and tvo-backup-target-cr-other-s3.yaml. ``` cd triliovault-cfg-scripts/redhat-director-scripts/rhosp18/ctlplane-scripts vi tvo-backup-target-cr-amazon-s3.yaml vi tvo-backup-target-cr-other-s3.yaml ``` ``` images: triliovault_object_store: "registry.connect.redhat.com/trilio/trilio-wlm:" ``` Now apply the changes. ``` oc -n trilio-openstack apply -f tvo-backup-target-cr-amazon-s3.yaml oc -n trilio-openstack apply -f tvo-backup-target-cr-other-s3.yaml ``` Check the s3 backup target containers up and running with new image. ``` oc -n trilio-openstack get pods | grep oc -n trilio-openstack describe pod | grep Image: ``` #### 5.2] Upgrade dynamic backup target on data plane Update the wlm image parameter with new image tag in respective files. ``` cd triliovault-cfg-scripts/redhat-director-scripts/rhosp18/dataplane-scripts vi /cm-trilio-backup-target.yaml trilio_env.yml: | triliovault_wlm_image: "registry.connect.redhat.com/trilio/trilio-wlm:" ``` Now apply the changes. ``` oc -n openstack apply -f /cm-trilio-backup-target.yaml ``` Update the ansible runner image with new image tag. ``` vi /trilio-add-backup-target-service.yaml openStackAnsibleEERunnerImage: registry.connect.redhat.com/trilio/trilio-ansible-runner-rhoso: ``` Now apply the changes. ``` oc -n openstack apply -f /trilio-add-backup-target-service.yaml ``` Update the deployment name with unique name. ``` vi /trilio-add-backup-target-deployment.yaml metadata: name: edpm-trilio-add-backup-target-s3-bt4 ``` Now apply the changes. ``` oc -n openstack apply -f /trilio-add-backup-target-deployment.yaml ``` Verify the s3 backup target containers on compute nodes. ``` podman ps -a | grep ``` --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.trilio.io/openstack/upgrading-to-t4o-6.x-from-older-supported-versions/upgradetrilio/upgrade_on_rhoso18.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.