Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Upgrading on RHOSO

Upgrading from 6.1.x to 6.2.x

Trilio for OpenStack 6.2.x replaces the per-backup-target object-store containers with the Dynamic Mount Service (DMS) — a single server container per host. Upgrading from 6.1.x therefore requires more than an image/package bump; the per-BT containers must be removed before the DMS containers are deployed.

The high-level flow is:

  1. Tear down the 6.1 per-BT pods/containers first. Before deploying any 6.2.x image, remove the existing triliovault-object-store-<bt-name> pods on the control plane and the matching per-BT backup-target containers on each compute node. They are replaced by a single trilio-dms-server per host and must not coexist with it.

  2. Migrate existing backup targets to the 6.2.x schema. A migration script generates the Barbican secrets for each S3 backup target (using the S3 credentials currently stored in the WLM config), updates each backup target row with the resulting secret_ref, and adds the backup_target_mount_ledger table on both the workloadmgr and dmapi schemas.

  3. Deploy the 6.2.x control plane and data plane. Run the upgrade playbook targeted at the 6.2.x triliovault-cfg-scripts branch. It rolls out the 6.2.x control-plane images (wlm, dmapi, horizon-plugin), deploys trilio-dms-server on the controller and on every compute node that needs backup-target access, and lays down /etc/triliovault-dms/{server,client}.conf plus the rootwrap files.

  4. Verify. After the upgrade, confirm that:

    • trilio-dms-server pods are running in the trilio-openstack namespace, one per controller, and the matching trilio-dms-server containers are running on each compute node;

    • no triliovault-object-store-<bt-name> pods or per-BT backup-target containers remain;

    • each existing backup target lists a non-empty secret_ref (for S3) and reaches online status;

    • a fresh snapshot and restore against each backup target succeed end-to-end.

Existing backup targets, backup-target types, workloads, snapshots, and policies are preserved across the upgrade; no workload data is migrated.

The general upgrade flow (control-plane refresh, data-plane refresh, Horizon plugin update, dynamic backup-target refresh) below remains the same as for 6.1.x upgrades.

1] Configuration change

If any config parameter is changed in tvo-operator-inputs.yaml, like DB user password or service endpoints, you can apply the changes using the following command.

Above command will output ‘configured' or 'unchanged’ depending upon changes happened in tvo-operator-inputs.yaml.

2] Upgrade to New Build

Please ensure the following requirements are met before starting the upgrade process:

  • No Snapshot or Restore is running

  • Global job scheduler is disabled

  • T4O should have been upgraded to the 6.1.7 or higher release.

Please follow the steps below to upgrade to the new build on the RHOSO18 setup.

Take a backup of existing triliovault-cfg-scripts and clone latest triliovault-cfg-scripts github repository.

In T4O 6.2.x, pre-mounted backup targets are not required. The T4O core code mounts the backup target when a backup is triggered. Therefore, existing backup targets must be unmounted from the cloud and related pods/containers removed.

Choose Upgrade Path

Select the appropriate upgrade path based on your current T4O version.

Current T4O Version
Upgrade Path

6.1.7+

3] Upgrade Path 1: Upgrade T4O 6.1.7+ to 6.2.1

6.2.0

4] Upgrade Path 2: Upgrade T4O 6.2.0 to 6.2.1

3] Upgrade Path 1: Upgrade T4O 6.1.7+ to 6.2.1

3.1] Collect Existing Backup Targets

Before upgrading to T4O 6.2.x, collect all backup target configurations from the current T4O 6.1.7+ installation. This script reads backup targets from the old tvo-operator-inputs.yaml and from any dynamic TVOBackupTarget CRs in the trilio-openstack namespace, writes a combined inventory YAML file, and verifies that the inventory matches workloadmgr backup-target-list exactly (count and names).

What the script does:

  • Reads backup_target / backup_targets section from tvo-operator-inputs.yaml (if present)

  • Reads all TVOBackupTarget CR instances from the trilio-openstack namespace

  • Merges both sources into existing_list_of_backup_targets.yaml

  • Verifies inventory count and names match workloadmgr backup-target-list and reports any mismatches

Prerequisites:

  • oc CLI logged in with cluster-admin (or appropriate RBAC permissions)

  • python3 with PyYAML (pip3 install pyyaml)

  • jq command

The T4O-6.1.7+ tvo-operator-inputs.yaml file used for installation should be present in the old cfg directory.

Run the script and provide the T4O 6.1.7+ tvo-operator-inputs.yaml file path as an argument:

This produces existing_list_of_backup_targets.yaml file.

Open the file and verify it contains all backup targets from the current T4O installation, especially S3 credentials:

3.2] Upgrade T4O to 6.2.1 Operator and Control Plane

3.2.1] Follow step 1.4 of this documentation

3.2.2] Follow step 2.1 of this documentation

3.2.3] Follow step 2.2 of this documentation

3.2.4] Patch Helm Ownership Metadata

Before deploying the control plane, run the following script to add the required Helm ownership metadata to the existing Galera and RabbitMQ resources.

In earlier T4O releases, the Galera and RabbitMQ resources were created without the Helm ownership metadata required for the 6.2.1 upgrade. This script updates the existing resources with the required ownership metadata so Helm can manage them during the control plane upgrade. Skipping this step may cause the upgrade to fail.

3.2.5] Run Deploy Command

3.2.6] Verify Successful Deployment of T4O Control Plane Services

Expected: All required services should be in Running or Completed state.

Note: The job-triliovault-wlm-cloud-trust pod may remain in the Running state during the upgrade. Do not wait for this job to complete before proceeding with the remaining upgrade steps. It will automatically transition to the Completed state once the migration script finishes updating the required S3 secret references.

3.3] Run T4O Backup Target Uninstall and Unmount for Control Plane

Verify:

  1. No orphaned object store or NFS DaemonSets remain:

Expected: No output

  1. No orphaned object store Kubernetes resources remain:

Expected: "Not found" error for each command.

  1. No hung mounts on control plane nodes (repeat for each node labeled trilio-control-plane=enabled):

Expected: No output

  1. No stale empty directories under vault_data_dir (repeat for each control plane node):

Expected: No output. If stale empty directories are present under /var/lib/trilio/triliovault-mounts/ but no active mount points are listed, they can be safely ignored.

  1. Verify T4O control plane pods are in healthy state:

Expected: All required services should be in Running or Completed state.

3.4] Upgrade T4O Data Plane to 6.2.1 Release

Follow step 3 to 3.8 of this documentation

3.5] Run Backup Target Uninstall and Unmount on Data Plane Nodes

This step will unmount backup targets mounted as part of T4O 6.1.7+. It will also remove object store and NFS containers and their config files from compute nodes.

  • Set the correct list of data plane node set names (you need to run this for all data plane node sets). You can copy these from the data plane deployment templates or get them using the following command:

  • Save the changes.

Run:

Verify:

  1. Check the OpenStackDataPlaneDeployment completed successfully:

Expected: Ready = True (Confirms the cleanup job has successfully completed on data plane nodes)

  1. Check Ansible runner job logs for success:

Expected: failed=0, unreachable=0 (Ensures all tasks executed successfully without errors on any host)

  1. No old object store or NFS containers on compute nodes (SSH into a compute node and run):

Expected: no output (Confirms old backup target containers are removed)

  1. No hung mounts on compute nodes (SSH into a compute node and run):

Expected: no output (Ensures no stale backup target mounts remain)

  1. No stale empty directories under vault_data_dir (SSH into a compute node and run):

Expected: empty (Verifies cleanup of leftover mount directories)

  1. Datamover service is healthy on compute nodes (SSH into a compute node and run):

Expected: active (running confirms datamover service is running properly after cleanup)

  1. Verify that only the required Trilio containers are running on the compute node.

Expected: Only triliovault-datamover and triliovault-dms containers should be listed.

3.6] Upgrade Trilio Horizon Plugin

Follow step 4 of this documentation and update the Horizon plugin with the new image tag.

3.7] Update T4O 6.1.7+ Backup Targets to T4O 6.2.1

Note: This step is required only if your deployment includes one or more S3 backup targets. If your deployment uses only NFS backup targets, you can skip this step.

After cleanup is complete, migrate the existing backup targets using the inventory file collected in Step 3 of this documentation.

In T4O 6.2.x, S3 credentials are no longer stored in Kubernetes secrets — they are stored in OpenStack Barbican. The migration script handles this automatically for each S3 backup target:

  • Runs trilio-dms-cli secret-payload create to build the Barbican secret JSON

  • Validates the JSON with trilio-dms-cli secret-payload validate

  • Stores the secret in Barbican via openstack secret store

  • Updates the existing backup target in WLM using workloadmgr backup-target-modify

Note: NFS backup targets are listed but skipped — they do not require a Barbican secret.

All operations run inside a temporary migration pod launched from the trilio-wlm image.

This pod:

  • Includes workloadmgr, trilio-dms-cli, and the openstack CLI

  • Mounts the triliovault-cloudrc ConfigMap

  • Injects OS_PASSWORD from openstack-secret

No manual credential setup is required.

Run the migration:

Verify:

  1. List backup targets via CLI:

Expected: All previously configured backup targets are listed with an online status.

  1. Verify Barbican secrets were created (one per S3 backup target):

Expected: One Barbican secret named secret-key-<backup-target-name> per S3 backup target.

Cleanup:

After successful migration, you can remove the temporary inventory file.

3.8] Install a new Backup Target

To add a new backup target, refer to the Adding a New Backup Target on RHOSO18.

4] Upgrade Path 2: Upgrade T4O 6.2.0 to 6.2.1

4.1] Upgrade T4O to 6.2.1 Operator and Control Plane

4.1.1] Follow step 2.1 of this documentation

4.1.2] Follow step 2.2 of this documentation

4.1.3] Patch Helm Ownership Metadata

Before deploying the control plane, run the following script to add the required Helm ownership metadata to the existing Galera and RabbitMQ resources.

In earlier T4O releases, the Galera and RabbitMQ resources were created without the Helm ownership metadata required for the 6.2.1 upgrade. This script updates the existing resources with the required ownership metadata so Helm can manage them during the control plane upgrade. Skipping this step may cause the upgrade to fail.

4.1.4] Run Deploy Command

4.1.5] Verify successful deployment of T4O control plane services

Expected: All required services should be in Running or Completed state.

4.2] Upgrade T4O Data Plane to 6.2.1 Release

Follow steps 3.2 to 3.8 of this documentation

4.3] Upgrade Trilio Horizon Plugin

Follow step 4 of this documentation and update the Horizon plugin with the new image tag.

4.4] Install a new Backup Target

To add a new backup target, refer to the Adding a New Backup Target on RHOSO18.

PreviousUpgrading on RHOSP
NextUpgrading on OpenStack-Helm/MOSK

Last updated

Was this helpful?