> For the complete documentation index, see [llms.txt](https://docs.trilio.io/openstack/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.trilio.io/openstack/upgrading-to-t4o-6.x-from-older-supported-versions/upgradetrilio/upgrading-on-canonical.md). # Upgrading on Canonical This page covers upgrading Trilio charms on a Canonical (Charmed) OpenStack cloud from an existing release (5.x, 6.0, 6.1, or 6.2.0) to T4O 6.2.1. {% hint style="info" %} This page is for upgrading an **existing** Trilio deployment. If you are deploying T4O 6.2.1 fresh onto a Canonical OpenStack cloud that has no prior Trilio installation, see [Getting started with Trilio on Canonical OpenStack](/openstack/deployment/installing-on-canonical.md) instead. {% endhint %} ## What Changed in 6.2 The following features have been removed in T4O 6.2 for Canonical OpenStack charms: * **Static backup targets** — The `trilio-backup-targets` charm config option (present on both `trilio-wlm` and `trilio-data-mover`) is removed. NFS and S3 backup targets are no longer configured via charm config. * **tvault-object-store service** — The service that managed NFS/FUSE mount lifecycle is removed. The 6.2 charm automatically disables and stops it after package installation if the unit file is present. * **Object-store config templates** — `triliovault-object-store.conf`, `object_store_logging.conf`, and `tvault-object-store.service` are removed from all charms. These changes are part of the [Dynamic Mount Service (DMS)](/openstack/about-trilio-for-openstack/triliovault-for-openstack-architecture.md#dynamic-mount-service-dms), introduced in 6.2, which replaces the per-backup-target object-store containers with a single mount-management service per host. ## Juju Version Note Some commands differ between Juju 2.x and Juju 3.x. Both variants are shown throughout this page where applicable. | Operation | Juju 2.x | Juju 3.x | | ------------- | -------------------- | ---------------- | | Upgrade charm | `juju upgrade-charm` | `juju refresh` | | Add relation | `juju add-relation` | `juju integrate` | | Run an action | `juju run-action` | `juju run` | ### Action syntax — applies to every action on this page Two constraints catch people out when running actions across all units: * **There is no** `/*` wildcard. Both `juju run` (3.x) and `juju run-action` (2.x) accept only a specific unit ID (`trilio-data-mover/19`) or leader syntax (`trilio-data-mover/leader`). Passing `trilio-data-mover/*` fails with `ERROR invalid unit or action name "trilio-data-mover/*"`. `/leader` is not a substitute where an action must run on *every* unit. * `--wait` requires a duration unit. `--wait=300` fails with `ERROR invalid value "300" for option --wait: time: missing unit in duration "300"`. Use `--wait=300s`. Because of the first constraint, this page uses the `trilio_units` helper below to enumerate units from `juju status`. Define it once in your shell before running Steps 4 and 7: ``` # Prints every unit of the given Trilio applications, including subordinates. # trilio-data-mover and trilio-horizon-plugin are subordinate charms, so their # units appear under the principal's "subordinates" — not under the # application's own "units" — and a naive loop over applications misses them. # To run, just paste the below code in any shell session from where the juju commands will be executed. trilio_units() { juju status --format=json | python3 -c " import sys, json d = json.load(sys.stdin) prefixes = tuple(a + '/' for a in sys.argv[1:]) found = [] for app in d['applications'].values(): for name, unit in app.get('units', {}).items(): if name.startswith(prefixes): found.append(name) for sub in unit.get('subordinates', {}): if sub.startswith(prefixes): found.append(sub) for u in sorted(found): print(u) " "$@" } ``` Verify it before relying on it — it should list every Trilio unit in your model: ``` #Usage trilio_units trilio-wlm trilio-dm-api trilio-data-mover trilio-horizon-plugin ``` ## Previous Release Bundle Structure (Reference) In releases prior to 6.2 (5.x, 6.0, 6.1), backup targets were defined in the overlay bundle under both `trilio-wlm` and `trilio-data-mover` options as a JSON list. Example: ``` applications: trilio-data-mover: options: trilio-backup-targets: > [ { "backup-target-name": "S3_BT1", "backup-target-type": "s3", "s3-type": "amazon_s3", ... }, { "backup-target-name": "NFS_BT2", "backup-target-type": "nfs", "nfs-shares": "192.2.1.33:/var/nfsshare1", "nfs-options": "nolock,soft,timeo=600,..." } ] trilio-wlm: options: trilio-backup-targets: > [ ... same targets ... ] ``` These targets were mounted under `/var/triliovault-mounts/` on all compute nodes (data-mover) and WLM nodes — NFS targets as NFS mounts, S3 targets as `s3vaultfuse` FUSE mounts. ## Prerequisites Ensure no snapshot or restore is running, and disable the Global Job Scheduler, before starting the upgrade. ## 1] Prepare the 6.2.1 Overlay Bundle **Take a backup of your existing overlay bundle before making any changes.** You will need it in Step 8 to recreate backup targets in 6.2 — the script reads `trilio-backup-targets` directly from the file. ``` cp tvo-overlay-bundle-.yaml tvo-overlay-bundle--backup.yaml ``` Do **not** modify the old bundle. Instead, create a new 6.2 overlay bundle based on the updated sample provided in the `triliovault-cfg-scripts` repository: ``` juju-charms/sample_overlay_bundles/tvo-overlay-bundle.yaml ``` This sample bundle already has the following 6.2 changes applied: * `trilio-backup-targets` removed from both `trilio-wlm` and `trilio-data-mover` * `backup-target-type` removed * `trilio-data-mover:identity-service` → `keystone:identity-service` relation added (new in **6.2.0**) — required for DMS server configuration * Charm channel updated to `6.2/stable` Copy the sample and customise it for your environment: ``` cp juju-charms/sample_overlay_bundles/tvo-overlay-bundle.yaml tvo-overlay-bundle-6.2.yaml # Edit tvo-overlay-bundle-6.2.yaml — update triliovault-pkg-source and revision numbers ``` {% hint style="info" %} **Upgrading:** `juju refresh` upgrades charm code in-place — existing units stay on their current machines. Machine placements in the new bundle are **not** used during an upgrade and do not need to be updated. Machine placements only matter for a fresh install — see [Getting started with Trilio on Canonical OpenStack](/openstack/deployment/installing-on-canonical.md). {% endhint %} **This new bundle file is NOT deployed as part of this upgrade.** The charm upgrade itself happens entirely via the per-application `juju refresh`/`juju upgrade-charm` commands in Step 2 (plus the package source change in Step 3 and the relation add in Step 5) — there is no `juju deploy --overlay` step in the upgrade flow. Preparing this bundle here serves two purposes only: (1) its `triliovault-pkg-source` and `revision` variables are the reference you copy into the Step 2 and Step 3 commands, and (2) it becomes your up-to-date bundle for any future fresh installs or disaster-recovery redeploys matching this exact release — it is not consumed by any command in this upgrade flow. ## 2] Upgrade the Charms Refresh each Trilio charm from Charmhub using the `6.2/stable` channel and the revision numbers from the 6.2.1 release build resources: **Juju 3.x:** ``` juju refresh trilio-wlm --channel 6.2/stable --revision juju refresh trilio-data-mover --channel 6.2/stable --revision juju refresh trilio-dm-api --channel 6.2/stable --revision juju refresh trilio-horizon-plugin --channel 6.2/stable --revision ``` **Juju 2.x:** ``` juju upgrade-charm trilio-wlm --channel 6.2/stable --revision juju upgrade-charm trilio-data-mover --channel 6.2/stable --revision juju upgrade-charm trilio-dm-api --channel 6.2/stable --revision juju upgrade-charm trilio-horizon-plugin --channel 6.2/stable --revision ``` Revision numbers are published in the build resources for each release. The 6.2 sample overlay bundle (`tvo-overlay-bundle.yaml`) includes revision variables at the top — update those values from the build resources; that bundle is what future fresh installs deploy with `juju deploy ./openstack_base_file.yaml --overlay tvo-overlay-bundle-6.2.yaml`. For upgrades, use the per-application commands above. The 6.2 charm automatically stops and disables the `tvault-object-store` service on each unit after package installation if the service unit file is present. Some units may report **blocked** or **waiting** immediately after the refresh — this is expected until Steps 3 and 4 have been completed. ## 3] Update the Trilio Package Source Each Trilio release publishes its packages to its own APT repository, so after a charm refresh the `triliovault-pkg-source` config option still points at the repository of the release you are upgrading **from**. Until it is updated, the charms keep installing packages from the old repository and the new release's packages are never fetched. **Required** when upgrading from 5.x, 6.0, or 6.1 — the package repository changes with the release. **Skip** only when the source is genuinely unchanged (for example a 6.2.X → 6.2.Y charm-only refresh on the same repository). Example — the repository changes between 6.1 and 6.2: | Release | `triliovault-pkg-source` | | ------- | ----------------------------------------------------- | | 6.1 | `deb [trusted=yes] https://apt.fury.io/trilio-6-1/ /` | | 6.2.1 | `deb [trusted=yes] https://apt.fury.io/trilio-6-2 /` | ### 3.1 — Confirm Step 2 has fully settled first **Do not change the package source until every unit is running the new charm revision and is in** `idle` agent state. The package source change is applied by whichever charm code is active at that moment. If it is applied while the previous release's charm is still running, that older charm consumes the change and installs its own package list; the new charm revision then sees the source as already up to date, skips installation, and any package introduced in the new release is never installed — leaving units **blocked** on a service that was never deployed. ``` juju status trilio-wlm trilio-data-mover trilio-dm-api trilio-horizon-plugin ``` Each application must report the charm revision you set in Step 2, and every unit must show agent status **idle**. ### 3.2 — Set the new package source on all four charms Use the same `triliovault-pkg-source` value as your prepared 6.2 overlay bundle (Step 1), so the bundle and the running deployment stay in sync: ``` PKG_SOURCE='deb [trusted=yes] https://apt.fury.io/trilio-6-2 /' juju config trilio-wlm triliovault-pkg-source="$PKG_SOURCE" juju config trilio-data-mover triliovault-pkg-source="$PKG_SOURCE" juju config trilio-dm-api triliovault-pkg-source="$PKG_SOURCE" juju config trilio-horizon-plugin triliovault-pkg-source="$PKG_SOURCE" ``` Changing this value triggers a package install on each unit automatically. Wait for all units to return to **idle** before continuing: ``` watch -n 5 juju status trilio-wlm trilio-data-mover trilio-dm-api trilio-horizon-plugin ``` ### 3.3 — Verify the source was applied for all applications ``` juju config trilio-data-mover triliovault-pkg-source juju config trilio-dm-api triliovault-pkg-source juju config trilio-wlm triliovault-pkg-source juju config trilio-horizon-plugin triliovault-pkg-source ``` ## 4] Run Post-Upgrade Package and Database Migration (update-trilio Action) **Required — run this step in all cases, including when Step 3 already changed the package source.** A plain `juju refresh` / `juju upgrade-charm` only swaps the charm hook code — it does not reinstall the Trilio apt packages or apply any pending database migrations on its own. The `update-trilio` action forces the package upgrade and applies pending DB migrations. It is idempotent and safe to re-run. The action must run on **every unit** of each application, not just the leader — packages are installed per unit. Define the `trilio_units` helper from the **Action syntax** section above first, then run the loop below. It preserves the application order (WLM first, so its DB migrations land before the other services reconnect) and prints each unit's result. **Juju 3.x:** ``` for app in trilio-wlm trilio-dm-api trilio-data-mover trilio-horizon-plugin; do for unit in $(trilio_units $app); do echo "=== $unit ===" juju run "$unit" update-trilio --wait=300s --format=yaml done done ``` **Juju 2.x:** ``` for app in trilio-wlm trilio-dm-api trilio-data-mover trilio-horizon-plugin; do for unit in $(trilio_units $app); do echo "=== $unit ===" juju run-action "$unit" update-trilio --wait=300s --format=yaml done done ``` **Check the** `status:` field in every unit's output before continuing — it must read `completed`. A unit that was skipped or that failed will not have had its packages upgraded, and the failure is easy to miss in a long loop. Verify the package versions updated on each application: ``` #Examples juju ssh trilio-wlm/0 "dpkg -l workloadmgr | tail -1" juju exec --application trilio-data-mover "dpkg -l python3-tvault-contego python3-trilio-dms | tail -2" ``` Every listed package must show status `ii` (installed) and the new release version. A package shown as `rc`, or missing entirely, means the upgrade did not reach that unit — re-run `update-trilio` on it before continuing. ## 5] Add the New Relation T4O 6.2.0 introduced one Juju relation that does not exist in 6.1 or earlier, and it must be added explicitly during an upgrade — the overlay bundle is not deployed, so relations declared in it are never created. | Relation | Introduced in | Purpose | Required when upgrading from | | ------------------------------------------------------------------ | ------------- | -------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- | | `trilio-data-mover:identity-service` ↔ `keystone:identity-service` | 6.2.0 | Supplies the Keystone URL used to render the DMS server config on compute nodes (replaces the `keystone-auth-url` config option) | 5.x, 6.0, 6.1 | Adding a relation that is already present is harmless — Juju reports it and makes no change. **Juju 3.x:** ``` juju integrate trilio-data-mover:identity-service keystone:identity-service ``` **Juju 2.x:** ``` juju add-relation trilio-data-mover:identity-service keystone:identity-service ``` Verify the relation is present before continuing: ``` juju status --relations | grep 'trilio-data-mover:identity-service' ``` Then confirm the DMS client on the DMAPI node can actually authenticate against the workloadmgr database. Checking only that `db_url` is populated is not sufficient — the config can render a perfectly valid-looking URL whose credential has no matching MySQL grant, which is exactly how TVAULT-7592 reached QA: ``` cat > /tmp/dms_db_check.py <<'EOF' import configparser, pymysql, urllib.parse as up c = configparser.ConfigParser(); c.read("/etc/triliovault-dms/client.conf") u = up.urlparse(c["client"]["db_url"]) if not u.hostname: raise SystemExit("FAILED: db_url is empty in client.conf") pymysql.connect(host=u.hostname, port=u.port or 3306, user=u.username, password=up.unquote(u.password), database=u.path.lstrip("/")) print("OK: DMS client authenticated as", u.username) EOF juju scp /tmp/dms_db_check.py trilio-dm-api/leader:/tmp/dms_db_check.py juju ssh trilio-dm-api/leader "sudo python3 /tmp/dms_db_check.py; rm -f /tmp/dms_db_check.py" ``` The output must be `OK: DMS client authenticated as workloadmgr`. If it fails with `Access denied for user 'workloadmgr'@`, the DMAPI host has no grant on the workloadmgr database. Confirm with `SELECT user,host FROM mysql.user WHERE user='workloadmgr';` on the `mysql-innodb-cluster` primary — there must be a row for the `trilio-dm-api` unit's address as well as the `trilio-wlm` one. A blank `db_url` instead means the charm has not yet been answered on `shared-db`; wait for the unit to leave `blocked` and re-check. ## 6] Wait for All Units to Settle Wait until all Trilio units are in **active/idle** state before proceeding. Do not run the unmount steps while any unit is still in a waiting, maintenance, or error state. ``` watch -n 5 juju status trilio-wlm trilio-data-mover trilio-dm-api ``` Expected output — all units should show: ``` Agent Status Message idle active Unit is ready ``` **Do not proceed to Step 7 until all trilio-wlm and trilio-data-mover units are in active/idle state.** Running the unmount action while a unit is still configuring or restarting services may leave the unit in an inconsistent state. ## 7] Unmount Old Backup Targets Run the `unmount-old-backup-targets` action on all WLM and data-mover units. The action: * Stops and disables all `tvault-object-store-*.service` systemd units so they cannot remount targets * Kills any running `s3vaultfuse` processes * Lazy-unmounts all mounts under `/var/triliovault-mounts` deepest-first — NFS with `umount -l -f`, FUSE with `umount -l` A helper script is provided in the `triliovault-cfg-scripts` repository that automatically detects the Juju client version, runs the action on all units, and verifies that no stale mounts remain. The script runs with `bash -x` so each command is printed as it executes, making it easy to follow progress and diagnose any failures. It exits with a non-zero status and prints a clear **FAILED** message if any stale mounts are still present after cleanup. ``` bash juju-charms/unmount-backup-targets.sh ``` If the script exits with **SUCCESS**, proceed directly to Step 8. If it exits with **FAILED**, review the listed units and investigate manually before continuing. To run the action manually instead, use the `trilio_units` helper from the **Action syntax** section above: **Juju 3.x:** ``` for unit in $(trilio_units trilio-wlm trilio-data-mover); do echo "=== $unit ===" juju run "$unit" unmount-old-backup-targets --wait=300s done ``` **Juju 2.x:** ``` for unit in $(trilio_units trilio-wlm trilio-data-mover); do echo "=== $unit ===" juju run-action "$unit" unmount-old-backup-targets --wait=300s done ``` The action is idempotent, so re-running it on a unit that is already clean is harmless. ## 8] Recreate Backup Targets After unmounting old backup targets, recreate them in T4O 6.2 using the `upgrade_backup_targets_62.sh` script. Run this script from the **Juju controller node** — it connects to the WLM unit via `juju ssh` and executes all commands remotely. No need to copy files to the WLM unit manually. This step is only needed when upgrading from a previous release (5.x, 6.0, 6.1) that used static backup targets configured via `trilio-backup-targets`. For fresh 6.2 installations, this step does not apply — see [Getting started with Trilio on Canonical OpenStack](/openstack/deployment/installing-on-canonical.md) instead. The script: * Reads `trilio-backup-targets` from the backed-up overlay bundle file * Automatically finds the active/blocked WLM leader unit * Checks if each backup target already exists by backend endpoint — creates if not found, updates if found * NFS targets: creates with NFS mount options; if already exists, updates mount options via `workloadmgr backup-target-modify` * S3 targets: creates a Barbican secret using `trilio-dms-cli` and `openstack secret store`, then creates or updates the backup target with `--secret-ref` * Prints every command and its full output to the screen **Prerequisites (on the Juju controller node):** * OpenRC file with OpenStack admin credentials * Backed-up overlay bundle YAML from the previous release **Run the script:** edit the old bundle file path and OpenStack RC file path. ``` ./juju-charms/upgrade_backup_targets_62.sh \ -b tvo-overlay-bundle-.yaml \ -r /home/ubuntu/openrc ``` The script exits with code 0 if all targets were created/updated successfully, or code 1 if any errors occurred. The script sources the OpenRC file on the controller node to resolve any variable substitutions, strips `OS_CACERT`, and copies a clean literal-value file to the WLM unit over `juju scp`. The temp file is removed from the WLM unit automatically on exit. ## Sample Overlay Bundle (Reference) The sample overlay bundle for T4O 6.2 is available in the `triliovault-cfg-scripts` repository under: ``` juju-charms/sample_overlay_bundles/tvo-overlay-bundle.yaml # 6.2 — use as base for new deployments and upgrades ``` ## Notes * T4O 6.2.0 adds one relation that does not exist in 6.1 or earlier: `trilio-data-mover:identity-service` ↔ `keystone:identity-service`. It must be added explicitly during an upgrade — the overlay bundle is not deployed, so relations declared in it are never created. See Step 5. * The `unmount-old-backup-targets` action is idempotent — safe to run multiple times. * The `trilio-backup-targets` and `backup-target-type` charm config options are absent in 6.2 charms — remove them from your overlay bundle before deploying. * The `triliovault-pkg-source` change (Step 3) must always be applied **after** the charm refresh in Step 2 has settled on every unit, never before or during it. Applying it first causes the outgoing charm to consume the change and skip packages added in the new release. * The `update-trilio` action (Step 4) exists on all four charms and must be run after every `juju refresh`/`juju upgrade-charm` — otherwise the new packages and pending DB migrations may never actually be applied. * The overlay bundle prepared in Step 1 is a reference/future-install artifact only — it is never deployed via `juju deploy --overlay` during an upgrade. The upgrade itself is applied entirely through Steps 2–5 (per-application `juju refresh`, the package source change, the `update-trilio` action, and the new Keystone relation).