1 of 96

T4O-4.1

About Trilio for OpenStack

Trilio, by TrilioData, is a native OpenStack service that provides policy-based comprehensive backup and recovery for OpenStack workloads. The solution captures point-in-time workloads (Application, OS, Compute, Network, Configurations, Data and Metadata of an environment) as full or incremental snapshots. These snapshots can be held in a variety of storage environments including NFS AWS S3 compatible storage. With Trilio and its single click recovery, organizations can improve Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO). With Trilio, IT departments are enabled to fully deploy OpenStack solutions and provide business assurance through enhanced data retention, protection and integrity.

With the use of Trilio’s VAST (Virtual Snapshot Technology), Enterprise IT and Cloud Service Providers can now deploy backup and disaster recovery as a service to prevent data loss or data corruption through point-in-time snapshots and seamless one-click recovery. Trilio takes point-in-time backup of the entire workload consisting of compute resources, network configurations and storage data as one unit. It also takes incremental backups that only captures the changes that were made since last backup. Incremental snapshots save time and storage space as the backup only includes changes since the last backup. The benefits of using VAST for backup and restore could be summarized as below:

Efficient capture and storage of snapshots. Since our full backups only include data that is committed to storage volume and the incremental backups only include changed blocks of data since last backup, our backup processes are efficient and storages backup images efficiently on the backup media
Faster and reliable recovery. When your applications become complex that snap multiple VMs and storage volumes, our efficient recovery process will bring your application from zero to operational with just click of button
Easy migration of workloads between clouds. Trilio captures all the details of your application and hence our migration includes your entire application stack without leaving any thing for guess work.
Through policy and automation lower the Total Cost of Ownership. Our tenant driven backup process and automation eliminates the need for dedicated backup administrators, there by improves your total cost of ownership.

Trilio for OpenStack Architecture

Backup-as-a-Service

Trilio is an add on service to OpenStack cloud infrastructure and provides backup and disaster recovery functions for tenant workloads. Trilio is very similar to other OpenStack services including nova, cinder, glance, etc and adheres to all tenets of OpenStack. It is a stateless service that scales with your cloud.

Main Components

Trilio has four main software components:

Trilio ships as a QCOW2 image. User can instantiate one or more VMs from the QCOW2 image on a standalone KVM boxes.
Trilio API is a python module that is installed on all OpenStack controller nodes where the nova-api service is running.
Trilio Datamover is a python module that is installed on every OpenStack compute nodes
Trilio horizon plugin is installed as an add on to horizon servers. This module is installed on every server that runs horizon service.

Service Endpoints

Trilio is both a provider and consumer into OpenStack ecosystem. It uses other OpenStack services such as nova, cinder, glance, neutron, and keystone and provides its own service to OpenStack tenants. To accomodate all possible OpenStack deployments, Trilio can be configured to use either public or internal URLs of services. Likewise Trilio provides its own public, internal and admin URLs.

Network Topology

This figure represents a typical network topology. Trilio exposes its public URL endpoint on public network and Trilio virtual appliances and data movers typically use either internal network or dedicated backup network for storing and retrieving backup images from backup store.

Trilio 4.1 Release Notes

Trilio release 4.1 introduces new features and capabilities including:

Openstack Ussuri support
New File Recovery Process
Increased Openstack independency
Installation Optimization for Kolla Ansible Openstack, Ansible Openstack, and Red Hat Openstack Platform
Support for External MySQL/MariaDB databases
Incremental Backup for nova booted instances
Support of Openstack User Groups
New Quota: Snapshots
S3 Support for Kolla Ansible Openstack
UI enhancement for selective Restore

Trilio 4.1.94 is the GA release of Trilio 4.1

Release Versions

Packages

Name

Type

Version

Containers and Gitbranch

Openstack Ussuri support

Trilio 4.1 continues to enable Openstack versions and Distributions, allowing Trilio customers to stay up to date with Openstack releases.

Trilio 4.1 introduces full support for Openstack Ussuri for Kolla Ansible Openstack, Ansible Openstack, and Canonical Openstack. In addition, Trilio 4.1 does of course support the active long-term releases from Red Hat and Canonical. Openstack users of those releases can continue to use the latest Trilio functionalities.

New File Recovery Manager Process

Since Trilio was released for the first time was the File Recovery Manager instance in tandem with Trilio. The File Recovery Manager instance helped customers to easily and quickly fetch and restore files and folders directly from the backups.

Over the years did more and more customers request to have the File Recovery Manager on their own images or on a specific Linux distribution. We looked into the possibility to create multiple versions of the File Recovery Manager which was identified as not getting to the point of flexibility our customers need.

The result was a revamp of the File Recovery Manager which allows the installation on any CentOS-based or Ubuntu-based instance.

Increased Openstack independency

Trilio has the goal to become an integrated part of Openstack since its first draft on a whiteboard. Back then were the Trilio services defined as sub-services of Nova and Trilio tied deeply into the already existing nova services and became an integral part of it.

Over the years did the nova service change and possibilities that were once available have been taken out from nova. This plus the required lengthy qualification of all Openstack Versions and Distributions made it clear that a change is required.

Trilio 4.1 is now a complete stand-alone service, which is communicating with nova and other Openstack services through APIs only. The goal of this independence is to speed up future qualification and requalification cycles to provide a broader support matrix again.

Installation optimization for Kolla Ansible Openstack, Ansible Openstack, and Red Hat Openstack Platform

Trilio's integration into Openstack already starts with the installation process. This process required manual installation steps in the past, which were hard to automize and scale on bigger environments.

Trilio 4.1 is therefore introducing Ansible Playbooks for Kolla-Ansible and Ansible Openstack, while the integration with Red Hat Director has been deepened. Further, the Trilio configurator does now allow to set the backup target on the Trilio Appliance directly based on Distribution.

Support for external MySQL/MariaDB

The Trilio Appliance provides its own database. A database that suits the needs of 99% of Trilio's customers. Some customers do have higher requirements. Be it performance, security, or just the general design of the Openstack itself.

For these customers and everyone who wants to use it is Trilio 4.1 introducing the possibility to configure Trilio with an external database.

Incremental Backups for nova booted instance root volumes

Openstack provides the possibility to start an instance from a Cinder Volume or directly using the Glance Image and a nova volume.

Trilio always provided incremental forever backups for all cinder volumes. Root-Volumes from nova-booted instances were always taken as a full backup of the Glance image and the actual VM root volume.

Trilio 4.1 does introduce incremental backups for nova booted instance root volumes. This allows Trilio 4.1 to provide incremental backups to any type of root volume.

Support for Openstack User Groups

Trilio is using the Openstack Keystone service to authenticate any user and to verify that the right permissions are set. Openstack Keystone allows to group users and set the permissions to the group.

These Openstack User Groups are now fully supported.

New Quota: Snapshots

Trilio 4.0 introduced the Quota functionality, which allowed to set quotas for the number of workloads, number of VMs and amount of storage used by a single tenant.

Trilio 4.1 extends this feature by the number of Trilio Snapshots that a Tenant is allowed to have.

S3 Support for Kolla Ansible Openstack

S3 is becoming the standard to transfer data to and from storage solutions. Trilio introduced S3 already in Version 3.0 but had to take it out for Kolla Ansible since Kolla Ansible has been added to the Support Matrix.

Trilio 4.1 is now closing that gap to other Openstack Distributions and provides full S3 support for Kolla Ansible Openstack Ussuri.

UI enhancement for the Selective Restore

The Selective Restore is the most powerful and complex restore Trilio has to offer. The UI needs to be easy to understand and help the user to fulfill its task.

Several points have been identified to improve this requirement of usability. The selective restore now allows to select or deselect all VMs at once. Further are the VMs now provided in an easy to overview list and the sub-controls can be expanded and collapsed as necessary.

Known Issues

This release contains the following known issues which are tracked for a future update.

Snapshots getting stuck in uploading stage on RHOSP13 with ISCSI based Volumes

Observation:

Login into the iscsi device is getting rejected for the Trilio service
Result is the Snapshot not moving forward until timing out

Workloadmgr Quota feature does not support RHOSP13 & Canonical Queens Horizon

The workloadmgr Quota feature is still fully supported through CLI.

[Openstack Ansible] snapshot mount failed with error 'Permission denied'

Observation:

It has been observed that on non-kolla, non-rhosp setups, such as openstack ansible, nova user id is not same as we consider as default(162).
Another observation is that, the id which is assigned to nova, was conflicting with id of system user in TVM, this created a situation where we had to redeploy Openstack.

Workaround:

Update permissions of /var/triliovault-mounts to 755

[Volume Backup Exclude] Excluded Ceph Volume after restore not mountable or formattable

Observation:

VM Volumes stored on Ceph are successfully excluded from backup if desired
Restore does create empty Ceph Volume
created empty Ceph Volume is not attachable or formattable

Restored VMs have blank metadata config_drive attached

Observation:

For every restore will the metadata config_drive be set as blank value
No impact on restored VMs known

Workaround

delete metadata config_drive
or set desired value

TVM reconfig fails when adding new Trilio VM to the cluster

Observation:

TVault re-configuration while adding nodes to existing TVM cluster fails at "Configuring Trilio Cluster"
Reason is that the prev mysql password was not working and mysql root access has be reset.

Workaround:

remove /root/.my.cnf file on already configured TVM and reconfigure it

Database does not sync after Trilio cluster gets new nodes

Observation:

After TVault re-configuration post addition of 2 more nodes to existing TVM cluster ("import workloads" was not seleted), the databases do not sync against already existing TVM.
It is expected that while adding the 2 new nodes, the db on node1 should get synced up with 2 new nodes and the existing workloads should be available post the reconfig on the new 3 node TVM cluster.

Workaround:

Run workload import from CLI

[exclude_boot_disk] Data on boot disk gets backed up despite exclusion

Observation:

VM was set with metadata exclude_boot_disk_from_backup set to true
Restored instance showed, that data was backed up and restored

After reinitialize and import Openstack certificates are missing

Observation:

Reinitialize does not keep the already uploaded Openstack Certificates used to communicate with Openstack.

Workaround:

Upload Certificates again

CLI import changes scheduler trust value to disabled

Observation:

Workaround:

Configure/re-configure T4O with import option from UI after reinitialize.

Unable to get node details after reinitializing the Trilio Appliance

Observation:

After reinitializing was neither the UI nor CLI showing Node information

Workaround:

Restart wlm-workloads and wlm-cron services on Trilio nodes
systemctl restart wlm-workloads
systemctl restart wlm-cron

Snapshots fails with "object is not subscriptable" for many workload jobs at the exact same time

Oberservation:

Running more than 25 workloads at the exact same time leads to error
dmapi service is not responding
Snapshots fail with "object is not subscriptable"

Workaround:

Contact Trilio Support to implement a known workaround.

[Kolla Ansible] dmapi container stuck in restarting if storage backend changes

Observation:

Just changing the backup target in the Kolla Ansible configuration files and redeploying leads to dmapi container stuck in restart

Workaround:

No operation is permitted in insecure way for ssl enabled Keystone URL

Observation:

SSL enabled Openstack
Backup and Restore jobs fail with with missing TLS CA certificate bundle error

Workaround:

Configure the Trilio appliance with Openstack CA provided
OR Provide Openstack CA to /etc/workloadmgr/ca-chain.pem

Trilio 4.1 HF1 Release Notes

Release Versions

Packages

Name

Type

Version

Containers and Gitbranch

Changelog

Enhancements

Support for passwordless SMTP servers

Trilio for Openstack supports sending notification emails upon succeeded or failed backup/restore jobs. The required SMTP server configuration enforced the usage of a password for the SMTP user.

A password is no longer necessary when the SMTP server doesn’t need it.

Support for Multi-Attach Volumes

Cinder supports a Volume Type, which allows attaching the same Volume to multiple instances simultaneously. Backups and Restore for this Volume Type failed. Trilio for Openstack is now providing base support for this Volume Type.

Cinder Boot Volumes with Multi-Attach activated are not yet supported.

This only allows the backup and restoration of Multi-Attach Volumes. Trilio will handle the Volume like any single attached Volume for now. For example, a multi-attach volume connected to 2 VMs will get backed up and restored twice.

Increased the timeout for data transfer

Trilio for Openstack is tracking the progress of backups and restore using a tracking file. If this file is not getting updated within a defined timeframe, the Trilio data transfer fails. This timeframe got extended from 10 minutes to 20 minutes.

This value will become configurable in T4O 4.1 SP1

Misleading and expected errors are no longer shown in logs by default

Trilio for Openstack was logging many system error messages, which were actually expected and handled internally without impacting the actual functions of the solution.

These error messages were misleading in the normal troubleshooting process. These error messages aren’t logged anymore by default. They can be reactivated using the debug mode for logging.

Image upload timeout window has been increased and made configurable

The upload of Trilio backups is limited in time to prevent stalling workloads with a stuck upload process.

This timeout window has been increased from 10 hours to 48h by default and is configurable in the workloadmgr config file.

Restart wlm-workloads after setting this value.

Grace time to retrigger a Snapshot in case of deactivated Global Job Scheduler

When the Global Job Scheduler is deactivated, no backups are triggered. The Global Job Scheduler contains a grace time for missed Snapshots. All Snapshots that were supposed to be triggered within this grace time before activation of the Global Job Scheduler are retriggered.

This grace period is now configurable in the workloadmgr config file.

After setting this value, restart the wlm-cron service.

Increased the amount of dmapi workers and made it configurable

In highly used environments, the dmapi worker got identified as a potential bottleneck. The amount of default workers used by a dmapi service has been increased to 16 and is configurable in the dmapi config file.

Restart the dmapi service after setting the configuration manually.

The upgrade process of RHOSP and Kolla Ansible will automatically set this value.

Added new settings in haproxy configuration for dmapi

Response times in highly used environments might be slow, leading to the dmapi service timing out in the haproxy connection. The default values of haproxy are not always suitable in that case.

The haproxy configuration for the dmapi service has been extended to the following values.

Restart the haproxy service after setting the configuration manually.

The upgrade process of RHOSP and Kolla Ansible will automatically set these values.

Added retries for Neutron API calls

The Openstack Neutron service is highly used up to the point that API calls are timing out. Trilio backups and restores failed when any Neutron API call timed out.

Trilio will now retry Neutron API calls three times before failing a backup or restore.

Added retries and rescan for mounting temporary Volumes using Cinder Storages with multipathing activated

It was observed in multipathing environments that sometimes backups failed due to errors with the temporary Cinder Volumes during the following actions:

Create Cinder Volume out of Cinder Snapshot
Mount Cinder Volume to Compute Node
Unmount Cinder Volume from Compute Node
Delete Cinder Volume

During these operations in multipath environments, errors are now handled by rescanning the connected devices and retrying the internal commands.

The amount of retries is configurable in the tvault-contego config file.

Restart the tvault-contego service after manually setting the value.

The upgrade process of RHOSP and Kolla Ansible will automatically set these values.

Enhanced logging of Trilio for Openstack GUI

The Trilio for Openstack GUI uses the admin account to secure access to the features and functionalities located on the Trilio appliance. The following events are now getting logged by the Trilio Appliance:

Login attempts
Logout events
Password changes for the admin user

The Trilio for Openstack login page can now is extendable to contain a text banner. This text banner is configurable on the Trilio appliance by editing the banner yaml file located under:

/etc/tvault-config/banner.yaml

The content of the file looks as follows:

Restart tvault-config after changing the banner to activate it.

Fixed Bugs and issues

Local job scheduler stays disabled after workload creation with enabled job scheduler

An issue got fixed for rare occasions in which the status of the local job scheduler of a single workload was disabled, despite the workload created with an enabled job scheduler.

Global Job Scheduler is shown as active even when wlm-cron service is down

An issue got fixed for the Global Job Scheduler returning enabled or disabled even when the wlm- cron service is deactivated. The status returned in this scenario is now an error message showing the wlm-cron service status.

Wrong documentation link in the Trilio Appliance

The documentation link available inside the Trilio Appliance was still pointing to the old outdated documentation webpage. The link has been updated to point towards the correct documentation.

Restore VMs into different Availability Zone failed

An issue got fixed, which prevented the restoration of VMs into different Availability Zones in the case of the original Availability Zone no longer being available.

Stopping workloadmgr service and all ongoing worker tasks

An issue got fixed, which lead to stall service jobs being left behind upon restart of workloadmgr services.

Mounting of LVM configured disks into the FRM failed

An issue got fixed, which prevented the correct mounting and access of Volumes partitioned and configured by LVM.

Upload of Backups failed intermittently using S3

An issue got fixed, which lead to a race condition between upload threads in the S3 fuse plugin, which led to backups failing during the upload phase.

Multipathing not enabled by default in the Data-Mover container used in RHOSP and Kolla Ansible Openstack

An issue got fixed, which lead to multipathing not being enabled in the Data-Mover container used by RHOSP and Kolla Ansible.

Upgrading to 4.1 HF1 will automatically activate multipathing where feasible.

An inaccessible S3 mount point got created when the S3 endpoint is not available during deployment or configuration

An issue got fixed, which lead to the creation of a Trilio mount point, even when the provided S3 backup target is not reachable during deployment or configuration.

The deployment will still succeed, but the tvault-object-store service will be in a failed state.

Trustee role not correctly inherited from user groups to users

An issue got fixed, which prevented the detection of the Trilio trustee role for a user, who had this role inherited from a user group.

The configurator always trying to use Keystone internal endpoint

An issue got fixed, during which the chosen endpoint type did not get honored for Keystone and the configurator always reached out to the Keystone internal endpoint.

The following value can be set in the api-paste ini file located under:

/etc/workloadmgr/api-paste.ini

Afterward the wlm-workloads service needs to be restarted.

It is recommended to reconfigure the appliance to activate the fix

Disk integrity check failing with a false positive

An issue got identified, which leads to the disk integrity check failing, although there is no data loss.

Snapshots with a failed disk integrity check are currently no longer failing and instead show a warning in the log files about the failed disk integrity check.

A complete fix of the root cause is planned for 4.1 SP1.

Workload creation and Backup creation failed due to Latin characters like á

An issue got identified in which Latin characters like á did lead to a Workload not being created or a backup not succeeding.

This hotfix implements support of Latin characters for the following:

Calendar shown and used during workload creation for the job scheduler
Name and description of security groups

Full support for Latin characters comes in 4.1 SP1

Backups for multipath environments using the FC protocol failed

An issue got fixed, which prevented successful backups in environments using multipathing with the FC storage protocol.

Trilio 4.1 HF2 Release Notes

Release Versions

Packages

Name

Type

Version

Containers and Gitbranch

Changelog

Contains all changes from HF1.

Fixed bugs and issues

Restore of Cinder boot volumes with multi-attach activated

An issue has been fixed that prevented a successful restore in the case of restoring a Cinder boot volume with a volume type that has the multi-attach functionality activated.

Backup of many instances or of instances with long names

An issue has been fixed which prevented the successful finish of the backup process for workloads with many protected instances or instances with long names.

Blank Ansible output for Trilio Appliance reconfiguration

An issue has been fixed which led to no visible Ansible logs upon reconfiguring the Trilio appliance.

misleading SMTP timeout error message upon sending a test email

An issue has been fixed which led to the SMTP configuration always throwing the misleading error smtp_timeout cannot be greater than 10 upon sending a test email.

Security Group restore for remote security groups with identical rules

An issue has been fixed which led to Security Groups not being restored when a remote Security Group was having the exact same Security Group Rule as another Security Group in the chain.

Failed Security Group Restore now leads to failed restore

An issue has been fixed which led to a restore apparently completing successfully despite an error during the restore of the Security Groups.

Trilio 4.1 HF3 Release Notes

Release Versions

Packages

Name

Type

Version

Containers and Gitbranch

Changelog

Contains all changes from HF1 + HF2.

This Hotfix extends the Support Matrix of T4O 4.1 as follows:

Canonical Openstack Victoria based on Focal (20.04) Support
Kolla Ansible Openstack Victoria on Ubuntu 20.04 and CentOS8
Openstack Ansible Victoria on Ubuntu 20.04 and CentOS8
TripleO train on CentOS7 and CentOS8

The installation into these environments requires upgrading the Trilio Appliance from 4.1 GA to 4.1 HF3 or higher

Trilio 4.1 HF4 Release

Release Versions

Packages

Name

Type

Version

Containers and Gitbranch

Changelog

Contains all changes from HF1, HF2 and HF3.

Fixes bugs and issues

Filesearch in ext2 filesystem

A bug has been fixed which prevented the correct presentation of files on an ext2 filesystem when running a filesearch.

Filesearch on CentOS8 and RHEL8 instances

An issue has been fixed which prevented a correct filesearch on CentOS8 and RHEL8 backups with the xfs filesystem.

Selective Restore without original Glance image being available

An issue has been fixed which prevented a successful selective restore in case of the original Glance image not being available anymore.

Creation of a new glance image during restore

An issue has been fixed which prevented the successful creation of a Glance image during a restore.

RHOSP13 installation mountpath ownership

An issue has been fixed which set the wrong ownership to the Trilio mountpoint.

Enhancements

Configurable maximum S3 connection

A new configuration parametervault_s3_max_pool_connectionshas been added to adjust the number of pool connections. The default value is 500.

This parameter can be set in the workloadmgr.conf on the Trilio appliance.

In the case of a Canonical installation this parameter is to be set in the tvault-object-store.conf in the workloadmgr container.

Default value for s3 read timeout has been decreased

The default value for the configuration parameter vault_s3_max_pool_connections has been reduced from 120 to 30.

This parameter can be set in the workloadmgr.conf on the Trilio appliance.

In the case of a Canonical installation this parameter is to be set in the tvault-object-store.conf in the workloadmgr container.

NFS mount process of datamover enhanced

A timeout with automatic process kill has been added to prevent high CPU usage from stale NFS mount operations in case of mounting errors.

Snapshot mount on Ubuntu 20.04

The documentation has been extended to support the usage of Ubuntu 20.04 cloud images for the Snapshot mount functionality.

Trilio 4.1 HF5 Release

Release Versions

Packages

Name

Type

Version

Containers and Gitbranch

Changelog

Contains all changes from HF1, HF2, HF3, and HF4.

Fixes bugs and issues

Post restore Windows VM boots into recovery console

An issue has been fixed which prevented VMs running Windows to boot properly after a restore in the case of Nova boot volumes being used.

Workload list can not be retrieved after manual changes in the Trilio database

An issue has been fixed which prevented the successful pull of data from the Trilio database in case of incorrectly applied manual changes.

admin endpoint type not honored by the configurator

An issue has been fixed which prevented the successful usage of the Openstack admin endpoint network as the standard communication network for T4O

Workloads and Snapshots stuck in delete status

An issue has been fixed which prevented resetting the status of Workloads and Snapshots in case they are stuck in the deletion state.

Enhancements

Workload scheduler stability

The workload scheduler stability has been enhanced to prevent the start of multiple scheduled jobs at the same time or delayed from its expected time.

Trilio 4.1 HF6 Release

Release Versions

Packages

Name

Type

Version

Containers and Gitbranch

Changelog

Contains all changes from previous hotfixes.

Fixes bugs and issues

Restore of tenant-shared security groups failed

An issue has been fixed which prevented a successful restore when a security group was referring to a shared security group from a different tenant.

Email alerts not working as intended

Fixed an issue where email alerts were not sent in the case of a passwordless SMTP server.

Trilio 4.1 HF7 Release

Release Versions

Packages

Name

Type

Version

Containers and Gitbranch

Changelog

Contains all changes from previous hotfixes.

Fixed bugs and issues

Temporary Volumes left behind under certain circumstances

An issue has been fixed which left the Trilio temporary Cinder Volumes behind when the upload was timing out or when the Trilio cluster got restarted.

Setting invalid date through workload edit

An issue has been fixed which allowed setting invalid dates through the workload edit command.

Parallel workload performing significantly worse

A race condition has been fixed which led to an exponential growth of required time with an increased amount of protected VMs.

Restore of ports with port-security disabled

An issue has been fixed which prevented a successful restore of Neutron ports that have the port-security functionality disabled.

Trilio 4.1 HF8 Release

Release Versions

Packages

Name

Type

Version

Containers and Gitbranch

Qualifications

The following OpenStack distributions and versions have been added to the Trilio support matrix.

Red Hat OpenStack 16.2

Changelog

Enhancements

Support for Ceph NFS-Ganesha

It is now possible to utilize the NFS-Ganesha gateway as backup target.

Restore continues even when Security Group restore fails

Failures upon restoring Security Groups are no longer leading to a complete fail of the restore. Failed Security Groups are logged to provide the required information for next steps.

Future versions and hotfixes will continue to improve the Security Group restore process and reduce the amount of reasons why Security Groups can't be restored.

Fixed bugs and issues

AttributeError: 'unicode' object has no attribute 'get'

An issue with multipathing environments has been fixed which led to failed backups with the AttributeError: 'unicode' object has no attribute 'get' error message.

Restricting the IP address of the Trilio GUI lead to logs not being downloadable

The documentation about restricting the IP access to the Trilio GUI has been updated to include the required port 3001 to enable the download of logs through the dashboard_ip.

Multipath.conf file not present in Datamover container

An issue has been fixed which prevented the correct placement of the multipath.conf file inside the Trilio Datamover container.

Trilio 4.1 HF9 Release

This hotfix contains only a package update for the Trilio appliance. There are no new containers available compared to earlier releases.

Release Versions

Packages

Name

Type

Version

Containers and Gitbranch

Changelog

200% cpu usage or spike in CLOSE_WAIT connections with S3 backup target

An issue has been fixed which lead to a high CPU resource usage in case of fluctuating connection to the S3 backup target.

Trilio 4.1 HF10 Release

Release Versions

Packages

Name

Type

Version

Containers and Gitbranch

Changelog

Fixed Bugs and issues

Rebase permission error

An issue has been fixed which prevented the correct rebase of T4O incremental backups in the case that root didn't have the required permissions on the backup target.

Trilio 4.1 HF11 Release

Release Versions

Packages

Name

Type

Version

Containers and Gitbranch

Changelog

Fixed Bugs and issues

If Trilio created Cinder snapshot is not in available state, all subsequent backups fail.

An issue has been fixed which led to backups ending unsuccessfully when Trilio wasn't able to delete older no longer required Cinder Snapshots.

Multipath rescan commands not executing in a timely manner

An issue has been fixed which led to failed detach and deletion of temporary volumes due to the rescan command not executing fast enough.

Improved stability of the S3fuse plugin

Several issues have been fixed which led to an unstable connection of the S3fuse plugin, leading to backups and restore failing during the data transfer phase.

Enhanced support for Latin characters

An issue has been fixed where the usage of Latin characters in the restore name or description did lead to the restore being unsuccessful.

Trilio 4.1 HF12 Release

Release Versions

Packages

Name

Type

Version

Containers and Gitbranch

Changelog

Fixed Bugs and issues

Datamover container restarting
/var/trilio/triliovault-mounts was having incorrect ownership. We have fixed it through DevOps code
Once the original project is deleted, the workload cannot be reassigned to a different UserID/ProjectID.
We removed the condition where we were checking if older tenant_ids if present in the newer tenant list
Trilio core functionality operations do not perform as expected when the master T4O node is powered off
Caching bug in the code where the in-memory dictionary was not in sync with the service table in MySQL.
tvault-config service is in crashloop on 2 out of 3 nodes in T4O cluster
workload policy shows an incorrect start time
default_tvault_dashboard_tvo-tvm not available after yum update
Reassign of the workload from deleted project fails

Trilio 4.1 HF13 Release

Release Artifacts

1. Introduction

This document provides information on TVO-4.1.HF13 Release.

Important Info:

To use this hotfix (4.1.HF13)

Customers (except Canonical Openstack) and having Openstack Ussuri need to have an already deployed and working TVO-4.1 GA OR TVO-4.1.HF1 OR TVO-4.1.HF2 OR TVO-4.1.HF3 OR TVO-4.1.HF4 OR TVO-4.1.HF5 OR TVO-4.1.HF6 OR TVO-4.1.HF7 OR TVO-4.1.HF8 OR TVO-4.1.HF9 OR TVO-4.1.HF10 OR TVO-4.1.HF11 OR HF12
Customers (except Canonical Openstack) and having Openstack Victoria OR TripleO Train need to follow the TVO-4.1 GA deployment process and directly upgrade to 4.1.HF13 containers/packages. The high-level flow below:
1. Deplo T4O-4.1 GA appliance.
2. Upgrade to 4.1.HF13 packages on the appliance.
3. Kolla & TripleO
  1. Deploy Trilio components via 4.1.HF13 containers/packages on Openstack Victoria/TripleO Train.
4. Openstack Ansible
  1. Deploy Trilio components Openstack Victoria [This will deploy 4.1 GA packages]
  2. Upgrade TrilioVault packages to 4.1.HF13 on Openstack Victoria.
5. Configure the Trilio appliance.
Canonical users having Openstack Ussuri can either upgrade (on top of 4.1 GA) using Trilio upgrade documents OR do a fresh deployment using 4.1 Deployment documents.
Canonical users having Openstack Victoria can either upgrade (on top of 4.1.HF4) using Trilio upgrade documents OR do a fresh deployment using 4.1 Deployment documents.

The deploy/upgrade documentations provide the detailed steps to deploy/upgrade to the hotfix.

2. Release Scope

Current Hotfix release targets the following:

Verification of Jira issues targeted for 4.1.HF13 release.
As part of the new process, the delivery will be via packages; end users would need to do the rolling upgrade on top of 4.1 GA OR 4.1.HF1 OR 4.1.HF2 OR TVO-4.1.HF3 OR TVO-4.1.HF4 OR TVO-4.1.HF5 OR TVO-4.1.HF6 OR TVO-4.1.HF7 OR TVO-4.1.HF8 OR TVO-4.1.HF9 OR TVO-4.1.HF10 OR TVO-4.1.HF11 OR TVO-4.1.HF12

3. Tag References for Rolling Upgrades

4. Resolved Issues

Issues logged by Customers to be documented in this section

5. Deliverables against 4.1.HF13

Following packages changed/added in the current release

6. T4O Deployment Coverage

The following table gives the overview of coverage against Trilio Deployment with Openstack:

7. Backup Store Coverage

The following table gives the overview of coverage against backup stores covered as part of the development and testing of 4.1.HF9 release.

8. Known Issues

Deployment Guide

Compatibility Matrix

Learn about Trilio Support for OpenStack Distributions

The CentOS community has moved over to CentOS stream.

The support for CentOS8 has ended on December 31st 2021. The official announcement can be found .

CentOS7 is still supported and maintained till June 30th 2024

Kolla Ansible environments running on CentOS8 are receiving continuoues limited support. This means that future updates from Trilio for Kolla Ansible environments on CentOS8 will use the latest available CentOS8 base containers and only the Trilio for OpenStack code gets updated. When the Kolla Ansible community provides CentOS Stream based containers, Trilio will provide CentOS Stream based containers as well.

Trilio for OpenStack Compatibility Matrix

Trilio Release

RHOSP

Canonical

Ansible

Kolla

TripleO

NFS & S3 Support:

All versions of Trilio for OpenStack support NFSv3 and S3 as backup targets.

Supported OS:

RHEL7 and RHEL8 for RHOSP; Ubuntu 18.04 and 20.04 for Canonical distributions; Ubuntu 18.04, 20.04, and CentOS Stream for Ansible OpenStack; Ubuntu 18.04, 20.04 and CentOS Stream for Kolla OpenStack; CentOS7 for TripleO Train.

Deployment:

RHOSP distributions are deployed using Red Hat Director, Canonical distributions are deployed using JuJu Charms, Ansible OpenStack and Kolla OpenStack distributions are deployed using Ansible, and TripleO Train is deployed using Heat.

Compatibility Matrix Detailed View

Trilio Appliance Administration Guide

User Guide

Admin Guide

Troubleshooting

API GUIDE

Trilio 4.1 Release Notes

Trilio release 4.1 introduces new features and capabilities including:

Openstack Ussuri support
New File Recovery Process
Increased Openstack independency
Installation Optimization for Kolla Ansible Openstack, Ansible Openstack, and Red Hat Openstack Platform
Support for External MySQL/MariaDB databases
Incremental Backup for nova booted instances
Support of Openstack User Groups
New Quota: Snapshots
S3 Support for Kolla Ansible Openstack
UI enhancement for selective Restore

Trilio 4.1.94 is the GA release of Trilio 4.1

Release Versions

Packages

Name

Type

Version

Containers and Gitbranch

Name

Tag

Openstack Ussuri support

Trilio 4.1 continues to enable Openstack versions and Distributions, allowing Trilio customers to stay up to date with Openstack releases.

New File Recovery Manager Process

The result was a revamp of the File Recovery Manager which allows the installation on any CentOS-based or Ubuntu-based instance.

Increased Openstack independency

Installation optimization for Kolla Ansible Openstack, Ansible Openstack, and Red Hat Openstack Platform

Support for external MySQL/MariaDB

For these customers and everyone who wants to use it is Trilio 4.1 introducing the possibility to configure Trilio with an external database.

Incremental Backups for nova booted instance root volumes

Openstack provides the possibility to start an instance from a Cinder Volume or directly using the Glance Image and a nova volume.

Trilio 4.1 does introduce incremental backups for nova booted instance root volumes. This allows Trilio 4.1 to provide incremental backups to any type of root volume.

Support for Openstack User Groups

These Openstack User Groups are now fully supported.

New Quota: Snapshots

Trilio 4.0 introduced the Quota functionality, which allowed to set quotas for the number of workloads, number of VMs and amount of storage used by a single tenant.

Trilio 4.1 extends this feature by the number of Trilio Snapshots that a Tenant is allowed to have.

S3 Support for Kolla Ansible Openstack

Trilio 4.1 is now closing that gap to other Openstack Distributions and provides full S3 support for Kolla Ansible Openstack Ussuri.

UI enhancement for the Selective Restore

The Selective Restore is the most powerful and complex restore Trilio has to offer. The UI needs to be easy to understand and help the user to fulfill its task.

Known Issues

This release contains the following known issues which are tracked for a future update.

Snapshots getting stuck in uploading stage on RHOSP13 with ISCSI based Volumes

Observation:

Login into the iscsi device is getting rejected for the Trilio service
Result is the Snapshot not moving forward until timing out

Workloadmgr Quota feature does not support RHOSP13 & Canonical Queens Horizon

The workloadmgr Quota feature is still fully supported through CLI.

[Openstack Ansible] snapshot mount failed with error 'Permission denied'

Observation:

It has been observed that on non-kolla, non-rhosp setups, such as openstack ansible, nova user id is not same as we consider as default(162).
Another observation is that, the id which is assigned to nova, was conflicting with id of system user in TVM, this created a situation where we had to redeploy Openstack.

Workaround:

Update permissions of /var/triliovault-mounts to 755

[Volume Backup Exclude] Excluded Ceph Volume after restore not mountable or formattable

Observation:

VM Volumes stored on Ceph are successfully excluded from backup if desired
Restore does create empty Ceph Volume
created empty Ceph Volume is not attachable or formattable

Restored VMs have blank metadata config_drive attached

Observation:

For every restore will the metadata config_drive be set as blank value
No impact on restored VMs known

Workaround

delete metadata config_drive
or set desired value

TVM reconfig fails when adding new Trilio VM to the cluster

Observation:

TVault re-configuration while adding nodes to existing TVM cluster fails at "Configuring Trilio Cluster"
Reason is that the prev mysql password was not working and mysql root access has be reset.

Workaround:

remove /root/.my.cnf file on already configured TVM and reconfigure it

Database does not sync after Trilio cluster gets new nodes

Observation:

After TVault re-configuration post addition of 2 more nodes to existing TVM cluster ("import workloads" was not seleted), the databases do not sync against already existing TVM.
It is expected that while adding the 2 new nodes, the db on node1 should get synced up with 2 new nodes and the existing workloads should be available post the reconfig on the new 3 node TVM cluster.

Workaround:

Run workload import from CLI

[exclude_boot_disk] Data on boot disk gets backed up despite exclusion

Observation:

VM was set with metadata exclude_boot_disk_from_backup set to true
Restored instance showed, that data was backed up and restored

After reinitialize and import Openstack certificates are missing

Observation:

Reinitialize does not keep the already uploaded Openstack Certificates used to communicate with Openstack.

Workaround:

Upload Certificates again

CLI import changes scheduler trust value to disabled

Observation:

When the is used via CLI is the scheduler trust changed from enabled to disabled.

Workaround:

Configure/re-configure T4O with import option from UI after reinitialize.

Unable to get node details after reinitializing the Trilio Appliance

Observation:

After reinitializing was neither the UI nor CLI showing Node information

Workaround:

Restart wlm-workloads and wlm-cron services on Trilio nodes
systemctl restart wlm-workloads
systemctl restart wlm-cron

Snapshots fails with "object is not subscriptable" for many workload jobs at the exact same time

Oberservation:

Running more than 25 workloads at the exact same time leads to error
dmapi service is not responding
Snapshots fail with "object is not subscriptable"

Workaround:

Contact Trilio Support to implement a known workaround.

[Kolla Ansible] dmapi container stuck in restarting if storage backend changes

Observation:

Just changing the backup target in the Kolla Ansible configuration files and redeploying leads to dmapi container stuck in restart

Workaround:

Follow guide to

No operation is permitted in insecure way for ssl enabled Keystone URL

Observation:

SSL enabled Openstack
Backup and Restore jobs fail with with missing TLS CA certificate bundle error

Workaround:

Configure the Trilio appliance with Openstack CA provided
OR Provide Openstack CA to /etc/workloadmgr/ca-chain.pem

Trilio 4.1 HF1 Release Notes

Release Versions

Packages

Name

Type

Version

Containers and Gitbranch

Name

Tag

Changelog

Enhancements

Support for passwordless SMTP servers

Trilio for Openstack supports sending notification emails upon succeeded or failed backup/restore jobs. The required SMTP server configuration enforced the usage of a password for the SMTP user.

A password is no longer necessary when the SMTP server doesn’t need it.

Support for Multi-Attach Volumes

Cinder Boot Volumes with Multi-Attach activated are not yet supported.

Increased the timeout for data transfer

This value will become configurable in T4O 4.1 SP1

Misleading and expected errors are no longer shown in logs by default

Trilio for Openstack was logging many system error messages, which were actually expected and handled internally without impacting the actual functions of the solution.

These error messages were misleading in the normal troubleshooting process. These error messages aren’t logged anymore by default. They can be reactivated using the debug mode for logging.

Image upload timeout window has been increased and made configurable

The upload of Trilio backups is limited in time to prevent stalling workloads with a stuck upload process.

This timeout window has been increased from 10 hours to 48h by default and is configurable in the workloadmgr config file.

[DEFAULT] 
max_wait_for_upload = 48

Restart wlm-workloads after setting this value.

Grace time to retrigger a Snapshot in case of deactivated Global Job Scheduler

This grace period is now configurable in the workloadmgr config file.

[global_job_scheduler] 
misfire_grace_time = 600

After setting this value, restart the wlm-cron service.

Increased the amount of dmapi workers and made it configurable

[DEFAULT] 
dmapi_workers = 16

Restart the dmapi service after setting the configuration manually.

The upgrade process of RHOSP and Kolla Ansible will automatically set this value.

Added new settings in haproxy configuration for dmapi

Response times in highly used environments might be slow, leading to the dmapi service timing out in the haproxy connection. The default values of haproxy are not always suitable in that case.

The haproxy configuration for the dmapi service has been extended to the following values.

retries 5 
timeout http-request 10m 
timeout queue 10m 
timeout connect 10m 
timeout client 10m 
timeout server 10m 
timeout check 10m 
balance roundrobin 
maxconn 50000

Restart the haproxy service after setting the configuration manually.

The upgrade process of RHOSP and Kolla Ansible will automatically set these values.

Added retries for Neutron API calls

The Openstack Neutron service is highly used up to the point that API calls are timing out. Trilio backups and restores failed when any Neutron API call timed out.

Trilio will now retry Neutron API calls three times before failing a backup or restore.

Added retries and rescan for mounting temporary Volumes using Cinder Storages with multipathing activated

It was observed in multipathing environments that sometimes backups failed due to errors with the temporary Cinder Volumes during the following actions:

Create Cinder Volume out of Cinder Snapshot
Mount Cinder Volume to Compute Node
Unmount Cinder Volume from Compute Node
Delete Cinder Volume

During these operations in multipath environments, errors are now handled by rescanning the connected devices and retrying the internal commands.

The amount of retries is configurable in the tvault-contego config file.

[cinder] 
http_retries = 10

Restart the tvault-contego service after manually setting the value.

The upgrade process of RHOSP and Kolla Ansible will automatically set these values.

Enhanced logging of Trilio for Openstack GUI

Login attempts
Logout events
Password changes for the admin user

The Trilio for Openstack login page can now is extendable to contain a text banner. This text banner is configurable on the Trilio appliance by editing the banner yaml file located under:

/etc/tvault-config/banner.yaml

The content of the file looks as follows:

header: 
header_color: blue 
body_text_color: "#DC143C" 
body_text: 
header_font_size: 25px 
body_text_font_size: 22px

Restart tvault-config after changing the banner to activate it.

Fixed Bugs and issues

Local job scheduler stays disabled after workload creation with enabled job scheduler

An issue got fixed for rare occasions in which the status of the local job scheduler of a single workload was disabled, despite the workload created with an enabled job scheduler.

Global Job Scheduler is shown as active even when wlm-cron service is down

Wrong documentation link in the Trilio Appliance

The documentation link available inside the Trilio Appliance was still pointing to the old outdated documentation webpage. The link has been updated to point towards the correct documentation.

Restore VMs into different Availability Zone failed

An issue got fixed, which prevented the restoration of VMs into different Availability Zones in the case of the original Availability Zone no longer being available.

Stopping workloadmgr service and all ongoing worker tasks

An issue got fixed, which lead to stall service jobs being left behind upon restart of workloadmgr services.

Mounting of LVM configured disks into the FRM failed

An issue got fixed, which prevented the correct mounting and access of Volumes partitioned and configured by LVM.

Upload of Backups failed intermittently using S3

An issue got fixed, which lead to a race condition between upload threads in the S3 fuse plugin, which led to backups failing during the upload phase.

Multipathing not enabled by default in the Data-Mover container used in RHOSP and Kolla Ansible Openstack

An issue got fixed, which lead to multipathing not being enabled in the Data-Mover container used by RHOSP and Kolla Ansible.

Upgrading to 4.1 HF1 will automatically activate multipathing where feasible.

An inaccessible S3 mount point got created when the S3 endpoint is not available during deployment or configuration

An issue got fixed, which lead to the creation of a Trilio mount point, even when the provided S3 backup target is not reachable during deployment or configuration.

The deployment will still succeed, but the tvault-object-store service will be in a failed state.

Trustee role not correctly inherited from user groups to users

An issue got fixed, which prevented the detection of the Trilio trustee role for a user, who had this role inherited from a user group.

The configurator always trying to use Keystone internal endpoint

An issue got fixed, during which the chosen endpoint type did not get honored for Keystone and the configurator always reached out to the Keystone internal endpoint.

The following value can be set in the api-paste ini file located under:

/etc/workloadmgr/api-paste.ini

[filter:authtoken] 
interface = internal

Afterward the wlm-workloads service needs to be restarted.

It is recommended to reconfigure the appliance to activate the fix

Disk integrity check failing with a false positive

An issue got identified, which leads to the disk integrity check failing, although there is no data loss.

Snapshots with a failed disk integrity check are currently no longer failing and instead show a warning in the log files about the failed disk integrity check.

A complete fix of the root cause is planned for 4.1 SP1.

Workload creation and Backup creation failed due to Latin characters like á

An issue got identified in which Latin characters like á did lead to a Workload not being created or a backup not succeeding.

This hotfix implements support of Latin characters for the following:

Calendar shown and used during workload creation for the job scheduler
Name and description of security groups

Full support for Latin characters comes in 4.1 SP1

Backups for multipath environments using the FC protocol failed

An issue got fixed, which prevented successful backups in environments using multipathing with the FC storage protocol.

Installing on RHOSP

The is the supported and recommended method to deploy and maintain any RHOSP installation. Trilio integrates natively into the RHOSP Director and manual deployment methods are not supported for RHOSP.

1. Prepare for deployment

1.1] Select 'backup target' type

Backup target storage is used to store backup images taken by Trilio and also associated configuration needs. The following backup target types are supported by Trilio:

Backup Target Types

Required Configuration

1.2] Clone triliovault-cfg-scripts repository

The overcloud-deploy command must already have been run successfully prior to this point and overcloud should be available. Perform the following steps for 'undercloud' node on an existing RHOSP environment:

All commands need to be run as user 'stack' on undercloud node

RHOSP 16.0 is not supported anymore as RedHat has officially stopped supporting it. However, Trilio maintained it for some time and stopped the support from 4.1HF11 onwards. The latest hotfix available for RHOSP16.0 is 41.HF10. Reach out to the Support team for any help.

Ensure that the Trilio appliance connected to this installation is on the latest Hotfix version. Failure to ensure this may lead to your installation not working as expected.
1. Refer to this doc :
Run the following command to clone the triliovault-cfg-scripts github repository:

If your backup target type is 'Ceph-based S3' with SSL, skip this step. Otherwise, access the Red Hat Director scripts according to the RHOSP version being used:
- RHOSP 13 - cd triliovault-cfg-scripts/redhat-director-scripts/rhosp13/
- RHOSP 16.1 - cd triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/
- RHOSP 16.2 - cd triliovault-cfg-scripts/redhat-director-scripts/rhosp16.2/
If your backup target is Ceph S3 with SSL and your SSL certificates are self-signed or authorized by private CA, you must provide the CA chain certificate to validate the SSL requests. Otherwise, skip this step. To do this:
1. Rename your CA chain cert file to s3-cert.pem.
2. Copy the renamed file into the following directory: triliovault-cfg-scripts/redhat-director-scripts/redhat-director-scripts/<RHOSP_RELEASE_Directory>/puppet/trilio/filesIf your overcloud deploy command uses any other deploy artifact through an environment file, then you must merge Trilio deploy artifact url and your url in a single file.
3. Then access the Red Hat Director scripts according to the version being used:
  - RHOSP 13 - cp s3-cert.pem /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp13/puppet/trilio/files/
  - RHOSP 16.1 - cp s3-cert.pem /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/puppet/trilio/files/
  - RHOSP 16.2 - cp s3-cert.pem /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.2/puppet/trilio/files/

From this point onwards in the documentation, only the following path will be used for examples: cd triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/

2] Upload Trilio-Puppet module

RHOSP 16.1

The following commands upload the Trilio puppet module to the overcloud registry. The upload only happens upon the next deployment.

Step 1: -

The output of the above command looks like the following.

Trilio puppet module is uploaded to overcloud as a swift deploy artifact with heat resource name 'DeployArtifactURLs'.

Step 2: - Check Trilio's Puppet module artifact file and ensure that it looks like the following:

Step 3: -

Firstly, check to make sure that your overcloud deploy environment files uses deploy artifacts. To do this check string DeployArtifactURLs in your environment files (only those mentioned in the overcloud deploy command with -e option). If you find any environment file with the -e option, then your deploy command is using deploy artifacts.
If your deploy command is using deploy artifact, you must merge all deploy artifacts in a single file. For example, if your artifact file path is /home/stack/templates/user-artifacts.yaml, then perform the following steps to merge both urls in single file, which is passed to the overcloud deploy command with the -e option.

3] Update overcloud roles data file to include Trilio services

Trilio contains multiple services. Add these services to your roles_data.yaml.

In the case of uncustomized roles_data.yaml can the default file be found on the undercloud at:

/usr/share/openstack-tripleo-heat-templates/roles_data.yaml

Add the following services to the roles_data.yaml

All commands need to be run as user 'stack'

3.1] Add Trilio Datamover Api Service to role data file

This service needs to share the same role as the keystone and database service. In case of the pre-defined roles will these services run on the role Controller. In case of custom-defined roles, it is necessary to use the same role where OS::TripleO::Services::Keystone service installed.

Add the following line to the identified role:

3.2] Add Trilio Datamover Service to role data file

This service needs to share the same role as the nova-compute service. In case of the pre-defined roles will the nova-compute service run on the role Compute. In case of custom-defined roles, it is necessary to use the role that the nova-compute service is using.

Add the following line to the identified role:

4] Prepare Trilio container images

All commands need to be run as user 'stack'

Trilio containers are pushed to 'RedHat Container Registry'. Registry URL: registry.connect.redhat.com Container pull URLs are given below.

Please note that using the hotfix containers requires that the Trilio Appliance is getting upgraded to the desired hotfix level as well.

Refer to the word <HOTFIX-TAG-VERSION> as 4.1.94-hotfix-16 in the below sections

RHOSP 13

RHOSP 16.1

RHOSP 16.2

There are three registry methods available in the RedHat OpenStack Platform.

Remote Registry
Local Registry
Satellite Server

4.1] Remote Registry

Follow this section when 'Remote Registry' is used.

In this method, container images get downloaded directly on overcloud nodes during overcloud deploy/update command execution. User can set the remote registry to the RedHat registry or any other private registry that he wants to use. The user needs to provide credentials for the registry in containers-prepare-parameter.yaml file.

Make sure other OpenStack service images are also using the same method to pull container images. If it's not the case you can not use this method.
Populate containers-prepare-parameter.yaml with content like following. Important parameters are 'push_destination: false', ContainerImageRegistryLogin: true and registry credentials. Trilio container images are published to the registry registry.connect.redhat.com Credentials of registry 'registry.redhat.io' will work for registry.connect.redhat.com registry too.

Note: This file - containers-prepare-parameter.yaml

Note: File 'containers-prepare-parameter.yaml' gets created as output of command 'openstack tripleo container image prepare'. Refer above document by RedHat

3. Make sure you have network connectivity to the above registries from all overcloud nodes. Otherwise, image pull operation will fail.

4. Populate the trilio_env.yaml with Trilio container image URLs:

Trilio Datamover container
Trilio Datamover API container
Trilio Horizon Plugin

trilio_env.yaml will be available in

4.2] Local Registry

Follow this section when 'local registry' is used on the undercloud.

In this case, it is necessary to push the Trilio containers to the undercloud registry. Trilio provides shell scripts that will pull the containers from 'registry.connect.redhat.com' and push them to the undercloud and update the trilio_env.yaml.

RHOSP13

Verify the changes

RHOSP16.1

Verify the changes:

RHOSP16.2

Verify the changes

The changes can be verified using the following commands.

4.3] Red Hat Satellite Server

Follow this section when a Satellite Server is used for the container registry.

Populate the trilio_env.yaml with container urls.

RHOSP 13

RHOSP16.1

RHOSP16.2

5] Provide environment details in trilio-env.yaml

Provide backup target details and other necessary details in the provided environment file. This environment file will be used in the overcloud deployment to configure Trilio components. Container image names have already been populated in the preparation of the container images. Still it is recommended to verify the container URLs.

The following information are required additionally:

Network for the datamover api
datamover password
Backup target type {nfs/s3}
In case of NFS
- list of NFS Shares
- NFS options
In case of S3
- S3 type {amazon_s3/ceph_s3}
- S3 Access key
- S3 Secret key
- S3 Region name
- S3 Bucket
- S3 Endpoint URL
- S3 Signature Version
- S3 Auth Version
- S3 SSL Enabled {true/false}
- S3 SSL Cert

Use ceph_s3 for any non-aws S3 backup targets.

6. Advanced Settings/Configuration

6.1 Haproxy customized configuration for Trilio dmapi service __

The existing default haproxy configuration works fine with most of the environments. Only when timeout issues with the dmapi are observed or other reasons are known, change the configuration as described here.

Following is the haproxy conf file location on haproxy nodes of the overcloud. Trilio datamover api service haproxy configuration gets added to this file.

Trilio datamover haproxy default configuration from the above file looks as follows:

The user can change the following configuration parameter values.

To change these default values, you need to do the following steps. i) On the undercloud node, open the following file for edit (Edit <RHOSP_RELEASE> with your cloud's release information. Valid values are - rhosp13, rhosp16, rhosp16.1)

For RHOSP13

For RHOSP16.0

For RHOSP16.1

For RHOSP16.2

ii) Search the following entries and edit as required

iii) Save the changes.

7] Deploy overcloud with trilio environment

Use the following heat environment file and roles data file in overcloud deploy command:

trilio_env.yaml
roles_data.yaml
Use the correct Trilio endpoint map file as per available Keystone endpoint configuration
1. Instead of tls-endpoints-public-dns.yaml file, use environments/trilio_env_tls_endpoints_public_dns.yaml
2. Instead of tls-endpoints-public-ip.yaml file, useenvironments/trilio_env_tls_endpoints_public_ip.yaml
3. Instead of tls-everywhere-endpoints-dns.yaml file, useenvironments/trilio_env_tls_everywhere_dns.yaml

To include new environment files use '-e' option and for roles data file use '-r' option. An example overcloud deploy command is shown below:

8] Verify deployment

If the containers are in restarting state or not listed by the following command then your deployment is not done correctly. Please recheck if you followed the complete documentation.

8.1] On Controller node

Make sure Trilio dmapi and horizon containers are in a running state and no other Trilio container is deployed on controller nodes. When the role for these containers is not "controller" check on respective nodes according to configured roles_data.yaml.

Verify the haproxy configuration under:

8.2] On Compute node

Make sure Trilio datamover container is in running state and no other Trilio container is deployed on compute nodes.

8.3] On the node with Horizon service

Make sure horizon container is in running state. Please note that 'Horizon' container is replaced with Trilio Horizon container. This container will have latest OpenStack horizon + Trilio's horizon plugin.

9] Additional Steps on Trilio Appliance

9.1] Change the nova user id on the Trilio Nodes

In RHOSP, nova user id on nova-compute docker container is set to '42436'. The 'nova' user id on the Trilio nodes needs to be set the same. Do the following steps on all Trilio nodes:

Download the shell script that will change the user id
Assign executable permissions
Execute the script
Verify that nova user and group id have changed to '42436'

10] Troubleshooting for overcloud deployment failures

Trilio components will be deployed using puppet scripts.

Post Installation Health-Check

After the installation and configuration of Trilio for Openstack did succeed the following steps can be done to verify that the Trilio installation is healthy.

Verify the Trilio Appliance

Verify the services are up

Trilio is using 4 main services on the Trilio Appliance:

wlm-api
wlm-scheduler
wlm-workloads
wlm-cron

systemctl | grep wlm
  wlm-api.service          loaded active running   workloadmanager api service
  wlm-cron.service         loaded active running   Cluster Controlled wlm-cron
  wlm-scheduler.service    loaded active running   Cluster Controlled wlm-scheduler
  wlm-workloads.service    loaded active running   workloadmanager workloads service

Those can be verified to be up and running using the systemctl status command.

systemctl status wlm-api
######
● wlm-api.service - workloadmanager api service
   Loaded: loaded (/etc/systemd/system/wlm-api.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2021-11-02 19:41:19 UTC; 2 months 21 days ago
 Main PID: 4688 (workloadmgr-api)
   CGroup: /system.slice/wlm-api.service
           ├─4688 /home/stack/myansible/bin/python3 /home/stack/myansible/bin/workloadmgr-api --config-file=/etc/workloadmgr/workloadmgr.conf

systemctl status wlm-scheduler
######
● wlm-scheduler.service - Cluster Controlled wlm-scheduler
   Loaded: loaded (/etc/systemd/system/wlm-scheduler.service; disabled; vendor preset: disabled)
  Drop-In: /run/systemd/system/wlm-scheduler.service.d
           └─50-pacemaker.conf
   Active: active (running) since Sat 2022-01-22 13:49:28 UTC; 1 day 23h ago
 Main PID: 9342 (workloadmgr-sch)
   CGroup: /system.slice/wlm-scheduler.service
           └─9342 /home/stack/myansible/bin/python3 /home/stack/myansible/bin/workloadmgr-scheduler --config-file=/etc/workloadmgr/workloadmgr.conf

systemctl status wlm-workloads
######
● wlm-workloads.service - workloadmanager workloads service
   Loaded: loaded (/etc/systemd/system/wlm-workloads.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2021-11-02 19:51:05 UTC; 2 months 21 days ago
 Main PID: 606 (workloadmgr-wor)
   CGroup: /system.slice/wlm-workloads.service
           ├─ 606 /home/stack/myansible/bin/python3 /home/stack/myansible/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf

systemctl status wlm-cron
######
● wlm-cron.service - Cluster Controlled wlm-cron
   Loaded: loaded (/etc/systemd/system/wlm-cron.service; disabled; vendor preset: disabled)
  Drop-In: /run/systemd/system/wlm-cron.service.d
           └─50-pacemaker.conf
   Active: active (running) since Sat 2022-01-22 13:49:28 UTC; 1 day 23h ago
 Main PID: 9209 (workloadmgr-cro)
   CGroup: /system.slice/wlm-cron.service
           ├─9209 /home/stack/myansible/bin/python3 /home/stack/myansible/bin/workloadmgr-cron --config-file=/etc/workloadmgr/workloadmgr.conf

Check the Trilio pacemaker and nginx cluster

The second component to check the Trilio Appliance's health is the nginx and pacemaker cluster.

pcs status
######
Cluster name: triliovault

WARNINGS:
Corosync and pacemaker node names do not match (IPs used in setup?)

Stack: corosync
Current DC: TVM1 (version 1.1.21-4.el7-f14e36fd43) - partition with quorum
Last updated: Mon Jan 24 13:42:01 2022
Last change: Tue Nov  2 19:07:04 2021 by root via crm_resource on TVM2

3 nodes configured
9 resources configured

Online: [ TVM1 TVM2 TVM3 ]

Full list of resources:

 virtual_ip     (ocf::heartbeat:IPaddr2):       Started TVM2
 virtual_ip_public      (ocf::heartbeat:IPaddr2):       Started TVM2
 virtual_ip_admin       (ocf::heartbeat:IPaddr2):       Started TVM2
 virtual_ip_internal    (ocf::heartbeat:IPaddr2):       Started TVM2
 wlm-cron       (systemd:wlm-cron):     Started TVM2
 wlm-scheduler  (systemd:wlm-scheduler):        Started TVM2
 Clone Set: lb_nginx-clone [lb_nginx]
     Started: [ TVM2 ]
     Stopped: [ TVM1 TVM3 ]

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

Verify API connectivity of the Trilio Appliance

Checking the availability of the Trilio API on the chosen endpoints is recommended.

The following example curl command lists the available workload-types and verifies that the connection is available and working:

curl http://10.10.2.34:8780/v1/8e16700ae3614da4ba80a4e57d60cdb9/workload_types/detail -X GET -H "X-Auth-Project-Id: admin" -H "User-Agent: python-workloadmgrclient" -H "Accept: application/json" -H "X-Auth-Token: gAAAAABe40NVFEtJeePpk1F9QGGh1LiGnHJVLlgZx9t0HRrK9rC5vqKZJRkpAcW1oPH6Q9K9peuHiQrBHEs1-g75Na4xOEESR0LmQJUZP6n37fLfDL_D-hlnjHJZ68iNisIP1fkm9FGSyoyt6IqjO9E7_YVRCTCqNLJ67ZkqHuJh1CXwShvjvjw

Please check the API guide for more commands and how to generate the X-Auth-Token.

Verify Trilio components on OpenStack

On OpenStack Ansible

Datamover-API service (dmapi)

The dmapi service has its own Keystone endpoints, which should be checked in addition to the actual service status.

root@ansible:~# openstack endpoint list | grep dmapi
| 190db2ce033e44f89de73abcbf12804e | US-WEST-2 | dmapi          | datamover    | True    | public    | https://osa-victoria-ubuntu20-2.triliodata.demo:8784/v2                    |
| dec1a323791b49f0ac7901a2dc806ee2 | US-WEST-2 | dmapi          | datamover    | True    | admin     | http://10.10.10.154:8784/v2                                                |
| f8c4162c9c1246ffb0190d0d093c48af | US-WEST-2 | dmapi          | datamover    | True    | internal  | http://10.10.10.154:8784/v2                                                |

root@ansible:~# curl http://10.10.10.154:8784
{"versions": [{"id": "v2.0", "status": "SUPPORTED", "version": "", "min_version": "", "updated": "2011-01-21T11:33:21Z", "links": [{"rel": "self", "href": "h

In order to check the dmapi service go to dmapi container which is residing on controller nodes and run below command

lxc-attach -n <dmapi-container-name>  (go to dmapi conatiner)
root@controller-dmapi-container-08df1e06:~# systemctl status tvault-datamover-api.service
● tvault-datamover-api.service - TrilioData DataMover API service
     Loaded: loaded (/lib/systemd/system/tvault-datamover-api.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-01-12 11:53:39 UTC; 1 day 17h ago
   Main PID: 23888 (dmapi-api)
      Tasks: 289 (limit: 57729)
     Memory: 607.7M
     CGroup: /system.slice/tvault-datamover-api.service
             ├─23888 /usr/bin/python3 /usr/bin/dmapi-api
             ├─23893 /usr/bin/python3 /usr/bin/dmapi-api
             ├─23894 /usr/bin/python3 /usr/bin/dmapi-api
             ├─23895 /usr/bin/python3 /usr/bin/dmapi-api
             ├─23896 /usr/bin/python3 /usr/bin/dmapi-api
             ├─23897 /usr/bin/python3 /usr/bin/dmapi-api
             ├─23898 /usr/bin/python3 /usr/bin/dmapi-api
             ├─23899 /usr/bin/python3 /usr/bin/dmapi-api
             ├─23900 /usr/bin/python3 /usr/bin/dmapi-api
             ├─23901 /usr/bin/python3 /usr/bin/dmapi-api
             ├─23902 /usr/bin/python3 /usr/bin/dmapi-api
             ├─23903 /usr/bin/python3 /usr/bin/dmapi-api
             ├─23904 /usr/bin/python3 /usr/bin/dmapi-api
             ├─23905 /usr/bin/python3 /usr/bin/dmapi-api
             ├─23906 /usr/bin/python3 /usr/bin/dmapi-api
             ├─23907 /usr/bin/python3 /usr/bin/dmapi-api
             └─23908 /usr/bin/python3 /usr/bin/dmapi-api

Jan 12 11:53:39 controller-dmapi-container-08df1e06 systemd[1]: Started TrilioData DataMover API service.
Jan 12 11:53:40 controller-dmapi-container-08df1e06 dmapi-api[23888]: Could not load
Jan 12 11:53:40 controller-dmapi-container-08df1e06 dmapi-api[23888]: Could not load

Datamover service (tvault-contego)

The datamover service is running on each compute node. Logging to compute node and run below command

root@compute:~# systemctl status tvault-contego
● tvault-contego.service - Tvault contego
     Loaded: loaded (/etc/systemd/system/tvault-contego.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2022-01-14 05:45:19 UTC; 2s ago
   Main PID: 1489651 (python3)
      Tasks: 19 (limit: 67404)
     Memory: 6.7G (max: 10.0G)
     CGroup: /system.slice/tvault-contego.service
             ├─ 998543 /bin/qemu-nbd -c /dev/nbd45 --object secret,id=sec0,data=payload-1234 --image-opts driver=qcow2,encrypt.format=luks,encrypt.key-secret=sec0,file>
             ├─ 998772 /bin/qemu-nbd -c /dev/nbd73 --object secret,id=sec0,data=payload-1234 --image-opts driver=qcow2,encrypt.format=luks,encrypt.key-secret=sec0,file>
             ├─ 998931 /bin/qemu-nbd -c /dev/nbd100 --object secret,id=sec0,data=payload-1234 --image-opts driver=qcow2,encrypt.format=luks,encrypt.key-secret=sec0,fil>
             ├─ 999147 /bin/qemu-nbd -c /dev/nbd35 --object secret,id=sec0,data=payload-1234 --image-opts driver=qcow2,encrypt.format=luks,encrypt.key-secret=sec0,file>
             ├─1371322 /bin/qemu-nbd -c /dev/nbd63 --object secret,id=sec0,data=payload-test1 --image-opts driver=qcow2,encrypt.format=luks,encrypt.key-secret=sec0,fil>
             ├─1371524 /bin/qemu-nbd -c /dev/nbd91 --object secret,id=sec0,data=payload-test1 --image-opts driver=qcow2,encrypt.format=luks,encrypt.key-secret=sec0,fil>
             └─1489651 /openstack/venvs/nova-22.3.1/bin/python3 /usr/bin/tvault-contego --config-file=/etc/nova/nova.conf --config-file=/etc/tvault-contego/tvault-cont>

Jan 14 05:45:19 compute systemd[1]: Started Tvault contego.
Jan 14 05:45:20 compute sudo[1489653]:     nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/openstack/venvs/nova-22.3.1/bin/nova-rootwrap /etc/nova/rootwrap.conf umou>
Jan 14 05:45:20 compute sudo[1489653]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jan 14 05:45:21 compute python3[1489655]: umount: /var/triliovault-mounts/VHJpbGlvVmF1bHQ=: no mount point specified.
Jan 14 05:45:21 compute sudo[1489653]: pam_unix(sudo:session): session closed for user root
Jan 14 05:45:21 compute tvault-contego[1489651]: 2022-01-14 05:45:21.499 1489651 INFO __main__ [req-48c32a39-38d0-45b9-9852-931e989133c6 - - - - -] CPU Control group m>
Jan 14 05:45:21 compute tvault-contego[1489651]: 2022-01-14 05:45:21.499 1489651 INFO __main__ [req-48c32a39-38d0-45b9-9852-931e989133c6 - - - - -] I/O Control Group m>
lines 1-22/22 (END)

On Kolla Ansible OpenStack

Datamover-API service (dmapi)

The dmapi service has its own Keystone endpoints, which should be checked in addition to the actual service status.

root@ansible:~# openstack endpoint list | grep dmapi
| 190db2ce033e44f89de73abcbf12804e | US-WEST-2 | dmapi          | datamover    | True    | public    | https://osa-victoria-ubuntu20-2.triliodata.demo:8784/v2                    |
| dec1a323791b49f0ac7901a2dc806ee2 | US-WEST-2 | dmapi          | datamover    | True    | admin     | http://10.10.10.154:8784/v2                                                |
| f8c4162c9c1246ffb0190d0d093c48af | US-WEST-2 | dmapi          | datamover    | True    | internal  | http://10.10.10.154:8784/v2                                                |

root@ansible:~# curl http://10.10.10.154:8784
{"versions": [{"id": "v2.0", "status": "SUPPORTED", "version": "", "min_version": "", "updated": "2011-01-21T11:33:21Z", "links": [{"rel": "self", "href": "h

Run the following command on “nova-api” nodes and make sure “triliovault_datamover_api” container is in started state.

[root@controller ~]# docker ps | grep triliovault_datamover_api
3f979c15cedc   trilio/centos-binary-trilio-datamover-api:4.2.50-victoria                     "dumb-init --single-…"   3 days ago    Up 3 days                         triliovault_datamover_api

Datamover service (tvault-contego)

Run the following command on "nova-compute" nodes and make sure the container is in a started state.

[root@compute1 ~]# docker ps | grep triliovault_datamover
2f1ece820a59   trilio/centos-binary-trilio-datamover:4.2.50-victoria                        "dumb-init --single-…"   3 days ago    Up 3 days                        triliovault_datamover

Trilio Horizon integration

Run the following command on horizon nodes and make sure the container is in a started state.

[root@controller ~]# docker ps | grep horizon
4a004c786d47   trilio/centos-binary-trilio-horizon-plugin:4.2.50-victoria                    "dumb-init --single-…"   3 days ago    Up 3 days (unhealthy)             horizon

On Canonical OpenStack

Run the following command on MAAS nodes and make sure all trilio units like trilio-data-mover, trilio-dm-api, trilio-horizon-plugin, trilio-wlmare in active state

root@jujumaas:~# juju status | grep trilio
trilio-data-mover       4.2.51   active       3  trilio-data-mover       jujucharms    9  ubuntu
trilio-dm-api           4.2.51   active       1  trilio-dm-api           jujucharms    7  ubuntu
trilio-horizon-plugin   4.2.51   active       1  trilio-horizon-plugin   jujucharms    6  ubuntu
trilio-wlm              4.2.51   active       1  trilio-wlm              jujucharms    9  ubuntu
  trilio-data-mover/8        active    idle            172.17.1.5                         Unit is ready
  trilio-data-mover/6        active    idle            172.17.1.6                         Unit is ready
  trilio-data-mover/7*       active    idle            172.17.1.7                         Unit is ready
  trilio-horizon-plugin/2*   active    idle            172.17.1.16                        Unit is ready
trilio-dm-api/2*             active    idle   1/lxd/4  172.17.1.27     8784/tcp           Unit is ready
trilio-wlm/2*                active    idle   7        172.17.1.28     8780/tcp           Unit is ready

On Red Hat OpenStack and TripleO

On controller node

Make sure the Trilio dmapi and horizon containers (shown below) are in a running state and no other Trilio container is deployed on controller nodes. If the containers are in restarting state or not listed by the following command then your deployment is not done correctly. Please note that the 'Horizon' container is replaced with the Trilio Horizon container. This container will have the latest OpenStack horizon + Trilio's horizon plugin.

On rhosp13 OS:	docker ps | grep trilio-
On other (rhosp16 onwards/tripleo) :	podman ps | grep trilio-

[root@overcloudtrain1-controller-0 heat-admin]# podman ps | grep trilio-
e3530d6f7bec  ucqa161.ctlplane.trilio.local:8787/trilio/trilio-datamover-api:4.2.47-rhosp16.1           kolla_start           2 weeks ago   Up 2 weeks ago          trilio_dmapi
f93f7019f934  ucqa161.ctlplane.trilio.local:8787/trilio/trilio-horizon-plugin:4.2.47-rhosp16.1          kolla_start           2 weeks ago   Up 2 weeks ago          horizon

On compute node

Make sure the Trilio datamover container (shown below) is in a running state and no other Trilio container is deployed on compute nodes. If the containers are in restarting state or not listed by the following command then your deployment is not done correctly.

On rhosp13 OS:	docker ps | grep trilio-
On other (rhosp/tripleo) :	podman ps | grep trilio-

[root@overcloudtrain3-novacompute-1 heat-admin]# podman ps | grep trilio-
4419b02e075c  undercloud162.ctlplane.trilio.local:8787/trilio/trilio-datamover:dev-osp16.2-1-rhosp16.2       kolla_start  2 days ago   Up 27 seconds ago          trilio_datamover

On overcloud

Please check dmapi endpoints on overcloud node.

 (overcloudtrain1) [stack@ucqa161 ~]$ openstack endpoint list | grep datamover
| 218b2f92569a4d259839fa3ea4d6103a | regionOne | dmapi          | datamover      | True    | internal  | https://overcloudtrain1internalapi.trilio.local:8784/v2                    |
| 4702c51aa5c24bed853e736499e194e2 | regionOne | dmapi          | datamover      | True    | public    | https://overcloudtrain1.trilio.local:13784/v2                              |
| c8169025eb1e4954ab98c7abdb0f53f6 | regionOne | dmapi          | datamover      | True    | admin     | https://overcloudtrain1internalapi.trilio.local:8784/v2

Uninstalling from RHOSP

Clean Trilio Datamover API service

The following steps need to be run on all nodes, which have the Trilio Datamover API service running. Those nodes can be identified by checking the roles_data.yaml for the role that contains the entry OS::TripleO::Services::TrilioDatamoverApi.

Once the role that runs the Trilio Datamover API service has been identified will the following commands clean the nodes from the service.

Run all commands as root or user with sudo permissions.

Stop trilio_dmapi container.

# For RHOSP13
systemctl disable tripleo_trilio_dmapi.service
systemctl stop tripleo_trilio_dmapi.service
docker stop trilio_dmapi

# For RHOSP16 onwards
systemctl disable tripleo_trilio_dmapi.service
systemctl stop tripleo_trilio_dmapi.service
podman stop trilio_dmapi

Remove trilio_dmapi container.

# For RHOSP13
docker rm trilio_dmapi
docker rm trilio_datamover_api_init_log
docker rm trilio_datamover_api_db_sync

# For RHOSP16 onwards
podman rm trilio_dmapi
podman rm trilio_datamover_api_init_log
podman rm trilio_datamover_api_db_sync

Clean Trilio Datamover API service conf directory.

rm -rf /var/lib/config-data/puppet-generated/triliodmapi
rm /var/lib/config-data/puppet-generated/triliodmapi.md5sum

Clean Trilio Datamover API service log directory.

rm -rf /var/log/containers/trilio-datamover-api/

Clean Trilio Datamover Service

The following steps need to be run on all nodes, which have the Trilio Datamover service running. Those nodes can be identified by checking the roles_data.yaml for the role that contains the entry OS::TripleO::Services::TrilioDatamover.

Once the role that runs the Trilio Datamover service has been identified will the following commands clean the nodes from the service.

Run all commands as root or user with sudo permissions.

Stop trilio_datamover container.

# For RHOSP13
docker stop trilio_datamover

# For RHOSP16 onwards
systemctl disable tripleo_trilio_datamover.service
systemctl stop tripleo_trilio_datamover.service
podman stop trilio_datamover

Remove trilio_datamover container.

# For RHOSP13
docker rm trilio_datamover

# For RHOSP16 onwards
podman rm trilio_datamover

Unmount Trilio Backup Target on compute host.

## Following steps applicable for all supported RHOSP releases.

# Check triliovault backup target mount point
mount | grep trilio

# Unmount it
-- If it's NFS	(COPY UUID_DIR from your compute host using above command)
umount /var/lib/nova/triliovault-mounts/<UUID_DIR>

-- If it's S3
umount /var/lib/nova/triliovault-mounts

# Verify that it's unmounted		
mount | grep trilio
	
df -h  | grep trilio

# Remove mount point directory after verifying that backup target unmounted successfully.
# Otherwise actual data from backup target may get cleaned.	

rm -rf /var/lib/nova/triliovault-mounts

Clean Trilio Datamover service conf directory.

rm -rf /var/lib/config-data/puppet-generated/triliodm/
rm /var/lib/config-data/puppet-generated/triliodm.md5sum

Clean log directory of Trilio Datamover service.

rm -rf /var/log/containers/trilio-datamover/

Clean Trilio haproxy resources

The following steps need to be run on all nodes, which have the haproxy service running. Those nodes can be identified by checking the roles_data.yaml for the role that contains the entry OS::TripleO::Services::HAproxy.

Once the role that runs the haproxy service has been identified will the following commands clean the nodes from all Trilio resources.

Run all commands as root or user with sudo permissions.

Edit the following file inside the haproxy container and remove all Trilio entries.

/var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg

An example of these entries is given below.

listen trilio_datamover_api
  bind 172.25.3.60:13784 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 172.25.3.60:8784 transparent
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  http-request set-header X-Forwarded-Port %[dst_port]
  option httpchk
  option httplog
  server overcloud-controller-0.internalapi.localdomain 172.25.3.59:8784 check fall 5 inter 2000 rise 2

Restart the haproxy container once all edits have been done.

# For RHOSP13
docker restart haproxy-bundle-docker-0

# For RHOSP16 onwards
podman restart haproxy-bundle-podman-0

Clean Trilio Keystone resources

Trilio registers services and users in Keystone. Those need to be unregistered and deleted.

openstack service delete dmapi
openstack user delete dmapi

Clean Trilio database resources

Trilio creates a database for the dmapi service. This database needs to be cleaned.

## On RHOSP13, run following command on node where database service runs
docker exec -ti -u root galera-bundle-docker-0 mysql -u root

## On RHOSP16
podman exec -it galera-bundle-podman-0 mysql -u root

Run the following SQL statements to clean the database.

## Clean database
DROP DATABASE dmapi;

## Clean dmapi user
=> List 'dmapi' user accounts
MariaDB [mysql]> select user, host from mysql.user where user='dmapi';
+-------+-------------+
| user  | host        |
+-------+-------------+
| dmapi | 172.25.2.10 |
| dmapi | 172.25.2.8  |
+-------+-------------+
2 rows in set (0.00 sec)

=> Delete those user accounts
MariaDB [mysql]> DROP USER dmapi@172.25.2.10;
Query OK, 0 rows affected (0.82 sec)

MariaDB [mysql]> DROP USER dmapi@172.25.2.8;
Query OK, 0 rows affected (0.05 sec)

=> Verify that dmapi user got cleaned
MariaDB [mysql]> select user, host from mysql.user where user='dmapi';
Empty set (0.00 sec)

Revert overcloud deploy command

Remove the following entries from roles_data.yaml used in the overcloud deploy command.

OS::TripleO::Services::TrilioDatamoverApi
OS::TripleO::Services::TrilioDatamover

In the case that the overcloud deploy command used prior to the deployment of Trilio is still available, it can directly be used.

Follow these steps to clean the overcloud deploy command from all Trilio entries.

Remove trilio_env.yaml entry
Remove trilio endpoint map file Replace with original map file if existing

Revert back to original RHOSP Horizon container

Run the cleaned overcloud deploy command.

Destroy the Trilio VM Cluster

List all VMs running on the KVM node

virsh list

Destroy the Trilio VMs

virsh destroy <Trilio VM Name or ID>

Undefine the Trilio VMs

virsh undefine <Trilio VM name>

Delete the TrlioVault VM disk from KVM Host storage

Upgrading on RHOSP

0] Pre-requisites

Please ensure the following points are met before starting the upgrade process:

No Snapshot or Restore is running
Global job scheduler is disabled
wlm-cron is disabled on the Trilio Appliance

Deactivating the wlm-cron service

The following sets of commands will disable the wlm-cron service and verify that it is has been completly shut-down.

[root@TVM2 ~]# pcs resource disable wlm-cron
[root@TVM2 ~]# systemctl status wlm-cron
● wlm-cron.service - workload's scheduler cron service
   Loaded: loaded (/etc/systemd/system/wlm-cron.service; disabled; vendor preset                                                                                                                                                                                                                                                                                                                                                           : disabled)
   Active: inactive (dead)

Jun 11 08:27:06 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:06 - INFO - 1...t
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 140686268624368 Child 11389 ki...5
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - 1...5
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: Shutting down thread pool
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - S...l
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: Stopping the threads
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - S...s
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: All threads are stopped succes...y
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - A...y
Jun 11 08:27:09 TVM2 systemd[1]: Stopped workload's scheduler cron service.
Hint: Some lines were ellipsized, use -l to show in full.
[root@TVM2 ~]# pcs resource show wlm-cron
 Resource: wlm-cron (class=systemd type=wlm-cron)
  Meta Attrs: target-role=Stopped
  Operations: monitor interval=30s on-fail=restart timeout=300s (wlm-cron-monito                                                                                                                                                                                                                                                                                                                                                           r-interval-30s)
              start interval=0s on-fail=restart timeout=300s (wlm-cron-start-int                                                                                                                                                                                                                                                                                                                                                           erval-0s)
              stop interval=0s timeout=300s (wlm-cron-stop-interval-0s)
[root@TVM2 ~]# ps -ef | grep -i workloadmgr-cron
root     15379 14383  0 08:27 pts/0    00:00:00 grep --color=auto -i workloadmgr                                                                                                                                                                                                                                                                                                                                                           -cron

1.] [On Undercloud node] Clone latest Trilio repository and upload Trilio puppet module

All commands need to be run as user 'stack' on undercloud node

The Trilio appliance connected to this installation needs to be of version 4.1 HF10

1.1] Clone Trilio cfg scripts repository

cd /home/stack
mv triliovault-cfg-scripts triliovault-cfg-scripts-old
git clone -b hotfix-13-TVO/4.1 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/

Separate directories are created as per Redhat OpenStack release under 'triliovault-cfg-scripts/redhat-director-scripts/' directory. Use all scripts/templates from respective directory. For ex, if your RHOSP release is 13, then use scripts/templates from 'triliovault-cfg-scripts/redhat-director-scripts/rhosp13' directory only.

Available RHOSP_RELEASE___DIRECTORY values are:

rhosp13 rhosp16.1 rhosp16.2

1.2] If backup target type is 'Ceph based S3' with SSL:

If your backup target is ceph S3 with SSL and SSL certificates are self signed or authorized by private CA, then user needs to provide CA chain certificate to validate the SSL requests. For that, user needs to rename his ca chain cert file to 's3-cert.pem' and copy it into the puppet directory of the right release.

cp s3-cert.pem /home/stack/triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/puppet/trilio/files/

2] Upload Trilio puppet module

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/scripts/
./upload_puppet_module.sh

## Output of above command looks like following.
Creating tarball...
Tarball created.
Creating heat environment file: /home/stack/.tripleo/environments/puppet-modules-url.yaml
Uploading file to swift: /tmp/puppet-modules-8Qjya2X/puppet-modules.tar.gz
+-----------------------+---------------------+----------------------------------+
| object                | container           | etag                             |
+-----------------------+---------------------+----------------------------------+
| puppet-modules.tar.gz | overcloud-artifacts | 368951f6a4d39cfe53b5781797b133ad |
+-----------------------+---------------------+----------------------------------+

## Above command creates following file.
ls -ll /home/stack/.tripleo/environments/puppet-modules-url.yaml

3] Update overcloud roles data file to include Trilio services

Trilio has two services as explained below. You need to add these two services to your roles_data.yaml. If you do not have customized roles_data file, you can find your default roles_data.yaml file at /usr/share/openstack-tripleo-heat-templates/roles_data.yaml on undercloud.

You need to find that role_data file and edit it to add the following Trilio services.

i) Trilio Datamover Api Service:

Service Entry in roles_data yaml: OS::TripleO::Services::TrilioDatamoverApi

This service needs to be co-located with database and keystone services. That said, you need to add this service on the same role as of keystone and database service.

Typically this service should be deployed on controller nodes where keystone and database runs. If you are using RHOSP's pre-defined roles, you need to addOS::TripleO::Services::TrilioDatamoverApiservice to Controller role.

ii) Trilio Datamover Service: Service Entry in roles_data yaml: OS::TripleO::Services::TrilioDatamover This service should be deployed on role where nova-compute service is running.

If you are using RHOSP's pre-defined roles, you need to add our OS::TripleO::Services::TrilioDatamover service to Compute role.

If you have defined your custom roles, then you need to identify the role name where in 'nova-compute' service is running and then you need to add 'OS::TripleO::Services::TrilioDatamover' service to that role.

4] Prepare latest Trilio container images

All commands need to be run as user 'stack'

Refer to the word <HOTFIX-TAG-VERSION> as 4.1.94-hotfix-16 in the below sections

Trilio containers are pushed to 'RedHat Container Registry'. Registry URL is 'registry.connect.redhat.com'. The Trilio container URLs are as follows:

4.1] available container images

RHOSP 13

Trilio Datamove container:        registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp13
Trilio Datamover Api Container:   registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp13
Trilio horizon plugin:            registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp13

RHOSP 16.1

Trilio Datamover container:       registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1
Trilio Datamover Api Container:   registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1
Trilio horizon plugin:            registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1

RHOSP 16.2

Trilio Datamover container:       registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.2
Trilio Datamover Api Container:   registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.2
Trilio horizon plugin:            registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.2

There are three registry methods available in RedHat OpenStack Platform.

Remote Registry
Local Registry
Satellite Server

4.2] Remote Registry

Please refer to the following overview to see which containers are available.

Follow this section when 'Remote Registry' is used.

For this method, it is not necessary to pull the containers in advance. It is only necessary to populate the trilio_env.yaml file with the Trilio container URLs from Redhat registry.

Populate the trilio_env.yaml with container URLs for:

Trilio Datamover container
Trilio Datamover api container
Trilio Horizon Plugin

trilio_env.yaml will be available in __triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments

Example

# For RHOSP13
$ grep 'Image' trilio_env.yaml
   DockerTrilioDatamoverImage: registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp13
   DockerTrilioDmApiImage: registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp13
   DockerHorizonImage: registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp13

# For RHOSP16.1
$ grep 'Image' trilio_env.yaml
   DockerTrilioDatamoverImage: registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1
   DockerTrilioDmApiImage: registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1
   ContainerHorizonImage: registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1
   
# For RHOSP16.2
$ grep 'Image' trilio_env.yaml
   DockerTrilioDatamoverImage: registry.connect.redhat.com/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.2
   DockerTrilioDmApiImage: registry.connect.redhat.com/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.2
   ContainerHorizonImage: registry.connect.redhat.com/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.2

4.2] Local Registry

Please refer to the this overview to see which containers are available.

Follow this section when 'local registry' is used on the undercloud.

In this case it is necessary to push the Trilio containers to the undercloud registry. Trilio provides shell scripts which will pull the containers from 'registry.connect.redhat.com' and push them to the undercloud and updates the trilio_env.yaml.

RHOSP13

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp13/scripts/
./prepare_trilio_images.sh <undercloud_ip/hostname> <HOTFIX-TAG-VERSION>-rhosp13

Verify the changes

$ grep 'Image' ../environments/trilio_env.yaml
   DockerTrilioDatamoverImage: 172.25.2.2:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp13
   DockerTrilioDmApiImage: 172.25.2.2:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp13
   DockerHorizonImage: 172.25.2.2:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp13

RHOSP16.1

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/scripts/
sudo ./prepare_trilio_images.sh <undercloud_ip/hostname> <HOTFIX-TAG-VERSION>-rhosp16.1

Verify the changes:

$ grep 'Image' ../environments/trilio_env.yaml
   DockerTrilioDatamoverImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1
   DockerTrilioDmApiImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1
   ContainerHorizonImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1

RHOSP16.2

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.2/scripts/
sudo ./prepare_trilio_images.sh <undercloud_ip/hostname> <HOTFIX-TAG-VERSION>-rhosp16.2

Verify the changes

$ grep 'Image' ../environments/trilio_env.yaml
   DockerTrilioDatamoverImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.2
   DockerTrilioDmApiImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.2
   ContainerHorizonImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.2

The changes can be verified using the following commands.

(undercloud) [stack@undercloud redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>]$ openstack tripleo container image list | grep trilio
| docker://undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1                       |                        |
| docker://undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1                   |                   |
| docker://undercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1                  |

-----------------------------------------------------------------------------------------------------

(undercloud) [stack@undercloud redhat-director-scripts]$ grep 'Image' ../environments/trilio_env.yaml
   DockerTrilioDatamoverImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1
   DockerTrilioDmApiImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1
   ContainerHorizonImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1

4.3] Red Hat Satellite Server

Please refer to the following overview to see which containers are available.

Follow this section when a Satellite Server is used for the container registry.

Pull the Trilio containers on the Red Hat Satellite using the given Red Hat registry URLs.

Populate the trilio_env.yaml with container urls.

RHOSP 13

$ grep 'Image' ../environments/trilio_env.yaml
   DockerTrilioDatamoverImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp13
   DockerTrilioDmApiImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp13
   DockerHorizonImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp13

RHOSP16.1

$ grep 'Image' trilio_env.yaml
   DockerTrilioDatamoverImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1
   DockerTrilioDmApiImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1
   ContainerHorizonImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1

RHOSP16.2

$ grep 'Image' trilio_env.yaml
   DockerTrilioDatamoverImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.2
   DockerTrilioDmApiImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.2
   ContainerHorizonImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.2

5. Verify Trilio environment details

It is recommended to re-populate the backup target details in the freshly downloaded trilio_env.yaml file. This will ensure that parameters that have been added since the last update/installation of Trilio are available and will be filled out too.

Locations of the trilio_env.yaml:

RHOSP13: /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp13/environments/trilio_env.yaml
RHOSP16.1: /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/environments/trilio_env.yaml
RHOSP16.2: /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.2/environments/trilio_env.yaml

For more details about the trilio_env.yaml please check here.

6] Update Overcloud Trilio components

Use the following heat environment file and roles data file in overcloud deploy command:

trilio_env.yaml
roles_data.yaml
Use correct Trilio endpoint map file as per available Keystone endpoint configuration
1. Instead of tls-endpoints-public-dns.yaml file, use environments/trilio_env_tls_endpoints_public_dns.yaml
2. Instead of tls-endpoints-public-ip.yaml file, useenvironments/trilio_env_tls_endpoints_public_ip.yaml
3. Instead of tls-everywhere-endpoints-dns.yaml file, useenvironments/trilio_env_tls_everywhere_dns.yaml

To include new environment files use '-e' option and for roles data file use '-r' option. An example overcloud deploy command is shown below:

openstack overcloud deploy --templates \
  -e /home/stack/templates/node-info.yaml \
  -e /home/stack/templates/overcloud_images.yaml \
  -e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_env.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-tls.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/inject-trust-anchor.yaml \
  -e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_env_tls_endpoints_public_dns.yaml \
  --ntp-server 192.168.1.34 \
  --libvirt-type qemu \
  --log-file overcloud_deploy.log \
  -r /usr/share/openstack-tripleo-heat-templates/roles_data.yaml

7] Verify deployment

If the containers are in restarting state or not listed by the following command then your deployment is not done correctly. Please recheck if you followed the complete documentation.

7.1] On Controller node

[root@overcloud-controller-0 heat-admin]# podman ps | grep trilio
26fcb9194566  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-REL-VERSION>-rhosp16.1        kolla_start           5 days ago  Up 5 days ago         trilio_dmapi
094971d0f5a9  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-REL-VERSION>-rhosp16.1       kolla_start           5 days ago  Up 5 days ago         horizon

7.2] On Compute node

Make sure Trilio datamover container is in running state and no other Trilio container is deployed on compute nodes.

[root@overcloud-novacompute-0 heat-admin]# podman ps | grep trilio
b1840444cc59  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-REL-VERSION>-rhosp16.1                    kolla_start           5 days ago  Up 5 days ago         trilio_datamover

7.3] On the node with Horizon service

[root@overcloud-controller-0 heat-admin]# podman ps | grep horizon
094971d0f5a9  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-REL-VERSION>-rhosp16.1       kolla_start           5 days ago  Up 5 days ago         horizon

Configuring Trilio

Added external database-support Added the Openstack distribution for storage (mount path)

Trilio configuration process is using Ansible scripts. Ansible, in the last few years, has grown in popularity as a preferred configuration management tool and Trilio uses ansible playbooks extensively to configure the Trilio cluster. To troubleshoot Trilio configuration issues, the user should have a basic understanding of Ansible playbook output.

Ansible modules are inherently idempotent and hence Trilio configuration can run any number of times to change or reconfigure Trilio cluster.

Once the VM is booted, point your browser (Chrome or Firefox) to Trilio node IP address.

This will bring you to the Trilio Dashboard, which contains the Trilio configurator.

The user is: admin The default password is: password

After the very first login, you are requested to change the admin password.

Unlike previous versions of Trilio, the current version only requires you to configure the cluster once and the Trilio dashboard provides cluster-wide management capability.

Uploading the OpenStack certificate bundle

OpenStack endpoints can be configured to use TLS. In such a configuration the Trilio appliance needs to trust the certificates provided by the OpenStack endpoints.

To achieve this trust it is required to upload the OpenStack certificate bundle through the OS API certificate tab of the Trilio appliance Dashboard.

The certificate bundle is located on the controller nodes of the OpenStack installation.

The default paths for each distribution are as follows:

RHOSP/TripleO: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
Kolla Ansible with CentOS: /etc/pki/tls/certs/ca-bundle.crt
Kolla Ansible with Ubuntu:  /usr/local/share/ca-certificates/
OpenStack Ansible (OSA) with Ubuntu in our lab: /etc/openstack_deploy/ssl/
OpenStack Asnible (OSA) with CentOS: /etc/openstack_deploy/ssl

The uploaded certificates can be verified on the Trilio appliance at the following location.

/etc/workloadmgr/ca-chain.pem

Details needed for the Trilio Appliance

Upon login into an unconfigured Trilio Appliance, the shown page is the configurator. The configurator requires some information about the Trilio Appliance, Openstack, and Backup Storage.

Trilio Cluster information

The Trilio Cluster needs to be integrated into an existing environment to be able to operate correctly. This block asks for information about the Trilio Cluster operating details.

Controller Nodes
- This is the list of Trilio virtual appliance IP addresses along with their hostnames.
- Format: comma-separated list with pairs combined through '='
- Example: 172.20.4.151=tvault-104-1,172.20.4.152=tvault-104-2,172.20.4.153=tvault-104-3’

The Trilio Cluster supports only 1 node and 3 node clusters.

Virtual IP Address
- This is the Trilio cluster IP address which is mandatory
- Format: IP/Subnet
- Example: 172.20.4.150/24

The Virtual IP is mandatory even for single-node clusters and has to be different from any IP given at the Controller Nodes.

Name Server
- List of nameservers, primarily used to resolve OpenStack service endpoints.
- Format: comma-separated list
- example: 10.10.10.1,172.20.4.1

If defining OpenStack endpoint hostnames in the /etc/hosts file on the VM is preferred over a DNS solution you may set the nameserver to 0.0.0.0, the default gateway.

Domain Search Order
- The domain the Trilio Cluster will use.
- Format: comma-separated list
- example: trilio.io,trilio.demo
NTP Servers
- NTP servers the Trilio Cluster will use
- format: comma-separated list
- example: 0.pool.ntp.org,10.10.10.10
Timezone
- Timezone the Trilio Cluster will use internally
- format: pre-populated list
- example: UTC

Openstack Credentials information

The Trilio appliance integrates with one RHV environment. This block asks for the information required to access and connect with the RHV Cluster.

Keystone URL
- The Keystone endpoint used to fetch authentication for configuration
- format: URL
- example: https://keystone.trilio.io:5000/v3
Endpoint Type
- Defines which endpoint type will be used to communicate with the Openstack endpoints
- format: predefined list of radio buttons
- example: Public

When FQDNs are used for the Keystone endpoints it is necessary to configure at least one DNS server before the configuration.

Otherwise, the validation of the Openstack Credentials will fail.

Domain ID
- domain the provided user and tenant are located in
- format: ID
- example: default
Administrator
- Username of an account with the domain admin role
- format: String
- example: admin
Password
- password for the prior provided user
- format: String
- example: password

Trilio requires domain admin role access. To provide domain admin role to a user, the following command can be used:

openstack role add --domain <domain id> --user <username> admin

The Trilio configurator verifies after every entry if it is possible to login into Openstack using the provided credentials.

This verification will fail until all entries are set and correct.

When the verification is successful it is possible to choose the Admin tenant, the Region, and the Trustee role without any error message shown.

Admin Tenant
- The tenant to be used together with the provided user
- format: a pre-populated list
- example: admin
Region
- Openstack Region the user and tenant are located in
- format: a pre-populated list
- example: RegionOne
Trustee Role
- The Openstack role required to be able to use Trilio functionalities
- format: a pre-populated list
- example: _member_

Backup Storage Configuration information

This block is requesting the necessary information about the backup target that the Trilio installation will be used to store and read backups.

Openstack Dist
- RHOSP and Kolla Ansible require a special mount point to be used
- format: predefined list
- example: RHOSP
Backup Storage
- Defines the Backup Storage protocol to use
- format: predefined list of radio buttons
- example: NFS

Using the NFS protocol

NFS Export
- The path under which the NFS Volumes to be used can be found
- format: comma-separated list of NFS Volumes paths
- example: 10.10.2.20:/upstream,10.10.5.100:/nfs2
NFS Options
- NFS options used by the Trilio Cluster when mounting the NFS Exports
- format: NFS options
- example: nolock,soft,timeo=180,intr,lookupcache=none

Please use the predefined NFS Options and only change them when it is know that changes are necessary.

Trilio is testing against the predefined NFS options.

Using the S3 protocol

S3 Compatible
- Switch between Amazon and other S3 compatible storage solutions
- format: predefined list
- example: Amazon S3
(S3 compatible) Endpoint URL
- URL to be used to reach and access the provided S3 compatible storage
- format: URL
- example: objects.trilio.io
Access Key
- Access Key necessary to login into the S3 storage
- format: access key
- example: SFHSAFHPFFSVVBSVBSZRF
Secret Key
- Secret Key necessary to login into the S3 storage
- format: secret key
- example: bfAEURFGHsnvd3435BdfeF
Region
- Configured Region for the S3 Bucket (keep the default for S3 compatible without Region)
- format: String
- example: us-east-1
Signature Version
- S3 signature version to use for signing into the S3 storage
- format: string
- example: default
Bucket Name
- Name of the bucket to be used as Backup target
- format: string
- example: Trilio-backup

Using secured non-aws S3 storage

When using secured connection with a non-aws S3 storage like CEPH you have to provide the certificate used for the connection.

To enter this certificate type the https:// based endpoint into the field Endpoint URL.

Once you tab out of the field will the upload certificate button be shown. See picture below.

Workload Import

Check this box in case of reinitialization or reinstallation of the Trilio Appliance to import all matching Workloads located on the Backup Target.

Workloads that are not assigned to an existing tenant will fail to import and need to be reassigned manually once the configuration is done.

Advanced settings

At the end of the configurator is the option to activate the advanced settings.

Activating this option does provide the possibility to configure the Keystone endpoints used for the Datamover API and Trilio.

Setup Trilio and Datamover API endpoints.

Trilio generates Keystone endpoints for 2 services. The Trilio Datamover API and the Trilio Workloadmanager.

Modern Openstack installation have the endpoint types split over multiple networks. The advanced settings for the Datamover API endpoints and Trilio Workloadmanager endpoints allow configuring Trilio accordingly.

Used IP addresses are added as additional VIPs to the Trilio cluster.

In the case of FQDN used for those endpoints will the Trilio configurator resolve the FQDN to learn of the IPs that are then set as VIPs.

It is recommended to verify the datamover api settings against the ones configured during installation of the Trilio components.

If these endpoints do already exist in Keystone are the values prefilled and can not be changed. In case of a change required, delete the old Keystone endpoints first.

Providing an URL with https activates the TLS enabled configuration, which requires the upload of certificates and the connected private key.

Set up an external database

Trilio allows the use of an external MySQL or MariaDB database.

This database needs to be prepared by creating the empty workloadmgr database, creating the workloadmgr user and setting the right permissions. An example command to create this database would be:

create database workloadmgr_auto;
CREATE USER 'trilio'@'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON workloadmgr_auto.* TO 'trilio'@'10.10.10.67' IDENTIFIED BY 'password';

Provide the connection string to the Trilio configurator.

mysql://trilio:password@10.10.10.67/workloadmgr_auto?charset=utf8

This value can only be set upon an initial configuration of the Trilio solution.

When the Cluster has been configured to use the internal database, then the connection string will not be shown in the next configuration attempt.

In case of an external database, will the connection string be shown, but is uneditable.

Define the Trilio service user password

Trilio is using a service user that is located in the Openstack service project.

The password for this service user will be generated randomly or can be defined in the advanced settings.

Starting the configurator

Once all entries have been set and all validations are error-free the configurator can be started.

Click Finish
Reconfirm in the pop-up that you want to start the configuration
Wait for the configurator to finish

Some elements of the configurator take time. Even when it looks like the configurator is stuck, please wait till the configurator finishes. Should the configurator have not finished after 6h, please contact Trilio Support for help.

The configurator is using Ansible and a few Trilio internal API calls. After each configuration block or after the configurator finished it is possible to visit the Ansible output.

At the end of a successful configuration does the configurator forward to the set VIP.

Restart Trilio Services

In complex environments it is sometimes necessary to restart a single service or the complete solution. Rarely is restarting the complete node, where a service is running possible or even the ideal solution.

This page describes the services running by Trilio and how to restart those.

Trilio Appliance Services

The Trilio Appliance is the controller of Trilio. Most services on the Appliance are running in a High Availability mode on a 3-node cluster.

wlm-api

The wlm-api service takes the API calls against the Trilio Appliance. It is running in active-active mode on all nodes of the Trilio cluster.

To restart the wlm-api service run on each Trilio node:

systemctl restart wlm-api

wlm-scheduler

The wlm-scheduler service is taking job requests and identifies which Trilio node should take the request. It is running in active-active mode on all nodes of the Trilio cluster.

To restart the wlm-scheduler service run on each Trilio node:

systemctl restart wlm-scheduler

wlm-workloads

The wlm-workloads service is the task worker of Trilio executing all jobs given to the Trilio node. It is running in active-active mode on all nodes of the Trilio cluster.

To restart the wlm-workloads service run on each Trilio node:

systemctl restart wlm-workloads

wlm-cron

The wlm-cron service is responsible for starting scheduled Backups according to the configurtation of Tenant Workloads. It is running in active-passive mode and controlled by the pacemaker cluster.

To restart the wlm-workloads service run on the Trilio node with VIP assigned:

pcs resource restart wlm-cron

VIP resources

The Trilio appliance is running 1 to 4 virtual IPs on the Trilio cluster. These are controlled by the pacemaker cluster and provided through NGINX.

To restart these resources the pacemaker NGINX resource is getting restarted:

pcs resource restart lb_nginx-clone

RabbitMQ

The Trilio cluster is using RabbitMQ as messaging service. It is running in active-active mode on all nodes of the Trilio cluster.

RabbitMQ is a complex system in itself. This guide will only provide the basic commands to do a restart of a node and check the health of the cluster afterward. For complete documentation of how to restart RabbitMQ, please follow the official RabbitMQ documentation.

To restart a RabbitMQ node run on each Trilio node:

It is recommended to wait for the node to rejoin and sync with the cluster before restarting another RabbitMQ node.

[root@TVM1 ~]# rabbitmqctl stop
Stopping and halting node rabbit@TVM1 ...
[root@TVM1 ~]# rabbitmq-server -detached
Warning: PID file not written; -detached was passed.
[root@TVM1 ~]# rabbitmqctl cluster_status
Cluster status of node rabbit@TVM1 ...
[{nodes,[{disc,[rabbit@TVM1,rabbit@TVM2,rabbit@TVM3]}]},
 {running_nodes,[rabbit@TVM2,rabbit@TVM3,rabbit@TVM1]},
 {cluster_name,<<"rabbit@TVM1">>},
 {partitions,[{rabbit@TVM2,[rabbit@TVM1,rabbit@TVM3]},
              {rabbit@TVM3,[rabbit@TVM1,rabbit@TVM2]}]},
 {alarms,[{rabbit@TVM2,[]},{rabbit@TVM3,[]},{rabbit@TVM1,[]}]}]

When the complete cluster is getting stopped and restarted it is important to keep the order of nodes in mind. The last node to be stopped needs to be the first node to be started.

Galera Cluster (MariaDB)

The Galera Cluster is managing the Trilio MariaDB database. It is running in active-active mode on all nodes of the Trilio cluster.

Galera Cluster is a complex system in itself. This guide will only provide the basic commands to do a restart of a node and check the health of the cluster afterward. For complete documentation of how to restart Galera clusters, please follow the official Galera documentation.

When restarting Galera two different use-cases need to be considered:

Restarting a single node
Restarting the whole cluster

Restarting a single node

A single node can be restarted without any issues. It will automatically rejoin the cluster and sync against the remaining nodes.

The following commands will gracefully stop and restart the mysqld service.

After a restart will the cluster start the syncing process. Don't restart node after node to reach a complete cluster restart.

systemctl stop mysqld
systemctl start mysqld

Check the cluster health after the restart.

Restarting the complete cluster

Restarting a complete cluster requires some additional steps as the Galera cluster is basically destroyed once all nodes have been shut down. It needs to be rebuild afterwards.

First gracefully shutdown the Galera cluster on all nodes:

systemctl stop mysqld

The second step is to identify the Galera node with the latest dataset. This can be achieved by reading the grastate.dat file on the Trilio nodes.

When this documentation is followed the last mysqld service that got shut down will be the one with the latest dataset.

cat /var/lib/mysql/grastate.dat

# GALERA saved state
version: 2.1
uuid:    353e129f-11f2-11eb-b3f7-76f39b7b455d
seqno:   213576545367
safe_to_bootstrap: 1

The value to check for are the seqno.

The node with the highest seqno is the node that contains the latest data. This node will also contain safe_to_bootstrap: 1 to indicate that the Galera cluster can be rebuild from this node.

On the identified node the new cluster is getting generated with the following command:

galera_new_cluster

Running galera_new_cluster on the wrong node will lead to data loss as this command will set the node the command is issued on as the first node of the cluster. All nodes which join afterward will sync against the data of this first node.

After the command has been issued is the mysqld service running on this node. Now the other nodes can be restarted one by one. The started nodes will automatically rejoin the cluster and sync against the master node. Once a synced status has been reached is each node a primary node in the cluster.

systemctl start mysqld

Check the Cluster health after all services are up again.

Verify Health of the Galera Cluster

Verify the cluster health by running the following commands inside each Trilio MariaDB. The values returned from these statements have to be the same for each node.

MariaDB [(none)]> show status like 'wsrep_incoming_addresses';
+--------------------------+-------------------------------------------------+
| Variable_name            | Value                                           |
+--------------------------+-------------------------------------------------+
| wsrep_incoming_addresses | 10.10.2.13:3306,10.10.2.14:3306,10.10.2.12:3306 |
+--------------------------+-------------------------------------------------+
1 row in set (0.01 sec)

MariaDB [(none)]> show status like 'wsrep_cluster_size';
+--------------------+-------+
| Variable_name      | Value |
+--------------------+-------+
| wsrep_cluster_size | 3     |
+--------------------+-------+
1 row in set (0.00 sec)

MariaDB [(none)]> show status like 'wsrep_cluster_state_uuid';
+--------------------------+--------------------------------------+
| Variable_name            | Value                                |
+--------------------------+--------------------------------------+
| wsrep_cluster_state_uuid | 353e129f-11f2-11eb-b3f7-76f39b7b455d |
+--------------------------+--------------------------------------+
1 row in set (0.00 sec)

MariaDB [(none)]> show status like 'wsrep_local_state_comment';
+---------------------------+--------+
| Variable_name             | Value  |
+---------------------------+--------+
| wsrep_local_state_comment | Synced |
+---------------------------+--------+
1 row in set (0.01 sec)

Canonical workloadmgr container services

Canonical Openstack is not using the Trilio Appliance. In Canonical environments is the Trilio controller unit part of the JuJu deployment as workloadmgr container.

To restart the services inside this container the following commands are to be issued.

Single Node deployment

juju ssh <workloadmgr unit name>/<unit-number>
Systemctl restart wlm-api wlm-scheduler wlm-workloads wlm-cron

HA deployment

On all nodes:

juju ssh <workloadmgr unit name>/<unit-number>
Systemctl restart wlm-api wlm-scheduler wlm-workloads

On a single node:

juju ssh <workloadmgr unit name>/<unit-number>
crm_resource --restart -r res_trilio_wlm_wlm_cron

Trilio dmapi service

The Trilio dmapi service is running on the Openstack controller nodes. Depending on the Openstack Distribution Trilio is installed on different commands are issued to restart the dmapi service.

RHOSP13

RHOSP13 is running the Trilio services as docker containers. The dmapi service can be restarted by issuing the following command on the host running the dmapi service.

docker restart trilio_dmapi

RHOSP16

RHOSP16 is running the Trilio services as docker containers. The dmapi service can be restarted by issuing the following command on the host running the dmapi service.

podman restart trilio_dmapi

Canonical

Canonical is running the Trilio services in JuJu controlled LXD containers. The dmapi service can be restarted by issuing the following command from the MASS node.

juju ssh <trilio-dm-api unit name>/<unit-number>
sudo systemctl restart tvault-datamover-api

Kolla-Ansible Openstack

Kolla-Ansible Openstack is running the Trilio services as docker containers. The dmapi service can be restarted by issuing the following command on the host running the dmapi service.

docker restart triliovault_datamover_api

Ansible Openstack

Ansible Openstack is running the Trilio services as LXD containers. The dmapi service can be restarted by issuing the following command on the host running the dmapi service.

lxc-stop -n <dmapi container name>
lxc-start -n <dmapi container name>

Trilio datamover service (tvault-contego)

The Trilio datamover service is running on the Openstack compute nodes. Depending on the Openstack Distribution Trilio is installed on different commands are issued to restart the datamover service.

RHOSP13

RHOSP13 is running the Trilio services as docker containers. The datamover service can be restarted by issuing the following command on the compute node.

docker restart trilio_datamover

RHOSP16

RHOSP16 is running the Trilio services as docker containers. The datamover service can be restarted by issuing the following command on the compute node.

podman restart trilio_datamover

Canonical

Canonical is running the Trilio services in JuJu controlled LXD containers. The datamover service can be restarted by issuing the following command from the MASS node.

juju ssh <trilio-data-mover unit name>/<unit-number>
sudo systemctl restart tvault-contego

Kolla-Ansible Openstack

Kolla-Ansible Openstack is running the Trilio services as docker containers. The dmapi service can be restarted by issuing the following command on the host running the dmapi service.

docker restart triliovault_datamover

Ansible Openstack

Ansible Openstack is running the Trilio datamover service directly on the compute node. The datamover service can be restarted by issuing the following command on.

service tvault-contego restart

Workloads

Definition

A workload is a backup job that protects one or more Virtual Machines according to a configured policy. There can be as many workloads as needed. But each VM can only be part of one Workload.

List of Workloads

Using Horizon

To view all available workloads of a project inside Horizon do:

Login to Horizon
Navigate to Backups
Navigate to Workloads

The overview in Horizon lists all workloads with the following additional information:

Creation time
Workload Name
Workload description
Total amount of Snapshots inside this workload
- Total amount of succeeded Snapshots
- Total amount of failed Snapshots
Workload Type
Status of the Workload

Using CLI

workloadmgr workload-list [--all {True,False}] [--nfsshare <nfsshare>]

Workload Create

Using Horizon

To create a workload inside Horizon do the following steps:

Login to Horizon
Navigate to the Backups
Navigate to Workloads
Click "Create Workload"
Provide Workload Name and Workload Description on the first tab "Details"
Choose between Serial or Parallel workload on the first tab "Details"
Choose the Policy if available to use on the first tab "Details"
Choose the VMs to protect on the second Tab "Workload Members"
Decide for the schedule of the workload on the Tab "Schedule"
Provide the Retention policy on the Tab "Policy"
Choose the Full Backup Interval on the Tab "Policy"
If required check "Pause VM" on the Tab "Options"
Click create

The created Workload will be available after a few seconds and starts to take backups according to the provided schedule and policy.

Using CLI

workloadmgr workload-create --instance <instance-id=instance-uuid>
                            [--display-name <display-name>]
                            [--display-description <display-description>]
                            [--workload-type-id <workload-type-id>]
                            [--source-platform <source-platform>]
                            [--jobschedule <key=key-name>]
                            [--metadata <key=key-name>]
                            [--policy-id <policy_id>]

Workload Overview

A workload contains many information, which can be seen in the workload overview.

Using Horizon

To enter the workload overview inside Horizon do the following steps:

Login to Horizon
Navigate to the Backups
Navigate to Workloads
Identify the workload to show the details on
Click the workload name to enter the Workload overview

Details Tab

The Workload Details tab provides you with the general most important information about the workload:

Name
Description
Availability Zone
List of protected VMs including the information of qemu guest agent availability

The status of the qemu-guest-agent just shows, whether the necessary Openstack configuration has been done for this VM to provide qemu guest agent integration. It does not check, whether the qemu guest agent is installed and configured on the VM.

It is possible to navigate to the protected VM directly from the list of protected VMs.

Snapshots Tab

The Workload Snapshots Tab shows the list of all available Snapshots in the chosen Workload.

From here it is possible to work with the Snapshots, create Snapshots on demand and start Restores.

Please refer to the Snapshot and Restore User Guide to learn more about those.

Policy Tab

The Workload Policy Tab gives an overview of the current configured scheduler and retention policy. The following elements are shown:

Scheduler Enabled / Disabled
Start Date / Time
End Date / Time
RPO
Time till next Snapshot run
Retention Policy and Value
Full Backup Interval policy and value

Filesearch Tab

The Workload Filesearch Tab provides access to the powerful search engine, which allows to find files and folders on Snapshots without the need of a restore.

Please refer to the File Search User Guide to learn more about this feature.

Misc. Tab

The Workload Miscellaneous Tab shows the remaining metadata of the Workload. The following information are provided:

Creation time
last update time
Workload ID
Workload Type

Using CLI

workloadmgr workload-show <workload_id> [--verbose <verbose>]

Edit a Workload

Workloads can be modified in all components to match changing needs.

Editing a Workload will set the User, who edits the Workload, as the new owner.

Using Horizon

To edit a workload in Horizon do the following steps:

Login to the Horizon
Navigate to Backups
Navigate to Workloads
Identify the workload to be modified
Click the small arrow next to "Create Snapshot" to open the sub-menu
Click "Edit Workload"
Modify the workload as desired - All parameters except workload type can be changed
Click "Update"

Using CLI

usage: workloadmgr workload-modify [--display-name <display-name>]
                                   [--display-description <display-description>]
                                   [--instance <instance-id=instance-uuid>]
                                   [--jobschedule <key=key-name>]
                                   [--metadata <key=key-name>]
                                   [--policy-id <policy_id>]
                                   <workload_id>

Delete a Workload

Once a workload is no longer needed it can be safely deleted.

All Snapshots need to be deleted before the workload gets deleted. Please refer to the Snapshots User Guide to learn how to delete Snapshots.

Using Horizon

To delete a workload do the following steps:

Login to Horizon
Navigate to the Backups
Navigate to Workloads
Identify the workload to be deleted
Click the small arrow next to "Create Snapshot" to open the sub-menu
Click "Delete Workload"
Confirm by clicking "Delete Workload" yet again

Using CLI

workloadmgr workload-delete [--database_only <True/False>] <workload_id>

Unlock a Workload

Workloads that are actively taking backups or restores are locked for further tasks. It is possible to unlock a workload by force if necessary.

It is highly recommend to use this feature only as last resort in case of backups/restores being stuck without failing or a restore is required while a backup is running.

Using Horizon

Login to the Horizon
Navigate to Backups
Navigate to Workloads
Identify the workload to unlock
Click the small arrow next to "Create Snapshot" to open the sub-menu
Click "Unlock Workload"
Confirm by clicking "Unlock Workload" yet again

Using CLI

workloadmgr workload-unlock <workload_id>

Reset a Workload

In rare cases it might be necessary to start a backup chain all over again, to ensure the quality of the created backups. To not recreate a Workload in such cases is it possible to reset a Workload.

The Workload reset will:

Cancel all ongoing tasks
Delete all existing Openstack Trilio Snapshots from the protected VMs
recalculate the next Snapshot time
take a full backup at the next Snapshot

Using Horizon

To reset a Workload do the following steps:

Login to the Horizon
Navigate to Backups
Navigate to Workloads
Identify the workload to reset
Click the small arrow next to "Create Snapshot" to open the sub-menu
Click "Reset Workload"
Confirm by clicking "Reset Workload" yet again

Using CLI

workloadmgr workload-reset <workload_id>

Restores

Definition

A Restore is the workflow to bring back the backed up VMs from a Trilio Snapshot.

List of Restores

Using Horizon

To reach the list of Restores for a Snapshot follow these steps:

Login to Horizon
Navigate to Backups
Navigate to Workloads
Identify the workload that contains the Snapshot to show
Click the workload name to enter the Workload overview
Navigate to the Snapshots tab
Identify the searched Snapshot in the Snapshot list
Click the Snapshot Name
Navigate to the Restores tab

Using CLI

Restores overview

Using Horizon

To reach the detailed Restore overview follow these steps:

Login to Horizon
Navigate to Backups
Navigate to Workloads
Identify the workload that contains the Snapshot to show
Click the workload name to enter the Workload overview
Navigate to the Snapshots tab
Identify the searched Snapshot in the Snapshot list
Click the Snapshot Name
Navigate to the Restores tab
Identify the restore to show
Click the restore name

Details Tab

The Restore Details Tab shows the most important information about the Restore.

Name
Description
Restore Type
Status
Time taken
Size
Progress Message
Progress
Host
Restore Options

The Restore Options are the restore.json provided to Trilio.

List of VMs restored
- restored VM Name
- restored VM Status
- restored VM ID

Misc Tab

The Misc tab provides additional Metadata information.

Creation Time
Restore ID
Snapshot ID containing the Restore
Workload

Using CLI

Delete a Restore

Once a Restore is no longer needed, it can be safely deleted from a Workload.

Deleting a Restore will only delete the Trilio information about this Restore. No Openstack resources are getting deleted.

Using Horizon

There are 2 possibilities to delete a Restore.

Possibility 1: Single Restore deletion through the submenu

To delete a single Restore through the submenu follow these steps:

Login to Horizon
Navigate to Backups
Navigate to Workloads
Identify the workload that contains the Snapshot to delete
Click the workload name to enter the Workload overview
Navigate to the Snapshots tab
Identify the searched Snapshot in the Snapshot list
Click the Snapshot Name
Navigate to the Restore tab
Click "Delete Restore" in the line of the restore in question
Confirm by clicking "Delete Restore"

Possibility 2: Multiple Restore deletion through a checkbox in Snapshot overview

To delete one or more Restores through the Restore list do the following:

Login to Horizon
Navigate to Backups
Navigate to Workloads
Identify the workload that contains the Snapshot to show
Click the workload name to enter the Workload overview
Navigate to the Snapshots tab
Identify the searched Snapshots in the Snapshot list
Enter the Snapshot by clicking the Snapshot name
Navigate to the Restore tab
Check the checkbox for each Restore that shall be deleted
Click "Delete Restore" in the menu above
Confirm by clicking "Delete Restore"

Using CLI

Cancel a Restore

Ongoing Restores can be canceled.

Using Horizon

To cancel a Restore in Horizon follow these steps:

Login to Horizon
Navigate to Backups
Navigate to Workloads
Identify the workload that contains the Snapshot to delete
Click the workload name to enter the Workload overview
Navigate to the Snapshots tab
Identify the searched Snapshot in the Snapshot list
Click the Snapshot Name
Navigate to the Restore tab
Identify the ongoing Restore
Click "Cancel Restore" in the line of the restore in question
Confirm by clicking "Cancel Restore"

Using CLI

One Click Restore

The One Click Restore will bring back all VMs from the Snapshot in the same state as they were backed up. They will:

be located in the same cluster in the same datacenter
use the same storage domain
connect to the same network
have the same flavor

The user can't change any Metadata.

The One Click Restore requires, that the original VM's that have been backed up are deleted or otherwise lost. If even one VM is still existing, will the One Click Restore fail.

The One Click Restore will automatically update the Workload to protect the restored VMs.

Using Horizon

There are 2 possibilities to start a One Click Restore.

Possibility 1: From the Snapshot list

Login to Horizon
Navigate to Backups
Navigate to Workloads
Identify the workload that contains the Snapshot to be restored
Click the workload name to enter the Workload overview
Navigate to the Snapshots tab
Identify the Snapshot to be restored
Click "One Click Restore" in the same line as the identified Snapshot
(Optional) Provide a name / description
Click "Create"

Possibility 2: From the Snapshot overview

Login to Horizon
Navigate to Backups
Navigate to Workloads
Identify the workload that contains the Snapshot to be restored
Click the workload name to enter the Workload overview
Navigate to the Snapshots tab
Identify the Snapshot to be restored
Click the Snapshot Name
Navigate to the "Restores" tab
Click "One Click Restore"
(Optional) Provide a name / description
Click "Create"

Using CLI

Selective Restore

The Selective Restore is the most complex restore Trilio has to offer. It allows to adapt the restored VMs to the exact needs of the User.

With the selective restore the following things can be changed:

Which VMs are getting restored
Name of the restored VMs
Which networks to connect with
Which Storage domain to use
Which DataCenter / Cluster to restore into
Which flavor the restored VMs will use

The Selective Restore is always available and doesn't have any prerequirements.

The Selective Restore will automatically update the Workload to protect the created instance in the case that the original instance is no longer existing.

Using Horizon

There are 2 possibilities to start a Selective Restore.

Possibility 1: From the Snapshot list

Login to Horizon
Navigate to Backups
Navigate to Workloads
Identify the workload that contains the Snapshot to be restored
Click the workload name to enter the Workload overview
Navigate to the Snapshots tab
Identify the Snapshot to be restored
Click on the small arrow next to "One Click Restore" in the same line as the identified Snapshot
Click on "Selective Restore"
Configure the Selective Restore as desired
Click "Restore"

Possibility 2: From the Snapshot overview

Login to Horizon
Navigate to Backups
Navigate to Workloads
Identify the workload that contains the Snapshot to be restored
Click the workload name to enter the Workload overview
Navigate to the Snapshots tab
Identify the Snapshot to be restored
Click the Snapshot Name
Navigate to the "Restores" tab
Click "Selective Restore"
Configure the Selective Restore as desired
Click "Restore"

Using CLI

Inplace Restore

The Inplace Restore covers those use cases, where the VM and its Volumes are still available, but the data got corrupted or needs to a rollback for other reasons.

It allows the user to restore only the data of a selected Volume, which is part of a backup.

The Inplace Restore only works when the original VM and the original Volume are still available and connected. Trilio is checking this by the saved Object-ID.

The Inplace Restore will not create any new RHV resources. Please use one of the other restore options if new Volumes or VMs are required.

Using Horizon

There are 2 possibilities to start an Inplace Restore.

Possibility 1: From the Snapshot list

Login to Horizon
Navigate to Backups
Navigate to Workloads
Identify the workload that contains the Snapshot to be restored
Click the workload name to enter the Workload overview
Navigate to the Snapshots tab
Identify the Snapshot to be restored
Click on the small arrow next to "One Click Restore" in the same line as the identified Snapshot
Click on "Inplace Restore"
Configure the Inplace Restore as desired
Click "Restore"

Possibility 2: From the Snapshot overview

Login to Horizon
Navigate to Backups
Navigate to Workloads
Identify the workload that contains the Snapshot to be restored
Click the workload name to enter the Workload overview
Navigate to the Snapshots tab
Identify the Snapshot to be restored
Click the Snapshot Name
Navigate to the "Restores" tab
Click "Inplace Restore"
Configure the Inplace Restore as desired
Click "Restore"

Using CLI

required restore.json for CLI

The workloadmgr client CLI is using a restore.json file to define the restore parameters for the selective and the inplace restore.

An example for a selective restore of this restore.json is shown below. A detailed analysis and explanation is given afterwards.

General required information

Before the exact details of the restore are to be provided it is necessary to provide the general metadata for the restore.

openstackstarts the exact definition of the restore

Selective Restore required information

The Selective Restore requires a lot of information to be able to execute the restore as desired.

Those information are divided into 3 components:

instances
restore_topology
networks_mapping

Information required in instances

This part contains all information about all instances that are part of the Snapshot to restore and how they are to be restored.

Even when VMs are not to be restored are they required inside the restore.json to allow a clean execution of the restore.

Each instance requires the following information

All further information are only required, when the instance is part of the restore.

To use the next free IP available in the set Nics to an empty list [ ]

Using an empty list for Nics combined with the Network Topology Restore, will the restore automatically restore the original IP address of the instance.

The root disk needs to be at least as big as the root disk of the backed up instance was.

The following example describes a single instance with all values.

Information required in network topology restore or network mapping

Do not mix network topology restore together with network mapping.

To activate a network topology restore set:

To activate network mapping set:

When the network mapping is activated it is used, it is necessary to provide the mapping details, which are part of the networks_mapping block:

Full selective restore example

Inplace Restore required information

The Inplace Restore requires less information thana selective restore. It only requires the base file with some information about the Instances and Volumes to be restored.

Information required in instances

When the boot disk is at the same time a Cinder Disk, both values need to be set true.

Network mapping information required

There are no network information required, but the field have to exist as empty value for the restore to work.

Full Inplace restore example

Backups-Admin Area

Trilio provides Backup-as-a-Service, which allows Openstack Users to manage and control their backups themselves. This doesn't eradicate the need for a Backup Administrator, who has an overview of the complete Backup Solution.

To provide Backup Administrators with the tools they need does Trilio for Openstack provide a Backup-Admin area in Horizon in addition to the API and CLI.

Access the Backups-Admin area

To access the Backups-Admin area follow these steps:

Login to Horizon using admin user.
Click on Admin Tab.
Navigate to Backups-Admin Tab.
Navigate to Trilio page.

The Backups-Admin area provides the following features.

It is possible to reduce the shown information down to a single tenant. That way seeing the exact impact the chosen Tenant has.

Status overview

The status overview is always visible in the Backups-Admin area. It provides the most needed information on a glance, including:

Storage Usage (nfs only)
Number of protected VMs compared to number of existing VMs
Number of currently running Snapshots
Status of TVault Nodes
Status of Contego Nodes

The status of nodes is filled when the services are running and in good status.

Workloads tab

This tab provides information about all currently existing Workloads. It is the most important overview tab for every Backup Administrator and therefor the default tab shown when opening the Backup-Admins area.

The following information are shown:

User-ID that owns the Workload
Project that contains the Workload
Workload name
Workload Type
Availability Zone
Amount of protected VMs
Performance information about the last 30 backups
- How much data was backed up (green bars)
- How long did the Backup take (red line)
Piechart showing amount of Full (Blue) Backups compared to Incremental (Red) Backups
Number of successful Backups
Number of failed Backups
Storage used by that Workload
Which Backup target is used
When is the next Snapshot run
What is the general intervall of the Workload
Scheduler Status including a Switch to deactivate/activate the Workload

Usage tab

Administrators often need to figure out, where a lot of resources are used up, or they need to quickly provide usage information to a billing system. This tab helps in these tasks by providing the following information:

Storage used by a Tenant
VMs protected by a Tenant

It is possible to drill down to see the same information per workload and finally per protected VM.

The Usage tab includes workloads and VMs that are no longer actively used by a Tenant, but exist on the backup target.

Nodes tab

This tab displays information about Trilio cluster nodes. The following information are shown:

Node name
Node ID
Trilio Version of the node
IP Address
Active Controller Node (True/False)
Status of the Node

The Virtual IP is shown as it's own node. It is typically shown directly below the current active Controller Node.

Data Movers tab (Trilio Data Mover Service)

This tab displays information about Trilio contego service. The following information are shown:

Service-Name
Compute Node the service is running on
Zone
Service Status from Openstack perspective (enabled/disabled)
Version of the Service
General Status
last time the Status was updated

Storage tab

This tab displays information about the backup target storage. It contains the following information:

Storage Name

Clicking on the Storage name provides an overview of all workloads stored on that storage.

Capacity of the storage
Total utilization of the storage
Status of the storage
Statistic information
- Percentage all storages are used
- Percentage how much storage is used for full backups
- Amount of Full backups versus Incremental backups

Audit tab

Audit logs provide the sequence of workload related activities done by users, like workload creation, snapshot creation, etc. The following information are shown:

Time of the entry
What task has been done
Project the task has performed in
User that performed the task

The Audit log can be searched for strings to find for example only entries down by a specific user.

Additionally, can the shown timeframe be changed as necessary.

License tab

The license tab provides an overview over the current license and allows to upload new licenses, or validate the current license.

A license validation is automatically done, when opening the tab.

The following information about an active license are shown:

Organization (License name)
License ID
Purchase date - when was the license created
License Expiry Date
Maintenance Expiry Date
License value
License Edition
License Version
License Type
Description of the License
Evaluation (True/False)

Trilio will stop all activities once a license is no longer valid or expired.

Policy tab

The policy tab gives Administrators the possibility to work with workload policies.

Please use Workload Policies in the Admin guide to learn more about how to create and use Workload Policies.

Settings tab

This tab manages all global settings for the whole cloud. Trilio has two types of settings:

Email settings
Job scheduler settings.

Email Settings

These settings will be used by Trilio to send email reports of snapshots and restores to users.

Configuring the Email settings is a must-have to provide Email notification to Openstack users.

The following information are required to configure the email settings:

SMTP Server
SMTP username
SMTP password
SMTP port
SMTP timeout
Sender email address

A test email can be sent directly from the configuration page.

To work with email settings through CLI use the following commands:

To set an email setting for the first time or after deletion use:

workloadmgr setting-create [--description <description>]
                           [--category <category>]
                           [--type <type>]
                           [--is-public {True,False}]
                           [--is-hidden {True,False}]
                           [--metadata <key=value>]
                           <name> <value>

To update an already set email setting through CLI use:

workloadmgr setting-update [--description <description>]
                           [--category <category>]
                           [--type <type>]
                           [--is-public {True,False}]
                           [--is-hidden {True,False}]
                           [--metadata <key=value>]
                           <name> <value>

To show an already set email setting use:

workloadmgr setting-show [--get_hidden {True,False}] <setting_name>

To delete a set email setting use:

workloadmgr setting-delete <setting_name>

Disable/Enable Job Scheduler

The Global Job Scheduler can be used to deactivate all scheduled workloads without modifying each one of them.

To activate/deactivate the Global Job Scheduler through the Backups-Admin area:

Login to Horizon using admin user.
Click on Admin Tab.
Navigate to Backups-Admin Tab.
Navigate to Trilio page.
Navigate to the Settings tab
Click "Disable/Enable Job Scheduler"
Check or Uncheck the box for "Job Scheduler Enabled"
Confirm by clicking on "Change"

The Global Job Scheduler can be controlled through CLI as well.

To get the status of the Global Job Scheduler use:

workloadmgr get-global-job-scheduler

To deactivate the Global Job Scheduler use:

workloadmgr disable-global-job-scheduler

To activate the Global Job Scheduler use:

workloadmgr enable-global-job-scheduler

Example runbook for Disaster Recovery using NFS

This runbook will demonstrate how to set up Disaster Recovery with Trilio for a given scenario.

The chosen scenario is following an actively used Trilio customer environment.

Scenario

There are two Openstack clouds available "Openstack Cloud A" and Openstack Cloud B". "Openstack Cloud B" is the Disaster Recovery restore point of "Openstack Cloud A" and vice versa. Both clouds have an independent Trilio installation integrated. These Trilio installations are writing their Backups to NFS targets. "Trilio A" is writing to "NFS A1" and "Trilio B" is writing to "NFS B1". The NFS Volumes used are getting synced against another NFS Volume on the other side. "NFS A1" is syncing with "NFS B2" and "NFS B1" is syncing with "NFS A2". The syncing process is set up independently from Trilio and will always favor the newer dataset.

This scenario will cover the Disaster Recovery of a single Workload and a complete Cloud. All processes are done be the Openstack administrator.

Prerequisites for the Disaster Recovery process

This runbook will assume that the following is true:

"Openstack Cloud A" and "Openstack Cloud B" both have an active Trilio installation with a valid license
"Openstack Cloud A" and "Openstack Cloud B" have free resources to host additional VMs
"Openstack Cloud A" and "Openstack Cloud B" have Tenants/Projects available that are the designated restore points for Tenant/Projects of the other side
Access to a user with the admin role permissions on domain level
One of the Openstack clouds is down/lost

For ease of writing will this runbook assume from here on, that "Openstack Cloud A" is down and the Workloads are getting restored into "Openstack Cloud B".

In the case of the usage of shared Tenant networks, beyond the floating IP, the following additional requirement is needed: All Tenant Networks, Routers, Ports, Floating IPs, and DNS Zones are created

Disaster Recovery of a single Workload

A single Workload can do a Disaster Recovery in this Scenario, while both Clouds are still active. To do so the following high-level process needs to be followed:

Copy the Workload directories to the configured NFS Volume
Make the right Mount-Paths available
Reassign the Workload
Restore the Workload
Clean up

Copy the Workload directories to the configured NFS Volume

This process only shows how to get a Workload from "Openstack Cloud A" to "Openstack Cloud B". The vice versa process is similar.

As only a single Workload is to be recovered it is more efficient to copy the data of that single Workload over to the "NFS B1" Volume, which is used by "Trilio B".

Mount "NFS B2" Volume to a Trilio VM

It is recommended to use the Trilio VM as a connector between both NFS Volumes, as the nova user is available on the Trilio VM.

# mount <NFS B2-IP/NFS B2-FQDN>:/<VOL-Path> /mnt

Identify the Workload on the "NFS B2" Volume

Trilio Workloads are identified by their ID und which they are stored on the Backup Target. See below example:

workload_ac9cae9b-5e1b-4899-930c-6aa0600a2105

In the case that the Workload ID is not known can available Metadata inside the Workload directories be used to identify the correct Workload.

/…/workload_<id>/workload_db <<< Contains User ID and Project ID of Workload owner
/…/workload_<id>/workload_vms_db <<< Contains VM IDs and VM Names of all VMs actively protected be the Workload

Copy the Workload

The identified workload needs to be copied with all subdirectories and files. Afterward, it is necessary to adjust the ownership to nova:nova with the right permissions.

# cp /mnt/workload_ac9cae9b-5e1b-4899-930c-6aa0600a2105 /var/triliovault-mounts/MTAuMTAuMi4yMDovdXBzdHJlYW0=/workload_ac9cae9b-5e1b-4899-930c-6aa0600a2105
# chown -R nova:nova /var/triliovault-mounts/MTAuMTAuMi4yMDovdXBzdHJlYW0=/workload_ac9cae9b-5e1b-4899-930c-6aa0600a2105
# chmod -R 644 /var/triliovault-mounts/MTAuMTAuMi4yMDovdXBzdHJlYW0=/workload_ac9cae9b-5e1b-4899-930c-6aa0600a2105

Make the Mount-Paths available

Trilio backups are using qcow2 backing files, which make every incremental backup a full synthetic backup. These backing files can be made visible using the qemu-img tool.

#qemu-img info bd57ec9b-c4ac-4a37-a4fd-5c9aa002c778
image: bd57ec9b-c4ac-4a37-a4fd-5c9aa002c778
file format: qcow2
virtual size: 1.0G (1073741824 bytes)
disk size: 516K
cluster_size: 65536

backing file: /var/triliovault-mounts/MTAuMTAuMi4yMDovdXBzdHJlYW0=/workload_ac9cae9b-5e1b-4899-930c-6aa0600a2105/snapshot_1415095d-c047-400b-8b05-c88e57011263/vm_id_38b620f1-24ae-41d7-b0ab-85ffc2d7958b/vm_res_id_d4ab3431-5ce3-4a8f-a90b-07606e2ffa33_vda/7c39eb6a-6e42-418e-8690-b6368ecaa7bb
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

The MTAuMTAuMi4yMDovdXBzdHJlYW0= part of the backing file path is the base64 hash value, which will be calculated upon the configuration of a Trilio installation for each provided NFS-Share.

This hash value is calculated based on the provided NFS-Share path: <NFS_IP>/<path> If even one character in the NFS-Share path is different between the provided NFS-Share paths a completely different hash value is generated.

Workloads, that have been moved between NFS-Shares, require that their incremental backups can follow the same path as on their original Source Cloud. To achieve this it is necessary to create the mount path on all compute nodes of the Target Cloud.

Afterwards a mount bind is used to make the workloads data accessible over the old and the new mount path. The following example shows the process of how to successfully identify the necessary mount points and create the mount bind.

Identify the base64 hash values

The used hash values can be calculated using the base64 tool in any Linux distribution.

# echo -n 10.10.2.20:/NFS_A1 | base64
MTAuMTAuMi4yMDovdXBzdHJlYW1fc291cmNl

# echo -n 10.20.3.22:/NFS_B2 | base64
MTAuMjAuMy4yMjovdXBzdHJlYW1fdGFyZ2V0

Create and bind the paths

Based on the identified base64 hash values the following paths are required on each Compute node.

/var/triliovault-mounts/MTAuMTAuMi4yMDovdXBzdHJlYW1fc291cmNl

and

/var/triliovault-mounts/MTAuMjAuMy4yMjovdXBzdHJlYW1fdGFyZ2V0

In the scenario of this runbook is the workload coming from the NFS_A1 NFS-Share, which means the mount path of that NFS-Share needs to be created and bound to the Target Cloud.

#mkdir /var/triliovault-mounts/MTAuMTAuMi4yMDovdXBzdHJlYW1fc291cmNl
#mount --bind 
/var/triliovault-mounts/MTAuMjAuMy4yMjovdXBzdHJlYW1fdGFyZ2V0/ /var/triliovault-mounts/MTAuMTAuMi4yMDovdXBzdHJlYW1fc291cmNl

To keep the desired mount past a reboot it is recommended to edit the fstab of all compute nodes accordingly.

#vi /etc/fstab
/var/triliovault-mounts/MTAuMjAuMy4yMjovdXBzdHJlYW1fdGFyZ2V0/ 		/ var/triliovault-mounts/ MTAuMTAuMi4yMDovdXBzdHJlYW1fc291cmNl	none		bind 		0 0

Reassign the workload

Trilio workloads have clear ownership. When a workload is moved to a different cloud it is necessary to change the ownership. The ownership can only be changed by Openstack administrators.

Add admin-user to required domains and projects

To fulfill the required tasks an admin role user is used. This user will be used until the workload has been restored. Therefore, it is necessary to provide this user access to the desired Target Project on the Target Cloud.

# source {customer admin rc file}  
# openstack role add Admin --user <my_admin_user> --user-domain <admin_domain> --domain <target_domain>  
# openstack role add Admin --user <my_admin_user> --user-domain <admin_domain> --project <target_project> --project-domain <target_domain>  
# openstack role add <Backup Trustee Role> --user <my_admin_user> --user-domain <admin_domain> --project <destination_project> --project-domain <target_domain>

Discover orphaned Workloads from NFS-Storage of Target Cloud

Each Trilio installation maintains a database of workloads that are known to the Trilio installation. Workloads that are not maintained by a specific Trilio installation, are from the perspective of that installation, orphaned workloads. An orphaned workload is a workload accessible on the NFS-Share, that is not assigned to any existing project in the Cloud the Trilio installation is protecting.

# workloadmgr workload-get-orphaned-workloads-list --migrate_cloud True    
+------------+--------------------------------------+----------------------------------+----------------------------------+  
|     Name   |                  ID                  |            Project ID            |  User ID                         |  
+------------+--------------------------------------+----------------------------------+----------------------------------+  
| Workload_1 | 6639525d-736a-40c5-8133-5caaddaaa8e9 | 4224d3acfd394cc08228cc8072861a35 |  329880dedb4cd357579a3279835f392 |  
| Workload_2 | 904e72f7-27bb-4235-9b31-13a636eb9c95 | 637a9ce3fd0d404cabf1a776696c9c04 |  329880dedb4cd357579a3279835f392 |  
+------------+--------------------------------------+----------------------------------+----------------------------------+

List available projects on Target Cloud in the Target Domain

The identified orphaned workloads need to be assigned to their new projects. The following provides the list of all available projects viewable by the used admin-user in the target_domain.

# openstack project list --domain <target_domain>  
+----------------------------------+----------+  
| ID                               | Name     |  
+----------------------------------+----------+  
| 01fca51462a44bfa821130dce9baac1a | project1 |  
| 33b4db1099ff4a65a4c1f69a14f932ee | project2 |  
| 9139e694eb984a4a979b5ae8feb955af | project3 |  
+----------------------------------+----------+

List available users on the Target Cloud in the Target Project that have the right backup trustee role

To allow project owners to work with the workloads as well will they get assigned to a user with the backup trustee role that is existing in the target project.

# openstack role assignment list --project <target_project> --project-domain <target_domain> --role <backup_trustee_role>
+----------------------------------+----------------------------------+-------+----------------------------------+--------+-----------+
| Role                             | User                             | Group | Project                          | Domain | Inherited |
+----------------------------------+----------------------------------+-------+----------------------------------+--------+-----------+
| 9fe2ff9ee4384b1894a90878d3e92bab | 72e65c264a694272928f5d84b73fe9ce |       | 8e16700ae3614da4ba80a4e57d60cdb9 |        | False     |
| 9fe2ff9ee4384b1894a90878d3e92bab | d5fbd79f4e834f51bfec08be6d3b2ff2 |       | 8e16700ae3614da4ba80a4e57d60cdb9 |        | False     |
| 9fe2ff9ee4384b1894a90878d3e92bab | f5b1d071816742fba6287d2c8ffcd6c4 |       | 8e16700ae3614da4ba80a4e57d60cdb9 |        | False     |
+----------------------------------+----------------------------------+-------+----------------------------------+--------+-----------+

Reassign the workload to the target project

Now that all informations have been gathered the workload can be reassigned to the target project.

# workloadmgr workload-reassign-workloads --new_tenant_id {target_project_id} --user_id {target_user_id} --workload_ids {workload_id} --migrate_cloud True    
+-----------+--------------------------------------+----------------------------------+----------------------------------+  
|    Name   |                  ID                  |            Project ID            |  User ID                         |  
+-----------+--------------------------------------+----------------------------------+----------------------------------+  
| project1  | 904e72f7-27bb-4235-9b31-13a636eb9c95 | 4f2a91274ce9491481db795dcb10b04f | 3e05cac47338425d827193ba374749cc |  
+-----------+--------------------------------------+----------------------------------+----------------------------------+

Verify the workload is available at the desired target_project

After the workload has been assigned to the new project it is recommended to verify the workload is managed by the Target Trilio and is assigned to the right project and user.

# workloadmgr workload-show ac9cae9b-5e1b-4899-930c-6aa0600a2105
+-------------------+------------------------------------------------------------------------------------------------------+
| Property          | Value                                                                                                |
+-------------------+------------------------------------------------------------------------------------------------------+
| availability_zone | nova                                                                                                 |
| created_at        | 2019-04-18T02:19:39.000000                                                                           |
| description       | Test Linux VMs                                                                                       |
| error_msg         | None                                                                                                 |
| id                | ac9cae9b-5e1b-4899-930c-6aa0600a2105                                                                 |
| instances         | [{"id": "38b620f1-24ae-41d7-b0ab-85ffc2d7958b", "name": "Test-Linux-1"}, {"id":                      |
|                   | "3fd869b2-16bd-4423-b389-18d19d37c8e0", "name": "Test-Linux-2"}]                                     |
| interval          | None                                                                                                 |
| jobschedule       | True                                                                                                 |
| name              | Test Linux                                                                                           |
| project_id        | 2fc4e2180c2745629753305591aeb93b                                                                     |
| scheduler_trust   | None                                                                                                 |
| status            | available                                                                                            |
| storage_usage     | {"usage": 60555264, "full": {"usage": 44695552, "snap_count": 1}, "incremental": {"usage": 15859712, |
|                   | "snap_count": 13}}                                                                                   |
| updated_at        | 2019-11-15T02:32:43.000000                                                                           |
| user_id           | 72e65c264a694272928f5d84b73fe9ce                                                                     |
| workload_type_id  | f82ce76f-17fe-438b-aa37-7a023058e50d                                                                 |
+-------------------+------------------------------------------------------------------------------------------------------+

Restore the workload

The reassigned workload can be restored using Horizon following the procedure described here.

This runbook will continue on the CLI only path.

Prepare the selective restore by getting the snapshot information

To be able to do the necessary selective restore a few pieces of information about the snapshot to be restored are required. The following process will provide all necessary information.

List all Snapshots of the workload to restore to identify the snapshot to restore

# workloadmgr snapshot-list --workload_id ac9cae9b-5e1b-4899-930c-6aa0600a2105 --all True

+----------------------------+--------------+--------------------------------------+--------------------------------------+---------------+-----------+-----------+
|         Created At         |     Name     |                  ID                  |             Workload ID              | Snapshot Type |   Status  |    Host   |
+----------------------------+--------------+--------------------------------------+--------------------------------------+---------------+-----------+-----------+
| 2019-11-02T02:30:02.000000 | jobscheduler | f5b8c3fd-c289-487d-9d50-fe27a6561d78 | ac9cae9b-5e1b-4899-930c-6aa0600a2105 |      full     | available | Upstream2 |
| 2019-11-03T02:30:02.000000 | jobscheduler | 7e39e544-537d-4417-853d-11463e7396f9 | ac9cae9b-5e1b-4899-930c-6aa0600a2105 |  incremental  | available | Upstream2 |
| 2019-11-04T02:30:02.000000 | jobscheduler | 0c086f3f-fa5d-425f-b07e-a1adcdcafea9 | ac9cae9b-5e1b-4899-930c-6aa0600a2105 |  incremental  | available | Upstream2 |
+----------------------------+--------------+--------------------------------------+--------------------------------------+---------------+-----------+-----------+

Get Snapshot Details with network details for the desired snapshot

# workloadmgr snapshot-show --output networks 7e39e544-537d-4417-853d-11463e7396f9

+-------------------+--------------------------------------+
| Snapshot property | Value                                |
+-------------------+--------------------------------------+
| description       | None                                 |
| host              | Upstream2                            |
| id                | 7e39e544-537d-4417-853d-11463e7396f9 |
| name              | jobscheduler                         |
| progress_percent  | 100                                  |
| restore_size      | 44040192 Bytes or Approx (42.0MB)    |
| restores_info     |                                      |
| size              | 1310720 Bytes or Approx (1.2MB)      |
| snapshot_type     | incremental                          |
| status            | available                            |
| time_taken        | 154 Seconds                          |
| uploaded_size     | 1310720                              |
| workload_id       | ac9cae9b-5e1b-4899-930c-6aa0600a2105 |
+-------------------+--------------------------------------+

+----------------+---------------------------------------------------------------------------------------------------------------------+
|   Instances    |                                                        Value                                                        |
+----------------+---------------------------------------------------------------------------------------------------------------------+
|     Status     |                                                      available                                                      |
| Security Group | [{u'name': u'Test', u'security_group_type': u'neutron'}, {u'name': u'default', u'security_group_type': u'neutron'}] |
|     Flavor     |                         {u'ephemeral': u'0', u'vcpus': u'1', u'disk': u'1', u'ram': u'512'}                         |
|      Name      |                                                     Test-Linux-1                                                    |
|       ID       |                                         38b620f1-24ae-41d7-b0ab-85ffc2d7958b                                        |
|                |                                                                                                                     |
|     Status     |                                                      available                                                      |
| Security Group | [{u'name': u'Test', u'security_group_type': u'neutron'}, {u'name': u'default', u'security_group_type': u'neutron'}] |
|     Flavor     |                         {u'ephemeral': u'0', u'vcpus': u'1', u'disk': u'1', u'ram': u'512'}                         |
|      Name      |                                                     Test-Linux-2                                                    |
|       ID       |                                         3fd869b2-16bd-4423-b389-18d19d37c8e0                                        |
|                |                                                                                                                     |
+----------------+---------------------------------------------------------------------------------------------------------------------+

+-------------+----------------------------------------------------------------------------------------------------------------------------------------------+
|   Networks  | Value                                                                                                                                        |
+-------------+----------------------------------------------------------------------------------------------------------------------------------------------+
|  ip_address | 172.20.20.20                                                                                                                                 |
|    vm_id    | 38b620f1-24ae-41d7-b0ab-85ffc2d7958b                                                                                                         |
|   network   | {u'subnet': {u'ip_version': 4, u'cidr': u'172.20.20.0/24', u'gateway_ip': u'172.20.20.1', u'id': u'3a756a89-d979-4cda-a7f3-dacad8594e44', 
u'name': u'Trilio Test'}, u'cidr': None, u'id': u'5f0e5d34-569d-42c9-97c2-df944f3924b1', u'name': u'Trilio_Test_Internal', u'network_type': u'neutron'}      |
| mac_address | fa:16:3e:74:58:bb                                                                                                                            |
|             |                                                                                                                                              |
|  ip_address | 172.20.20.13                                                                                                                                 |
|    vm_id    | 3fd869b2-16bd-4423-b389-18d19d37c8e0                                                                                                         |
|   network   | {u'subnet': {u'ip_version': 4, u'cidr': u'172.20.20.0/24', u'gateway_ip': u'172.20.20.1', u'id': u'3a756a89-d979-4cda-a7f3-dacad8594e44',
u'name': u'Trilio Test'}, u'cidr': None, u'id': u'5f0e5d34-569d-42c9-97c2-df944f3924b1', u'name': u'Trilio_Test_Internal', u'network_type': u'neutron'}      |
| mac_address | fa:16:3e:6b:46:ae                                                                                                                            |
+-------------+----------------------------------------------------------------------------------------------------------------------------------------------+

Get Snapshot Details with disk details for the desired Snapshot

[root@upstreamcontroller ~(keystone_admin)]# workloadmgr snapshot-show --output disks 7e39e544-537d-4417-853d-11463e7396f9

+-------------------+--------------------------------------+
| Snapshot property | Value                                |
+-------------------+--------------------------------------+
| description       | None                                 |
| host              | Upstream2                            |
| id                | 7e39e544-537d-4417-853d-11463e7396f9 |
| name              | jobscheduler                         |
| progress_percent  | 100                                  |
| restore_size      | 44040192 Bytes or Approx (42.0MB)    |
| restores_info     |                                      |
| size              | 1310720 Bytes or Approx (1.2MB)      |
| snapshot_type     | incremental                          |
| status            | available                            |
| time_taken        | 154 Seconds                          |
| uploaded_size     | 1310720                              |
| workload_id       | ac9cae9b-5e1b-4899-930c-6aa0600a2105 |
+-------------------+--------------------------------------+

+----------------+---------------------------------------------------------------------------------------------------------------------+
|   Instances    |                                                        Value                                                        |
+----------------+---------------------------------------------------------------------------------------------------------------------+
|     Status     |                                                      available                                                      |
| Security Group | [{u'name': u'Test', u'security_group_type': u'neutron'}, {u'name': u'default', u'security_group_type': u'neutron'}] |
|     Flavor     |                         {u'ephemeral': u'0', u'vcpus': u'1', u'disk': u'1', u'ram': u'512'}                         |
|      Name      |                                                     Test-Linux-1                                                    |
|       ID       |                                         38b620f1-24ae-41d7-b0ab-85ffc2d7958b                                        |
|                |                                                                                                                     |
|     Status     |                                                      available                                                      |
| Security Group | [{u'name': u'Test', u'security_group_type': u'neutron'}, {u'name': u'default', u'security_group_type': u'neutron'}] |
|     Flavor     |                         {u'ephemeral': u'0', u'vcpus': u'1', u'disk': u'1', u'ram': u'512'}                         |
|      Name      |                                                     Test-Linux-2                                                    |
|       ID       |                                         3fd869b2-16bd-4423-b389-18d19d37c8e0                                        |
|                |                                                                                                                     |
+----------------+---------------------------------------------------------------------------------------------------------------------+

+-------------------+--------------------------------------------------+
|       Vdisks      |                      Value                       |
+-------------------+--------------------------------------------------+
| volume_mountpoint |                     /dev/vda                     |
|    restore_size   |                     22020096                     |
|    resource_id    |       ebc2fdd0-3c4d-4548-b92d-0e16734b5d9a       |
|    volume_name    |       0027b140-a427-46cb-9ccf-7895c7624493       |
|    volume_type    |                       None                       |
|       label       |                       None                       |
|    volume_size    |                        1                         |
|     volume_id     |       0027b140-a427-46cb-9ccf-7895c7624493       |
| availability_zone |                       nova                       |
|       vm_id       |       38b620f1-24ae-41d7-b0ab-85ffc2d7958b       |
|      metadata     | {u'readonly': u'False', u'attached_mode': u'rw'} |
|                   |                                                  |
| volume_mountpoint |                     /dev/vda                     |
|    restore_size   |                     22020096                     |
|    resource_id    |       8007ed89-6a86-447e-badb-e49f1e92f57a       |
|    volume_name    |       2a7f9e78-7778-4452-af5b-8e2fa43853bd       |
|    volume_type    |                       None                       |
|       label       |                       None                       |
|    volume_size    |                        1                         |
|     volume_id     |       2a7f9e78-7778-4452-af5b-8e2fa43853bd       |
| availability_zone |                       nova                       |
|       vm_id       |       3fd869b2-16bd-4423-b389-18d19d37c8e0       |
|      metadata     | {u'readonly': u'False', u'attached_mode': u'rw'} |
|                   |                                                  |
+-------------------+--------------------------------------------------+

Prepare the selective restore by creating the restore.json file

The selective restore is using a restore.json file for the CLI command. This restore.json file needs to be adjusted according to the desired restore.

{
   u'description':u'<description of the restore>',
   u'oneclickrestore':False,
   u'restore_type':u'selective',
   u'type':u'openstack',
   u'name':u'<name of the restore>'
   u'openstack':{
      u'instances':[
         {
            u'name':u'<name instance 1>',
            u'availability_zone':u'<AZ instance 1>',
            u'nics':[ #####Leave empty for network topology restore
            ],
            u'vdisks':[
               {
                  u'id':u'<old disk id>',
                  u'new_volume_type':u'<new volume type name>',
                  u'availability_zone':u'<new cinder volume AZ>'
               }
            ],
            u'flavor':{
               u'ram':<RAM in MB>,
               u'ephemeral':<GB of ephemeral disk>,
               u'vcpus':<# vCPUs>,
               u'swap':u'<GB of Swap disk>',
               u'disk':<GB of boot disk>,
               u'id':u'<id of the flavor to use>'
            },
            u'include':<True/False>,
            u'id':u'<old id of the instance>'
         } #####Repeat for each instance in the snapshot
      ],
      u'restore_topology':<True/False>,
      u'networks_mapping':{
         u'networks':[ #####Leave empty for network topology restore
            
         ]
      }
   }
}

Run the selective restore

To do the actual restore use the following command:

# workloadmgr snapshot-selective-restore --filename restore.json {snapshot id}

Verify the restore

To verify the success of the restore from a Trilio perspective the restore status is checked.

[root@upstreamcontroller ~(keystone_admin)]# workloadmgr restore-list --snapshot_id 5928554d-a882-4881-9a5c-90e834c071af

+----------------------------+------------------+--------------------------------------+--------------------------------------+----------+-----------+
|         Created At         |       Name       |                  ID                  |             Snapshot ID              |   Size   |   Status  |
+----------------------------+------------------+--------------------------------------+--------------------------------------+----------+-----------+
| 2019-09-24T12:44:38.000000 | OneClick Restore | 5b4216d0-4bed-460f-8501-1589e7b45e01 | 5928554d-a882-4881-9a5c-90e834c071af | 41126400 | available |
+----------------------------+------------------+--------------------------------------+--------------------------------------+----------+-----------+

[root@upstreamcontroller ~(keystone_admin)]# workloadmgr restore-show 5b4216d0-4bed-460f-8501-1589e7b45e01
+------------------+------------------------------------------------------------------------------------------------------+
| Property         | Value                                                                                                |
+------------------+------------------------------------------------------------------------------------------------------+
| created_at       | 2019-09-24T12:44:38.000000                                                                           |
| description      | -                                                                                                    |
| error_msg        | None                                                                                                 |
| finished_at      | 2019-09-24T12:46:07.000000                                                                           |
| host             | Upstream2                                                                                            |
| id               | 5b4216d0-4bed-460f-8501-1589e7b45e01                                                                 |
| instances        | [{"status": "available", "id": "b8506f04-1b99-4ca8-839b-6f5d2c20d9aa", "name": "temp", "metadata":   |
|                  | {"instance_id": "c014a938-903d-43db-bfbb-ea4998ff1a0f", "production": "1", "config_drive": ""}}]     |
| name             | OneClick Restore                                                                                     |
| progress_msg     | Restore from snapshot is complete                                                                    |
| progress_percent | 100                                                                                                  |
| project_id       | 8e16700ae3614da4ba80a4e57d60cdb9                                                                     |
| restore_options  | {"description": "-", "oneclickrestore": true, "restore_type": "oneclick", "openstack": {"instances": |
|                  | [{"availability_zone": "US-West", "id": "c014a938-903d-43db-bfbb-ea4998ff1a0f", "name": "temp"}]},   |
|                  | "type": "openstack", "name": "OneClick Restore"}                                                     |
| restore_type     | restore                                                                                              |
| size             | 41126400                                                                                             |
| snapshot_id      | 5928554d-a882-4881-9a5c-90e834c071af                                                                 |
| status           | available                                                                                            |
| time_taken       | 89                                                                                                   |
| updated_at       | 2019-09-24T12:44:38.000000                                                                           |
| uploaded_size    | 41126400                                                                                             |
| user_id          | d5fbd79f4e834f51bfec08be6d3b2ff2                                                                     |
| warning_msg      | None                                                                                                 |
| workload_id      | 02b1aca2-c51a-454b-8c0f-99966314165e                                                                 |
+------------------+------------------------------------------------------------------------------------------------------+

Clean up

After the Desaster Recovery Process has been successfully completed it is recommended to bring the TVM installation back into its original state to be ready for the next DR process.

Delete the workload

Delete the workload that got restored.

# workloadmgr workload-delete <workload_id>

Remove the database entry

The Trilio database is following the Openstack standard of not deleting any database entries upon deletion of the cloud object. Any Workload, Snapshot or Restore, which gets deleted, is marked as deleted only.

To allow the Trilio installation to be ready for another disaster recovery it is necessary to completely delete the entries of the Workloads, which have been restored.

Trilio does provide and maintain a script to safely delete workload entries and all connected entities from the Trilio database.

This script can be found here: https://github.com/trilioData/solutions/tree/master/openstack/CleanWlmDatabase

Remove the admin user from the project

After all restores for the target project have been achieved it is recommended to remove the used admin user from the project again.

# source {customer admin rc file}  
# openstack role remove Admin --user <my_admin_user> --user-domain <admin_domain> --domain <target_domain>  
# openstack role remove Admin --user <my_admin_user> --user-domain <admin_domain> --project <target_project> --project-domain <target_domain>  
# openstack role remove <Backup Trustee Role> --user <my_admin_user> --user-domain <admin_domain> --project <destination_project> --project-domain <target_domain>

Disaster Recovery of a complete cloud

This Scenario will cover the Disaster Recovery of a full cloud. It is assumed that the source cloud is down or lost completly. To do the disaster recovery the following high-level process needs to be followed:

Reconfigure the Target Trilio installation
Make the right Mount-Paths available
Reassign the Workload
Restore the Workload
Reconfigure the Target Trilio installation back to the original one
Clean up

Reconfigure the Target Trilio installation

Before the Desaster Recovery Process can start is it necessary to make the backups to be restored available for the Trilio installation. The following steps need to be done to completely reconfigure the Trilio installation.

During the reconfiguration process will all backups of the Target Region be on hold and it is not recommended to create new Backup Jobs until the Desaster Recovery Process has finished and the original Trilio configuration has been restored.

Add NFS B2 to the Trilio Appliance Cluster

To add the NFS-Vol2 to the Trilio Appliance cluster the Trilio can either be fully reconfigured to use both NFS Volumes or it is possible to edit the configuration file and then restart all services. This procedure describes how to edit the conf file and restart the services. This needs to be repeated on every Trilio Appliance.

Edit the workloadmgr.conf

# vi /etc/workloadmgr/workloadmgr.conf

Look for the line defining the NFS mounts

vault_storage_nfs_export = <NFS_B1/NFS_B1-FQDN>:/<VOL-B1-Path>

Add NFS B2 to that as comma-seperated list. Space is not necessary, but can be set.

vault_storage_nfs_export = <NFS-IP/NFS-FQDN>:/<VOL-1-Path>,<NFS-IP/NFS-FQDN>:/<VOL—2-Path>

Write and close the workloadmgr.conf

Restart the wlm-workloads service

# systemctl restart wlm-workloads

Add NFS B2 to the Trilio Datamovers

Trilio is integrating natively into the Openstack deployment tools. When using the Red Hat director or JuJu charms it is recommended to adapt the environment files for these orchestrators and update the Datamovers through them.

To add the NFS B2 to the Trilio Datamovers manually the tvault-contego.conf file needs to be edited and the service restarted.

Edit the tvault-contego.conf

# vi /etc/tvault-contego/tvault-contego.conf

Look for the line defining the NFS mounts

vault_storage_nfs_export = <NFS_B1-IP/NFS_B1-FQDN>:/<VOL-B1-Path>

Add NFS B2 to that as comma-seperated list. Space is not necessary, but can be set.

vault_storage_nfs_export = <NFS_B1-IP/NFS-FQDN>:/<VOL-B1-Path>,<NFS_B2-IP/NFS-FQDN>:/<VOL—B2-Path>

Write and close the tvault-contego.conf

Restart the tvault-contego service

# systemctl restart tvault-contego

Make the Mount-Paths available

Trilio backups are using qcow2 backing files, which make every incremental backup a full synthetic backup. These backing files can be made visible using the qemu-img tool.

#qemu-img info bd57ec9b-c4ac-4a37-a4fd-5c9aa002c778
image: bd57ec9b-c4ac-4a37-a4fd-5c9aa002c778
file format: qcow2
virtual size: 1.0G (1073741824 bytes)
disk size: 516K
cluster_size: 65536

backing file: /var/triliovault-mounts/MTAuMTAuMi4yMDovdXBzdHJlYW0=/workload_ac9cae9b-5e1b-4899-930c-6aa0600a2105/snapshot_1415095d-c047-400b-8b05-c88e57011263/vm_id_38b620f1-24ae-41d7-b0ab-85ffc2d7958b/vm_res_id_d4ab3431-5ce3-4a8f-a90b-07606e2ffa33_vda/7c39eb6a-6e42-418e-8690-b6368ecaa7bb
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

The MTAuMTAuMi4yMDovdXBzdHJlYW0= part of the backing file path is the base64 hash value, which will be calculated upon the configuration of a Trilio installation for each provided NFS-Share.

Identify the base64 hash values

The used hash values can be calculated using the base64 tool in any Linux distribution.

# echo -n 10.10.2.20:/NFS_A1 | base64
MTAuMTAuMi4yMDovdXBzdHJlYW1fc291cmNl

# echo -n 10.20.3.22:/NFS_B2 | base64
MTAuMjAuMy4yMjovdXBzdHJlYW1fdGFyZ2V0

Create and bind the paths

Based on the identified base64 hash values the following paths are required on each Compute node.

/var/triliovault-mounts/MTAuMTAuMi4yMDovdXBzdHJlYW1fc291cmNl

and

/var/triliovault-mounts/MTAuMjAuMy4yMjovdXBzdHJlYW1fdGFyZ2V0

In the scenario of this runbook is the workload coming from the NFS_A1 NFS-Share, which means the mount path of that NFS-Share needs to be created and bound to the Target Cloud.

#mkdir /var/triliovault-mounts/MTAuMTAuMi4yMDovdXBzdHJlYW1fc291cmNl
#mount --bind 
/var/triliovault-mounts/MTAuMjAuMy4yMjovdXBzdHJlYW1fdGFyZ2V0/ /var/triliovault-mounts/MTAuMTAuMi4yMDovdXBzdHJlYW1fc291cmNl

To keep the desired mount past a reboot it is recommended to edit the fstab of all compute nodes accordingly.

#vi /etc/fstab
/var/triliovault-mounts/MTAuMjAuMy4yMjovdXBzdHJlYW1fdGFyZ2V0/ 		/ var/triliovault-mounts/ MTAuMTAuMi4yMDovdXBzdHJlYW1fc291cmNl	none		bind 		0 0

Reassign the workload

Trilio workloads have clear ownership. When a workload is moved to a different cloud it is necessary to change the ownership. The ownership can only be changed by Openstack administrators.

Add admin-user to required domains and projects

# source {customer admin rc file}  
# openstack role add Admin --user <my_admin_user> --user-domain <admin_domain> --domain <target_domain>  
# openstack role add Admin --user <my_admin_user> --user-domain <admin_domain> --project <target_project> --project-domain <target_domain>  
# openstack role add <Backup Trustee Role> --user <my_admin_user> --user-domain <admin_domain> --project <destination_project> --project-domain <target_domain>

Discover orphaned Workloads from NFS-Storage of Target Cloud

# workloadmgr workload-get-orphaned-workloads-list --migrate_cloud True    
+------------+--------------------------------------+----------------------------------+----------------------------------+  
|     Name   |                  ID                  |            Project ID            |  User ID                         |  
+------------+--------------------------------------+----------------------------------+----------------------------------+  
| Workload_1 | 6639525d-736a-40c5-8133-5caaddaaa8e9 | 4224d3acfd394cc08228cc8072861a35 |  329880dedb4cd357579a3279835f392 |  
| Workload_2 | 904e72f7-27bb-4235-9b31-13a636eb9c95 | 637a9ce3fd0d404cabf1a776696c9c04 |  329880dedb4cd357579a3279835f392 |  
+------------+--------------------------------------+----------------------------------+----------------------------------+

List available projects on Target Cloud in the Target Domain

The identified orphaned workloads need to be assigned to their new projects. The following provides the list of all available projects viewable by the used admin-user in the target_domain.

# openstack project list --domain <target_domain>  
+----------------------------------+----------+  
| ID                               | Name     |  
+----------------------------------+----------+  
| 01fca51462a44bfa821130dce9baac1a | project1 |  
| 33b4db1099ff4a65a4c1f69a14f932ee | project2 |  
| 9139e694eb984a4a979b5ae8feb955af | project3 |  
+----------------------------------+----------+

List available users on the Target Cloud in the Target Project that have the right backup trustee role

To allow project owners to work with the workloads as well will they get assigned to a user with the backup trustee role that is existing in the target project.

# openstack role assignment list --project <target_project> --project-domain <target_domain> --role <backup_trustee_role>
+----------------------------------+----------------------------------+-------+----------------------------------+--------+-----------+
| Role                             | User                             | Group | Project                          | Domain | Inherited |
+----------------------------------+----------------------------------+-------+----------------------------------+--------+-----------+
| 9fe2ff9ee4384b1894a90878d3e92bab | 72e65c264a694272928f5d84b73fe9ce |       | 8e16700ae3614da4ba80a4e57d60cdb9 |        | False     |
| 9fe2ff9ee4384b1894a90878d3e92bab | d5fbd79f4e834f51bfec08be6d3b2ff2 |       | 8e16700ae3614da4ba80a4e57d60cdb9 |        | False     |
| 9fe2ff9ee4384b1894a90878d3e92bab | f5b1d071816742fba6287d2c8ffcd6c4 |       | 8e16700ae3614da4ba80a4e57d60cdb9 |        | False     |
+----------------------------------+----------------------------------+-------+----------------------------------+--------+-----------+

Reassign the workload to the target project

Now that all informations have been gathered the workload can be reassigned to the target project.

# workloadmgr workload-reassign-workloads --new_tenant_id {target_project_id} --user_id {target_user_id} --workload_ids {workload_id} --migrate_cloud True    
+-----------+--------------------------------------+----------------------------------+----------------------------------+  
|    Name   |                  ID                  |            Project ID            |  User ID                         |  
+-----------+--------------------------------------+----------------------------------+----------------------------------+  
| project1  | 904e72f7-27bb-4235-9b31-13a636eb9c95 | 4f2a91274ce9491481db795dcb10b04f | 3e05cac47338425d827193ba374749cc |  
+-----------+--------------------------------------+----------------------------------+----------------------------------+

Verify the workload is available at the desired target_project

After the workload has been assigned to the new project it is recommended to verify the workload is managed by the Target Trilio and is assigned to the right project and user.

# workloadmgr workload-show ac9cae9b-5e1b-4899-930c-6aa0600a2105
+-------------------+------------------------------------------------------------------------------------------------------+
| Property          | Value                                                                                                |
+-------------------+------------------------------------------------------------------------------------------------------+
| availability_zone | nova                                                                                                 |
| created_at        | 2019-04-18T02:19:39.000000                                                                           |
| description       | Test Linux VMs                                                                                       |
| error_msg         | None                                                                                                 |
| id                | ac9cae9b-5e1b-4899-930c-6aa0600a2105                                                                 |
| instances         | [{"id": "38b620f1-24ae-41d7-b0ab-85ffc2d7958b", "name": "Test-Linux-1"}, {"id":                      |
|                   | "3fd869b2-16bd-4423-b389-18d19d37c8e0", "name": "Test-Linux-2"}]                                     |
| interval          | None                                                                                                 |
| jobschedule       | True                                                                                                 |
| name              | Test Linux                                                                                           |
| project_id        | 2fc4e2180c2745629753305591aeb93b                                                                     |
| scheduler_trust   | None                                                                                                 |
| status            | available                                                                                            |
| storage_usage     | {"usage": 60555264, "full": {"usage": 44695552, "snap_count": 1}, "incremental": {"usage": 15859712, |
|                   | "snap_count": 13}}                                                                                   |
| updated_at        | 2019-11-15T02:32:43.000000                                                                           |
| user_id           | 72e65c264a694272928f5d84b73fe9ce                                                                     |
| workload_type_id  | f82ce76f-17fe-438b-aa37-7a023058e50d                                                                 |
+-------------------+------------------------------------------------------------------------------------------------------+

Restore the workload

The reassigned workload can be restored using Horizon following the procedure described here.

This runbook will continue on the CLI only path.

Prepare the selective restore by getting the snapshot information

To be able to do the necessary selective restore a few pieces of information about the snapshot to be restored are required. The following process will provide all necessary information.

List all Snapshots of the workload to restore to identify the snapshot to restore

# workloadmgr snapshot-list --workload_id ac9cae9b-5e1b-4899-930c-6aa0600a2105 --all True

+----------------------------+--------------+--------------------------------------+--------------------------------------+---------------+-----------+-----------+
|         Created At         |     Name     |                  ID                  |             Workload ID              | Snapshot Type |   Status  |    Host   |
+----------------------------+--------------+--------------------------------------+--------------------------------------+---------------+-----------+-----------+
| 2019-11-02T02:30:02.000000 | jobscheduler | f5b8c3fd-c289-487d-9d50-fe27a6561d78 | ac9cae9b-5e1b-4899-930c-6aa0600a2105 |      full     | available | Upstream2 |
| 2019-11-03T02:30:02.000000 | jobscheduler | 7e39e544-537d-4417-853d-11463e7396f9 | ac9cae9b-5e1b-4899-930c-6aa0600a2105 |  incremental  | available | Upstream2 |
| 2019-11-04T02:30:02.000000 | jobscheduler | 0c086f3f-fa5d-425f-b07e-a1adcdcafea9 | ac9cae9b-5e1b-4899-930c-6aa0600a2105 |  incremental  | available | Upstream2 |
+----------------------------+--------------+--------------------------------------+--------------------------------------+---------------+-----------+-----------+

Get Snapshot Details with network details for the desired snapshot

# workloadmgr snapshot-show --output networks 7e39e544-537d-4417-853d-11463e7396f9

+-------------------+--------------------------------------+
| Snapshot property | Value                                |
+-------------------+--------------------------------------+
| description       | None                                 |
| host              | Upstream2                            |
| id                | 7e39e544-537d-4417-853d-11463e7396f9 |
| name              | jobscheduler                         |
| progress_percent  | 100                                  |
| restore_size      | 44040192 Bytes or Approx (42.0MB)    |
| restores_info     |                                      |
| size              | 1310720 Bytes or Approx (1.2MB)      |
| snapshot_type     | incremental                          |
| status            | available                            |
| time_taken        | 154 Seconds                          |
| uploaded_size     | 1310720                              |
| workload_id       | ac9cae9b-5e1b-4899-930c-6aa0600a2105 |
+-------------------+--------------------------------------+

+----------------+---------------------------------------------------------------------------------------------------------------------+
|   Instances    |                                                        Value                                                        |
+----------------+---------------------------------------------------------------------------------------------------------------------+
|     Status     |                                                      available                                                      |
| Security Group | [{u'name': u'Test', u'security_group_type': u'neutron'}, {u'name': u'default', u'security_group_type': u'neutron'}] |
|     Flavor     |                         {u'ephemeral': u'0', u'vcpus': u'1', u'disk': u'1', u'ram': u'512'}                         |
|      Name      |                                                     Test-Linux-1                                                    |
|       ID       |                                         38b620f1-24ae-41d7-b0ab-85ffc2d7958b                                        |
|                |                                                                                                                     |
|     Status     |                                                      available                                                      |
| Security Group | [{u'name': u'Test', u'security_group_type': u'neutron'}, {u'name': u'default', u'security_group_type': u'neutron'}] |
|     Flavor     |                         {u'ephemeral': u'0', u'vcpus': u'1', u'disk': u'1', u'ram': u'512'}                         |
|      Name      |                                                     Test-Linux-2                                                    |
|       ID       |                                         3fd869b2-16bd-4423-b389-18d19d37c8e0                                        |
|                |                                                                                                                     |
+----------------+---------------------------------------------------------------------------------------------------------------------+

+-------------+----------------------------------------------------------------------------------------------------------------------------------------------+
|   Networks  | Value                                                                                                                                        |
+-------------+----------------------------------------------------------------------------------------------------------------------------------------------+
|  ip_address | 172.20.20.20                                                                                                                                 |
|    vm_id    | 38b620f1-24ae-41d7-b0ab-85ffc2d7958b                                                                                                         |
|   network   | {u'subnet': {u'ip_version': 4, u'cidr': u'172.20.20.0/24', u'gateway_ip': u'172.20.20.1', u'id': u'3a756a89-d979-4cda-a7f3-dacad8594e44', 
u'name': u'Trilio Test'}, u'cidr': None, u'id': u'5f0e5d34-569d-42c9-97c2-df944f3924b1', u'name': u'Trilio_Test_Internal', u'network_type': u'neutron'}      |
| mac_address | fa:16:3e:74:58:bb                                                                                                                            |
|             |                                                                                                                                              |
|  ip_address | 172.20.20.13                                                                                                                                 |
|    vm_id    | 3fd869b2-16bd-4423-b389-18d19d37c8e0                                                                                                         |
|   network   | {u'subnet': {u'ip_version': 4, u'cidr': u'172.20.20.0/24', u'gateway_ip': u'172.20.20.1', u'id': u'3a756a89-d979-4cda-a7f3-dacad8594e44',
u'name': u'Trilio Test'}, u'cidr': None, u'id': u'5f0e5d34-569d-42c9-97c2-df944f3924b1', u'name': u'Trilio_Test_Internal', u'network_type': u'neutron'}      |
| mac_address | fa:16:3e:6b:46:ae                                                                                                                            |
+-------------+----------------------------------------------------------------------------------------------------------------------------------------------+

Get Snapshot Details with disk details for the desired Snapshot

[root@upstreamcontroller ~(keystone_admin)]# workloadmgr snapshot-show --output disks 7e39e544-537d-4417-853d-11463e7396f9

+-------------------+--------------------------------------+
| Snapshot property | Value                                |
+-------------------+--------------------------------------+
| description       | None                                 |
| host              | Upstream2                            |
| id                | 7e39e544-537d-4417-853d-11463e7396f9 |
| name              | jobscheduler                         |
| progress_percent  | 100                                  |
| restore_size      | 44040192 Bytes or Approx (42.0MB)    |
| restores_info     |                                      |
| size              | 1310720 Bytes or Approx (1.2MB)      |
| snapshot_type     | incremental                          |
| status            | available                            |
| time_taken        | 154 Seconds                          |
| uploaded_size     | 1310720                              |
| workload_id       | ac9cae9b-5e1b-4899-930c-6aa0600a2105 |
+-------------------+--------------------------------------+

+----------------+---------------------------------------------------------------------------------------------------------------------+
|   Instances    |                                                        Value                                                        |
+----------------+---------------------------------------------------------------------------------------------------------------------+
|     Status     |                                                      available                                                      |
| Security Group | [{u'name': u'Test', u'security_group_type': u'neutron'}, {u'name': u'default', u'security_group_type': u'neutron'}] |
|     Flavor     |                         {u'ephemeral': u'0', u'vcpus': u'1', u'disk': u'1', u'ram': u'512'}                         |
|      Name      |                                                     Test-Linux-1                                                    |
|       ID       |                                         38b620f1-24ae-41d7-b0ab-85ffc2d7958b                                        |
|                |                                                                                                                     |
|     Status     |                                                      available                                                      |
| Security Group | [{u'name': u'Test', u'security_group_type': u'neutron'}, {u'name': u'default', u'security_group_type': u'neutron'}] |
|     Flavor     |                         {u'ephemeral': u'0', u'vcpus': u'1', u'disk': u'1', u'ram': u'512'}                         |
|      Name      |                                                     Test-Linux-2                                                    |
|       ID       |                                         3fd869b2-16bd-4423-b389-18d19d37c8e0                                        |
|                |                                                                                                                     |
+----------------+---------------------------------------------------------------------------------------------------------------------+

+-------------------+--------------------------------------------------+
|       Vdisks      |                      Value                       |
+-------------------+--------------------------------------------------+
| volume_mountpoint |                     /dev/vda                     |
|    restore_size   |                     22020096                     |
|    resource_id    |       ebc2fdd0-3c4d-4548-b92d-0e16734b5d9a       |
|    volume_name    |       0027b140-a427-46cb-9ccf-7895c7624493       |
|    volume_type    |                       None                       |
|       label       |                       None                       |
|    volume_size    |                        1                         |
|     volume_id     |       0027b140-a427-46cb-9ccf-7895c7624493       |
| availability_zone |                       nova                       |
|       vm_id       |       38b620f1-24ae-41d7-b0ab-85ffc2d7958b       |
|      metadata     | {u'readonly': u'False', u'attached_mode': u'rw'} |
|                   |                                                  |
| volume_mountpoint |                     /dev/vda                     |
|    restore_size   |                     22020096                     |
|    resource_id    |       8007ed89-6a86-447e-badb-e49f1e92f57a       |
|    volume_name    |       2a7f9e78-7778-4452-af5b-8e2fa43853bd       |
|    volume_type    |                       None                       |
|       label       |                       None                       |
|    volume_size    |                        1                         |
|     volume_id     |       2a7f9e78-7778-4452-af5b-8e2fa43853bd       |
| availability_zone |                       nova                       |
|       vm_id       |       3fd869b2-16bd-4423-b389-18d19d37c8e0       |
|      metadata     | {u'readonly': u'False', u'attached_mode': u'rw'} |
|                   |                                                  |
+-------------------+--------------------------------------------------+

Prepare the selective restore by creating the restore.json file

The selective restore is using a restore.json file for the CLI command. This restore.json file needs to be adjusted according to the desired restore.

{
   u'description':u'<description of the restore>',
   u'oneclickrestore':False,
   u'restore_type':u'selective',
   u'type':u'openstack',
   u'name':u'<name of the restore>'
   u'openstack':{
      u'instances':[
         {
            u'name':u'<name instance 1>',
            u'availability_zone':u'<AZ instance 1>',
            u'nics':[ #####Leave empty for network topology restore
            ],
            u'vdisks':[
               {
                  u'id':u'<old disk id>',
                  u'new_volume_type':u'<new volume type name>',
                  u'availability_zone':u'<new cinder volume AZ>'
               }
            ],
            u'flavor':{
               u'ram':<RAM in MB>,
               u'ephemeral':<GB of ephemeral disk>,
               u'vcpus':<# vCPUs>,
               u'swap':u'<GB of Swap disk>',
               u'disk':<GB of boot disk>,
               u'id':u'<id of the flavor to use>'
            },
            u'include':<True/False>,
            u'id':u'<old id of the instance>'
         } #####Repeat for each instance in the snapshot
      ],
      u'restore_topology':<True/False>,
      u'networks_mapping':{
         u'networks':[ #####Leave empty for network topology restore
            
         ]
      }
   }
}

Run the selective restore

To do the actual restore use the following command:

# workloadmgr snapshot-selective-restore --filename restore.json {snapshot id}

Verify the restore

To verify the success of the restore from a Trilio perspective the restore status is checked.

[root@upstreamcontroller ~(keystone_admin)]# workloadmgr restore-list --snapshot_id 5928554d-a882-4881-9a5c-90e834c071af

+----------------------------+------------------+--------------------------------------+--------------------------------------+----------+-----------+
|         Created At         |       Name       |                  ID                  |             Snapshot ID              |   Size   |   Status  |
+----------------------------+------------------+--------------------------------------+--------------------------------------+----------+-----------+
| 2019-09-24T12:44:38.000000 | OneClick Restore | 5b4216d0-4bed-460f-8501-1589e7b45e01 | 5928554d-a882-4881-9a5c-90e834c071af | 41126400 | available |
+----------------------------+------------------+--------------------------------------+--------------------------------------+----------+-----------+

[root@upstreamcontroller ~(keystone_admin)]# workloadmgr restore-show 5b4216d0-4bed-460f-8501-1589e7b45e01
+------------------+------------------------------------------------------------------------------------------------------+
| Property         | Value                                                                                                |
+------------------+------------------------------------------------------------------------------------------------------+
| created_at       | 2019-09-24T12:44:38.000000                                                                           |
| description      | -                                                                                                    |
| error_msg        | None                                                                                                 |
| finished_at      | 2019-09-24T12:46:07.000000                                                                           |
| host             | Upstream2                                                                                            |
| id               | 5b4216d0-4bed-460f-8501-1589e7b45e01                                                                 |
| instances        | [{"status": "available", "id": "b8506f04-1b99-4ca8-839b-6f5d2c20d9aa", "name": "temp", "metadata":   |
|                  | {"instance_id": "c014a938-903d-43db-bfbb-ea4998ff1a0f", "production": "1", "config_drive": ""}}]     |
| name             | OneClick Restore                                                                                     |
| progress_msg     | Restore from snapshot is complete                                                                    |
| progress_percent | 100                                                                                                  |
| project_id       | 8e16700ae3614da4ba80a4e57d60cdb9                                                                     |
| restore_options  | {"description": "-", "oneclickrestore": true, "restore_type": "oneclick", "openstack": {"instances": |
|                  | [{"availability_zone": "US-West", "id": "c014a938-903d-43db-bfbb-ea4998ff1a0f", "name": "temp"}]},   |
|                  | "type": "openstack", "name": "OneClick Restore"}                                                     |
| restore_type     | restore                                                                                              |
| size             | 41126400                                                                                             |
| snapshot_id      | 5928554d-a882-4881-9a5c-90e834c071af                                                                 |
| status           | available                                                                                            |
| time_taken       | 89                                                                                                   |
| updated_at       | 2019-09-24T12:44:38.000000                                                                           |
| uploaded_size    | 41126400                                                                                             |
| user_id          | d5fbd79f4e834f51bfec08be6d3b2ff2                                                                     |
| warning_msg      | None                                                                                                 |
| workload_id      | 02b1aca2-c51a-454b-8c0f-99966314165e                                                                 |
+------------------+------------------------------------------------------------------------------------------------------+

Reconfigure the Target Trilio installation back to the original one

After the Desaster Recovery Process has finished it is necessary to return the Trilio installation to its original configuration. The following steps need to be done to completely reconfigure the Trilio installation.

Delete NFS B2 to the Trilio Appliance Cluster

Edit the workloadmgr.conf

# vi /etc/workloadmgr/workloadmgr.conf

Look for the line defining the NFS mounts

vault_storage_nfs_export = <NFS_B1-IP/NFS-FQDN>:/<VOL-B1-Path>,<NFS_B2-IP/NFS-FQDN>:/<VOL—B2-Path>

Delete NFS B2 from the comma-seperated list

vault_storage_nfs_export = <NFS_B1-IP/NFS_B1-FQDN>:/<VOL-B1-Path>

Write and close the workloadmgr.conf

Restart the wlm-workloads service

# systemctl restart wlm-workloads

Delete NFS B2 to the Trilio Datamovers

To add the NFS B2 to the Trilio Datamovers manually the tvault-contego.conf file needs to be edited and the service restarted.

Edit the tvault-contego.conf

# vi /etc/tvault-contego/tvault-contego.conf

Look for the line defining the NFS mounts

vault_storage_nfs_export = <NFS_B1-IP/NFS-FQDN>:/<VOL-B1-Path>,<NFS_B2-IP/NFS-FQDN>:/<VOL—B2-Path>

Add NFS B2 to that as comma-seperated list. Space is not necessary, but can be set.

vault_storage_nfs_export = <NFS-IP/NFS-FQDN>:/<VOL-1-Path>

Write and close the tvault-contego.conf

Restart the tvault-contego service

# systemctl restart tvault-contego

Clean up

After the Desaster Recovery Process has been successfully completed and the Trilio installation reconfigured to its original state, it is recommended to do the following additional steps to be ready for the next Disaster Recovery process.

Remove the database entry

To allow the Trilio installation to be ready for another disaster recovery it is necessary to completely delete the entries of the Workloads, which have been restored.

Trilio does provide and maintain a script to safely delete workload entries and all connected entities from the Trilio database.

This script can be found here: https://github.com/trilioData/solutions/tree/master/openstack/CleanWlmDatabase

Remove the admin user from the project

After all restores for the target project have been achieved it is recommended to remove the used admin user from the project again.

# source {customer admin rc file}  
# openstack role remove Admin --user <my_admin_user> --user-domain <admin_domain> --domain <target_domain>  
# openstack role remove Admin --user <my_admin_user> --user-domain <admin_domain> --project <target_project> --project-domain <target_domain>  
# openstack role remove <Backup Trustee Role> --user <my_admin_user> --user-domain <admin_domain> --project <destination_project> --project-domain <target_domain>

Snapshots

List Snapshots

GET https://$(tvm_address):8780/v1/$(tenant_id)/snapshots

Lists all Snapshots.

Path Parameters

Name

Type

Description

Query Parameters

Name

Type

Description

Headers

Name

Type

Description

HTTP/1.1 200 OK
Server: nginx/1.16.1
Date: Wed, 04 Nov 2020 12:58:38 GMT
Content-Type: application/json
Content-Length: 266
Connection: keep-alive
X-Compute-Request-Id: req-ed391cf9-aa56-4c53-8153-fd7fb238c4b9

{
   "snapshots":[
      {
         "id":"1ff16412-a0cd-4e6a-9b4a-b5d4440fffc4",
         "created_at":"2020-11-02T14:03:18.000000",
         "status":"available",
         "snapshot_type":"full",
         "workload_id":"18b809de-d7c8-41e2-867d-4a306407fb11",
         "name":"snapshot",
         "description":"-",
         "host":"TVM1"
      }
   ]
}

Take Snapshot

POST https://$(tvm_address):8780/v1/$(tenant_id)/workloads/<workload_id>

Path Parameters

Query Parameters

Headers

HTTP/1.1 200 OK
Server: nginx/1.16.1
Date: Wed, 04 Nov 2020 13:58:38 GMT
Content-Type: application/json
Content-Length: 283
Connection: keep-alive
X-Compute-Request-Id: req-fb8dc382-e5de-4665-8d88-c75b2e473f5c

{
   "snapshot":{
      "id":"2e56d167-bad7-43c7-8ede-a613c3fe7844",
      "created_at":"2020-11-04T13:58:37.694637",
      "status":"creating",
      "snapshot_type":"full",
      "workload_id":"18b809de-d7c8-41e2-867d-4a306407fb11",
      "name":"API taken 2",
      "description":"API taken description 2",
      "host":""
   }
}

Body format

When creating a Snapshot it is possible to provide additional information

This Body is completely optional

{
   "snapshot":{
      "is_scheduled":<true/false>,
      "name":"<name>",
      "description":"<description>"
   }
}

Show Snapshot

GET https://$(tvm_address):8780/v1/$(tenant_id)/snapshots/<snapshot_id>

Shows the details of a specified Snapshot

Path Parameters

Headers

HTTP/1.1 200 OK
Server: nginx/1.16.1
Date: Wed, 04 Nov 2020 14:07:18 GMT
Content-Type: application/json
Content-Length: 6609
Connection: keep-alive
X-Compute-Request-Id: req-f88fb28f-f4ce-4585-9c3c-ebe08a3f60cd

{
   "snapshot":{
      "id":"2e56d167-bad7-43c7-8ede-a613c3fe7844",
      "created_at":"2020-11-04T13:58:37.000000",
      "updated_at":"2020-11-04T14:06:03.000000",
      "finished_at":"2020-11-04T14:06:03.000000",
      "user_id":"ccddc7e7a015487fa02920f4d4979779",
      "project_id":"c76b3355a164498aa95ddbc960adc238",
      "status":"available",
      "snapshot_type":"full",
      "workload_id":"18b809de-d7c8-41e2-867d-4a306407fb11",
      "instances":[
         {
            "id":"67d6a100-fee6-4aa5-83a1-66b070d2eabe",
            "name":"cirros-2",
            "status":"available",
            "metadata":{
               "availability_zone":"nova",
               "config_drive":"",
               "data_transfer_time":"0",
               "object_store_transfer_time":"0",
               "root_partition_type":"Linux",
               "trilio_ordered_interfaces":"192.168.100.80",
               "vm_metadata":"{\"workload_name\": \"Workload_1\", \"workload_id\": \"18b809de-d7c8-41e2-867d-4a306407fb11\", \"trilio_ordered_interfaces\": \"192.168.100.80\", \"config_drive\": \"\"}",
               "workload_id":"18b809de-d7c8-41e2-867d-4a306407fb11",
               "workload_name":"Workload_1"
            },
            "flavor":{
               "vcpus":"1",
               "ram":"512",
               "disk":"1",
               "ephemeral":"0"
            },
            "security_group":[
               {
                  "name":"default",
                  "security_group_type":"neutron"
               }
            ],
            "nics":[
               {
                  "mac_address":"fa:16:3e:cf:10:91",
                  "ip_address":"192.168.100.80",
                  "network":{
                     "id":"5fb7027d-a2ac-4a21-9ee1-438c281d2b26",
                     "name":"robert_internal",
                     "cidr":null,
                     "network_type":"neutron",
                     "subnet":{
                        "id":"b7b54304-aa82-4d50-91e6-66445ab56db4",
                        "name":"robert_internal",
                        "cidr":"192.168.100.0/24",
                        "ip_version":4,
                        "gateway_ip":"192.168.100.1"
                     }
                  }
               }
            ],
            "vdisks":[
               {
                  "label":null,
                  "resource_id":"fa888089-5715-4228-9e5a-699f8f9d59ba",
                  "restore_size":1073741824,
                  "vm_id":"67d6a100-fee6-4aa5-83a1-66b070d2eabe",
                  "volume_id":"51491d30-9818-4332-b056-1f174e65d3e3",
                  "volume_name":"51491d30-9818-4332-b056-1f174e65d3e3",
                  "volume_size":"1",
                  "volume_type":"iscsi",
                  "volume_mountpoint":"/dev/vda",
                  "availability_zone":"nova",
                  "metadata":{
                     "readonly":"False",
                     "attached_mode":"rw"
                  }
               }
            ]
         },
         {
            "id":"e33c1eea-c533-4945-864d-0da1fc002070",
            "name":"cirros-1",
            "status":"available",
            "metadata":{
               "availability_zone":"nova",
               "config_drive":"",
               "data_transfer_time":"0",
               "object_store_transfer_time":"0",
               "root_partition_type":"Linux",
               "trilio_ordered_interfaces":"192.168.100.176",
               "vm_metadata":"{\"workload_name\": \"Workload_1\", \"workload_id\": \"18b809de-d7c8-41e2-867d-4a306407fb11\", \"trilio_ordered_interfaces\": \"192.168.100.176\", \"config_drive\": \"\"}",
               "workload_id":"18b809de-d7c8-41e2-867d-4a306407fb11",
               "workload_name":"Workload_1"
            },
            "flavor":{
               "vcpus":"1",
               "ram":"512",
               "disk":"1",
               "ephemeral":"0"
            },
            "security_group":[
               {
                  "name":"default",
                  "security_group_type":"neutron"
               }
            ],
            "nics":[
               {
                  "mac_address":"fa:16:3e:cf:4d:27",
                  "ip_address":"192.168.100.176",
                  "network":{
                     "id":"5fb7027d-a2ac-4a21-9ee1-438c281d2b26",
                     "name":"robert_internal",
                     "cidr":null,
                     "network_type":"neutron",
                     "subnet":{
                        "id":"b7b54304-aa82-4d50-91e6-66445ab56db4",
                        "name":"robert_internal",
                        "cidr":"192.168.100.0/24",
                        "ip_version":4,
                        "gateway_ip":"192.168.100.1"
                     }
                  }
               }
            ],
            "vdisks":[
               {
                  "label":null,
                  "resource_id":"c8293bb0-031a-4d33-92ee-188380211483",
                  "restore_size":1073741824,
                  "vm_id":"e33c1eea-c533-4945-864d-0da1fc002070",
                  "volume_id":"365ad75b-ca76-46cb-8eea-435535fd2e22",
                  "volume_name":"365ad75b-ca76-46cb-8eea-435535fd2e22",
                  "volume_size":"1",
                  "volume_type":"iscsi",
                  "volume_mountpoint":"/dev/vda",
                  "availability_zone":"nova",
                  "metadata":{
                     "readonly":"False",
                     "attached_mode":"rw"
                  }
               }
            ]
         }
      ],
      "name":"API taken 2",
      "description":"API taken description 2",
      "host":"TVM1",
      "size":44171264,
      "restore_size":2147483648,
      "uploaded_size":44171264,
      "progress_percent":100,
      "progress_msg":"Snapshot of workload is complete",
      "warning_msg":null,
      "error_msg":null,
      "time_taken":428,
      "pinned":false,
      "metadata":[
         {
            "created_at":"2020-11-04T14:05:57.000000",
            "updated_at":null,
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"16fc1ce5-81b2-4c07-ac63-6c9232e0418f",
            "snapshot_id":"2e56d167-bad7-43c7-8ede-a613c3fe7844",
            "key":"backup_media_target",
            "value":"10.10.2.20:/upstream"
         },
         {
            "created_at":"2020-11-04T13:58:37.000000",
            "updated_at":null,
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"5a56bbad-9957-4fb3-9bbc-469ec571b549",
            "snapshot_id":"2e56d167-bad7-43c7-8ede-a613c3fe7844",
            "key":"cancel_requested",
            "value":"0"
         },
         {
            "created_at":"2020-11-04T14:05:29.000000",
            "updated_at":"2020-11-04T14:05:45.000000",
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"d36abef7-9663-4d88-8f2e-ef914f068fb4",
            "snapshot_id":"2e56d167-bad7-43c7-8ede-a613c3fe7844",
            "key":"data_transfer_time",
            "value":"0"
         },
         {
            "created_at":"2020-11-04T14:05:57.000000",
            "updated_at":null,
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"c75f9151-ef87-4a74-acf1-42bd2588ee64",
            "snapshot_id":"2e56d167-bad7-43c7-8ede-a613c3fe7844",
            "key":"hostnames",
            "value":"[\"cirros-1\", \"cirros-2\"]"
         },
         {
            "created_at":"2020-11-04T14:05:29.000000",
            "updated_at":"2020-11-04T14:05:45.000000",
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"02916cce-79a2-4ad9-a7f6-9d9f59aa8424",
            "snapshot_id":"2e56d167-bad7-43c7-8ede-a613c3fe7844",
            "key":"object_store_transfer_time",
            "value":"0"
         },
         {
            "created_at":"2020-11-04T14:05:57.000000",
            "updated_at":null,
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"96efad2f-a24f-4cde-8e21-9cd78f78381b",
            "snapshot_id":"2e56d167-bad7-43c7-8ede-a613c3fe7844",
            "key":"pause_at_snapshot",
            "value":"0"
         },
         {
            "created_at":"2020-11-04T14:05:57.000000",
            "updated_at":null,
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"572a0b21-a415-498f-b7fa-6144d850ef56",
            "snapshot_id":"2e56d167-bad7-43c7-8ede-a613c3fe7844",
            "key":"policy_id",
            "value":"b79aa5f3-405b-4da4-96e2-893abf7cb5fd"
         },
         {
            "created_at":"2020-11-04T14:05:57.000000",
            "updated_at":null,
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"dfd7314d-8443-4a95-8e2a-7aad35ef97ea",
            "snapshot_id":"2e56d167-bad7-43c7-8ede-a613c3fe7844",
            "key":"preferredgroup",
            "value":"[]"
         },
         {
            "created_at":"2020-11-04T14:05:57.000000",
            "updated_at":null,
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"2e17e1e4-4bb1-48a9-8f11-c4cd2cfca2a9",
            "snapshot_id":"2e56d167-bad7-43c7-8ede-a613c3fe7844",
            "key":"topology",
            "value":"\"\\\"\\\"\""
         },
         {
            "created_at":"2020-11-04T14:05:57.000000",
            "updated_at":null,
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"33762790-8743-4e20-9f50-3505a00dbe76",
            "snapshot_id":"2e56d167-bad7-43c7-8ede-a613c3fe7844",
            "key":"workload_approx_backup_size",
            "value":"6"
         }
      ],
      "restores_info":""
   }
}

Delete Snapshot

DELETE https://$(tvm_address):8780/v1/$(tenant_id)/snapshots/<snapshot_id>

Deletes a specified Snapshot

Path Parameters

Headers

HTTP/1.1 200 OK
Server: nginx/1.16.1
Date: Wed, 04 Nov 2020 14:18:36 GMT
Content-Type: application/json
Content-Length: 56
Connection: keep-alive
X-Compute-Request-Id: req-82ffb2b6-b28e-4c73-89a4-310890960dbc

{"task": {"id": "a73de236-6379-424a-abc7-33d553e050b7"}}

Cancel Snapshot

GET https://$(tvm_address):8780/v1/$(tenant_id)/snapshots/<snapshot_id>/cancel

Cancels the Snapshot process of a given Snapshot

Path Parameters

Headers

HTTP/1.1 200 OK
Server: nginx/1.16.1
Date: Wed, 04 Nov 2020 14:26:44 GMT
Content-Type: application/json
Content-Length: 0
Connection: keep-alive
X-Compute-Request-Id: req-47a5a426-c241-429e-9d69-d40aed0dd68d

Restores

List Restores

GET https://$(tvm_address):8780/v1/$(tenant_id)/restores/detail

Lists Restores with details

Path Parameters

Name

Type

Description

Query Parameters

Name

Type

Description

Headers

Name

Type

Description

HTTP/1.1 200 OK
Server: nginx/1.16.1
Date: Thu, 05 Nov 2020 11:28:43 GMT
Content-Type: application/json
Content-Length: 4308
Connection: keep-alive
X-Compute-Request-Id: req-0bc531b6-be6e-43b4-90bd-39ef26ef1463

{
   "restores":[
      {
         "id":"29fdc1f8-1d53-4a10-bb45-e539a64cdbfc",
         "created_at":"2020-11-05T10:17:40.000000",
         "updated_at":"2020-11-05T10:17:40.000000",
         "finished_at":"2020-11-05T10:27:20.000000",
         "user_id":"ccddc7e7a015487fa02920f4d4979779",
         "project_id":"c76b3355a164498aa95ddbc960adc238",
         "status":"available",
         "restore_type":"restore",
         "snapshot_id":"2e56d167-bad7-43c7-8ede-a613c3fe7844",
         "links":[
            {
               "rel":"self",
               "href":"http://wlm_backend/v1/c76b3355a164498aa95ddbc960adc238/restores/29fdc1f8-1d53-4a10-bb45-e539a64cdbfc"
            },
            {
               "rel":"bookmark",
               "href":"http://wlm_backend/c76b3355a164498aa95ddbc960adc238/restores/29fdc1f8-1d53-4a10-bb45-e539a64cdbfc"
            }
         ],
         "name":"OneClick Restore",
         "description":"-",
         "host":"TVM2",
         "size":2147483648,
         "uploaded_size":2147483648,
         "progress_percent":100,
         "progress_msg":"Restore from snapshot is complete",
         "warning_msg":null,
         "error_msg":null,
         "time_taken":580,
         "restore_options":{
            "name":"OneClick Restore",
            "oneclickrestore":true,
            "restore_type":"oneclick",
            "openstack":{
               "instances":[
                  {
                     "name":"cirros-2",
                     "id":"67d6a100-fee6-4aa5-83a1-66b070d2eabe",
                     "availability_zone":"nova"
                  },
                  {
                     "name":"cirros-1",
                     "id":"e33c1eea-c533-4945-864d-0da1fc002070",
                     "availability_zone":"nova"
                  }
               ]
            },
            "type":"openstack",
            "description":"-"
         },
         "metadata":[
            {
               "created_at":"2020-11-05T10:27:20.000000",
               "updated_at":null,
               "deleted_at":null,
               "deleted":false,
               "version":"4.0.115",
               "id":"91ab2495-1903-4d75-982b-08a4e480835b",
               "restore_id":"29fdc1f8-1d53-4a10-bb45-e539a64cdbfc",
               "key":"data_transfer_time",
               "value":"0"
            },
            {
               "created_at":"2020-11-05T10:27:20.000000",
               "updated_at":null,
               "deleted_at":null,
               "deleted":false,
               "version":"4.0.115",
               "id":"e0e01eec-24e0-4abd-9b8c-19993a320e9f",
               "restore_id":"29fdc1f8-1d53-4a10-bb45-e539a64cdbfc",
               "key":"object_store_transfer_time",
               "value":"0"
            },
            {
               "created_at":"2020-11-05T10:27:20.000000",
               "updated_at":null,
               "deleted_at":null,
               "deleted":false,
               "version":"4.0.115",
               "id":"eb909267-ba9b-41d1-8861-a9ec22d6fd84",
               "restore_id":"29fdc1f8-1d53-4a10-bb45-e539a64cdbfc",
               "key":"restore_user_selected_value",
               "value":"Oneclick Restore"
            }
         ]
      },
      {
         "id":"4673d962-f6a5-4209-8d3e-b9f2e9115f07",
         "created_at":"2020-11-04T14:37:31.000000",
         "updated_at":"2020-11-04T14:37:31.000000",
         "finished_at":"2020-11-04T14:45:27.000000",
         "user_id":"ccddc7e7a015487fa02920f4d4979779",
         "project_id":"c76b3355a164498aa95ddbc960adc238",
         "status":"error",
         "restore_type":"restore",
         "snapshot_id":"2e56d167-bad7-43c7-8ede-a613c3fe7844",
         "links":[
            {
               "rel":"self",
               "href":"http://wlm_backend/v1/c76b3355a164498aa95ddbc960adc238/restores/4673d962-f6a5-4209-8d3e-b9f2e9115f07"
            },
            {
               "rel":"bookmark",
               "href":"http://wlm_backend/c76b3355a164498aa95ddbc960adc238/restores/4673d962-f6a5-4209-8d3e-b9f2e9115f07"
            }
         ],
         "name":"OneClick Restore",
         "description":"-",
         "host":"TVM2",
         "size":2147483648,
         "uploaded_size":2147483648,
         "progress_percent":100,
         "progress_msg":"",
         "warning_msg":null,
         "error_msg":"Failed restoring snapshot: Error creating instance e271bd6e-f53e-4ebc-875a-5787cc4dddf7",
         "time_taken":476,
         "restore_options":{
            "name":"OneClick Restore",
            "oneclickrestore":true,
            "restore_type":"oneclick",
            "openstack":{
               "instances":[
                  {
                     "name":"cirros-2",
                     "id":"67d6a100-fee6-4aa5-83a1-66b070d2eabe",
                     "availability_zone":"nova"
                  },
                  {
                     "name":"cirros-1",
                     "id":"e33c1eea-c533-4945-864d-0da1fc002070",
                     "availability_zone":"nova"
                  }
               ]
            },
            "type":"openstack",
            "description":"-"
         },
         "metadata":[
            {
               "created_at":"2020-11-04T14:45:27.000000",
               "updated_at":null,
               "deleted_at":null,
               "deleted":false,
               "version":"4.0.115",
               "id":"be6dc7e2-1be2-476b-9338-aed986be3b55",
               "restore_id":"4673d962-f6a5-4209-8d3e-b9f2e9115f07",
               "key":"data_transfer_time",
               "value":"0"
            },
            {
               "created_at":"2020-11-04T14:45:27.000000",
               "updated_at":null,
               "deleted_at":null,
               "deleted":false,
               "version":"4.0.115",
               "id":"2e4330b7-6389-4e21-b31b-2503b5441c3e",
               "restore_id":"4673d962-f6a5-4209-8d3e-b9f2e9115f07",
               "key":"object_store_transfer_time",
               "value":"0"
            },
            {
               "created_at":"2020-11-04T14:45:27.000000",
               "updated_at":null,
               "deleted_at":null,
               "deleted":false,
               "version":"4.0.115",
               "id":"561c806b-e38a-496c-a8de-dfe96cb3e956",
               "restore_id":"4673d962-f6a5-4209-8d3e-b9f2e9115f07",
               "key":"restore_user_selected_value",
               "value":"Oneclick Restore"
            }
         ]
      }
   ]
}

Get Restore

GET https://$(tvm_address):8780/v1/$(tenant_id)/restores/<restore_id>

Provides all details about the specified Restore

Path Parameters

Headers

HTTP/1.1 200 OK
Server: nginx/1.16.1
Date: Thu, 05 Nov 2020 14:04:45 GMT
Content-Type: application/json
Content-Length: 2639
Connection: keep-alive
X-Compute-Request-Id: req-30640219-e94e-4651-9b9e-49f5574e2a7f

{
   "restore":{
      "id":"29fdc1f8-1d53-4a10-bb45-e539a64cdbfc",
      "created_at":"2020-11-05T10:17:40.000000",
      "updated_at":"2020-11-05T10:17:40.000000",
      "finished_at":"2020-11-05T10:27:20.000000",
      "user_id":"ccddc7e7a015487fa02920f4d4979779",
      "project_id":"c76b3355a164498aa95ddbc960adc238",
      "status":"available",
      "restore_type":"restore",
      "snapshot_id":"2e56d167-bad7-43c7-8ede-a613c3fe7844",
      "snapshot_details":{
         "created_at":"2020-11-04T13:58:37.000000",
         "updated_at":"2020-11-05T10:27:22.000000",
         "deleted_at":null,
         "deleted":false,
         "version":"4.0.115",
         "id":"2e56d167-bad7-43c7-8ede-a613c3fe7844",
         "user_id":"ccddc7e7a015487fa02920f4d4979779",
         "project_id":"c76b3355a164498aa95ddbc960adc238",
         "workload_id":"18b809de-d7c8-41e2-867d-4a306407fb11",
         "snapshot_type":"full",
         "display_name":"API taken 2",
         "display_description":"API taken description 2",
         "size":44171264,
         "restore_size":2147483648,
         "uploaded_size":44171264,
         "progress_percent":100,
         "progress_msg":"Creating Instance: cirros-2",
         "warning_msg":null,
         "error_msg":null,
         "host":"TVM1",
         "finished_at":"2020-11-04T14:06:03.000000",
         "data_deleted":false,
         "pinned":false,
         "time_taken":428,
         "vault_storage_id":null,
         "status":"available"
      },
      "workload_id":"18b809de-d7c8-41e2-867d-4a306407fb11",
      "instances":[
         {
            "id":"1fb104bf-7e2b-4cb6-84f6-96aabc8f1dd2",
            "name":"cirros-2",
            "status":"available",
            "metadata":{
               "config_drive":"",
               "instance_id":"67d6a100-fee6-4aa5-83a1-66b070d2eabe",
               "production":"1"
            }
         },
         {
            "id":"b083bb70-e384-4107-b951-8e9e7bbac380",
            "name":"cirros-1",
            "status":"available",
            "metadata":{
               "config_drive":"",
               "instance_id":"e33c1eea-c533-4945-864d-0da1fc002070",
               "production":"1"
            }
         }
      ],
      "networks":[
         
      ],
      "subnets":[
         
      ],
      "routers":[
         
      ],
      "links":[
         {
            "rel":"self",
            "href":"http://wlm_backend/v1/c76b3355a164498aa95ddbc960adc238/restores/29fdc1f8-1d53-4a10-bb45-e539a64cdbfc"
         },
         {
            "rel":"bookmark",
            "href":"http://wlm_backend/c76b3355a164498aa95ddbc960adc238/restores/29fdc1f8-1d53-4a10-bb45-e539a64cdbfc"
         }
      ],
      "name":"OneClick Restore",
      "description":"-",
      "host":"TVM2",
      "size":2147483648,
      "uploaded_size":2147483648,
      "progress_percent":100,
      "progress_msg":"Restore from snapshot is complete",
      "warning_msg":null,
      "error_msg":null,
      "time_taken":580,
      "restore_options":{
         "name":"OneClick Restore",
         "oneclickrestore":true,
         "restore_type":"oneclick",
         "openstack":{
            "instances":[
               {
                  "name":"cirros-2",
                  "id":"67d6a100-fee6-4aa5-83a1-66b070d2eabe",
                  "availability_zone":"nova"
               },
               {
                  "name":"cirros-1",
                  "id":"e33c1eea-c533-4945-864d-0da1fc002070",
                  "availability_zone":"nova"
               }
            ]
         },
         "type":"openstack",
         "description":"-"
      },
      "metadata":[
         
      ]
   }
}

Delete Restore

DELETE https://$(tvm_address):8780/v1/$(tenant_id)/restores/<restore_id>

Deletes the specified Restore

Path Parameters

Headers

HTTP/1.1 200 OK
Server: nginx/1.16.1
Date: Thu, 05 Nov 2020 14:21:07 GMT
Content-Type: application/json
Content-Length: 0
Connection: keep-alive
X-Compute-Request-Id: req-0e155b21-8931-480a-a749-6d8764666e4d

Cancel Restore

GET https://$(tvm_address):8780/v1/$(tenant_id)/restores/<restore_id>/cancel

Cancels an ongoing Restore

Path Parameters

Headers

HTTP/1.1 200 OK
Server: nginx/1.16.1
Date: Thu, 05 Nov 2020 15:13:30 GMT
Content-Type: application/json
Content-Length: 0
Connection: keep-alive
X-Compute-Request-Id: req-98d4853c-314c-4f27-bd3f-f81bda1a2840

One Click Restore

POST https://$(tvm_address):8780/v1/$(tenant_id)/snapshots/<snapshot_id>

Starts a restore according to the provided information

Path Parameters

Headers

HTTP/1.1 202 Accepted
Server: nginx/1.16.1
Date: Thu, 05 Nov 2020 14:30:56 GMT
Content-Type: application/json
Content-Length: 992
Connection: keep-alive
X-Compute-Request-Id: req-7e18c309-19e5-49cb-a07e-90dd368fddae

{
   "restore":{
      "id":"3df1d432-2f76-4ebd-8f89-1275428842ff",
      "created_at":"2020-11-05T14:30:56.048656",
      "updated_at":"2020-11-05T14:30:56.048656",
      "finished_at":null,
      "user_id":"ccddc7e7a015487fa02920f4d4979779",
      "project_id":"c76b3355a164498aa95ddbc960adc238",
      "status":"restoring",
      "restore_type":"restore",
      "snapshot_id":"2e56d167-bad7-43c7-8ede-a613c3fe7844",
      "links":[
         {
            "rel":"self",
            "href":"http://wlm_backend/v1/c76b3355a164498aa95ddbc960adc238/restores/3df1d432-2f76-4ebd-8f89-1275428842ff"
         },
         {
            "rel":"bookmark",
            "href":"http://wlm_backend/c76b3355a164498aa95ddbc960adc238/restores/3df1d432-2f76-4ebd-8f89-1275428842ff"
         }
      ],
      "name":"One Click Restore",
      "description":"One Click Restore",
      "host":"",
      "size":0,
      "uploaded_size":0,
      "progress_percent":0,
      "progress_msg":null,
      "warning_msg":null,
      "error_msg":null,
      "time_taken":0,
      "restore_options":{
         "openstack":{
            
         },
         "type":"openstack",
         "oneclickrestore":true,
         "vmware":{
            
         },
         "restore_type":"oneclick"
      },
      "metadata":[
         
      ]
   }
}

Body Format

The One-Click restore requires a body to provide all necessary information in json format.

{
   "restore":{
      "options":{
         "openstack":{
            
         },
         "type":"openstack",
         "oneclickrestore":true,
         "vmware":{},
         "restore_type":"oneclick"
      },
      "name":"One Click Restore",
      "description":"One Click Restore"
   }
}

Selective Restore

POST https://$(tvm_address):8780/v1/$(tenant_id)/snapshots/<snapshot_id>

Starts a restore according to the provided information.

Path Parameters

Headers

HTTP/1.1 202 Accepted
Server: nginx/1.16.1
Date: Mon, 09 Nov 2020 09:53:31 GMT
Content-Type: application/json
Content-Length: 1713
Connection: keep-alive
X-Compute-Request-Id: req-84f00d6f-1b12-47ec-b556-7b3ed4c2f1d7

{
   "restore":{
      "id":"778baae0-6c64-4eb1-8fa3-29324215c43c",
      "created_at":"2020-11-09T09:53:31.037588",
      "updated_at":"2020-11-09T09:53:31.037588",
      "finished_at":null,
      "user_id":"ccddc7e7a015487fa02920f4d4979779",
      "project_id":"c76b3355a164498aa95ddbc960adc238",
      "status":"restoring",
      "restore_type":"restore",
      "snapshot_id":"2e56d167-bad7-43c7-8ede-a613c3fe7844",
      "links":[
         {
            "rel":"self",
            "href":"http://wlm_backend/v1/c76b3355a164498aa95ddbc960adc238/restores/778baae0-6c64-4eb1-8fa3-29324215c43c"
         },
         {
            "rel":"bookmark",
            "href":"http://wlm_backend/c76b3355a164498aa95ddbc960adc238/restores/778baae0-6c64-4eb1-8fa3-29324215c43c"
         }
      ],
      "name":"API",
      "description":"API Created",
      "host":"",
      "size":0,
      "uploaded_size":0,
      "progress_percent":0,
      "progress_msg":null,
      "warning_msg":null,
      "error_msg":null,
      "time_taken":0,
      "restore_options":{
         "openstack":{
            "instances":[
               {
                  "vdisks":[
                     {
                        "new_volume_type":"iscsi",
                        "id":"365ad75b-ca76-46cb-8eea-435535fd2e22",
                        "availability_zone":"nova"
                     }
                  ],
                  "name":"cirros-1-selective",
                  "availability_zone":"nova",
                  "nics":[
                     
                  ],
                  "flavor":{
                     "vcpus":1,
                     "disk":1,
                     "swap":"",
                     "ram":512,
                     "ephemeral":0,
                     "id":"1"
                  },
                  "include":true,
                  "id":"e33c1eea-c533-4945-864d-0da1fc002070"
               },
               {
                  "include":false,
                  "id":"67d6a100-fee6-4aa5-83a1-66b070d2eabe"
               }
            ],
            "restore_topology":false,
            "networks_mapping":{
               "networks":[
                  {
                     "snapshot_network":{
                        "subnet":{
                           "id":"b7b54304-aa82-4d50-91e6-66445ab56db4"
                        },
                        "id":"5fb7027d-a2ac-4a21-9ee1-438c281d2b26"
                     },
                     "target_network":{
                        "subnet":{
                           "id":"b7b54304-aa82-4d50-91e6-66445ab56db4"
                        },
                        "id":"5fb7027d-a2ac-4a21-9ee1-438c281d2b26",
                        "name":"internal"
                     }
                  }
               ]
            }
         },
         "restore_type":"selective",
         "type":"openstack",
         "oneclickrestore":false
      },
      "metadata":[
         
      ]
   }
}

Body Format

The One-Click restore requires a body to provide all necessary information in json format.

{
   "restore":{
    "name":"<restore name>",
    "description":"<restore description>",
	  "options":{
         "openstack":{
            "instances":[
               {
                  "name":"<new name of instance>",
                  "include":<true/false>,
                  "id":"<original id of instance to be restored>"
				  "availability_zone":"<availability zone>",
				  "vdisks":[
                     {
                        "id":"<original ID of Volume>",
                        "new_volume_type":"<new volume type>",
                        "availability_zone":"<Volume availability zone>"
                     }
                  ],
                  "nics":[
                     {
                         'mac_address':'<mac address of the pre-created port>',
                         'ip_address':'<IP of the pre-created port>',
                         'id':'<ID of the pre-created port>',
                         'network':{
                            'subnet':{
                               'id':'<ID of the subnet of the pre-created port>'
                            },
                         'id':'<ID of the network of the pre-created port>'
                      }
                  ],
                  "flavor":{
                     "vcpus":<Integer>,
                     "disk":<Integer>,
                     "swap":<Integer>,
                     "ram":<Integer>,
                     "ephemeral":<Integer>,
                     "id":<Integer>
                  }
               }
            ],
            "restore_topology":<true/false>,
            "networks_mapping":{
               "networks":[
                  {
                     "snapshot_network":{
                        "subnet":{
                           "id":"<ID of the original Subnet ID>"
                        },
                        "id":"<ID of the original Network ID>"
                     },
                     "target_network":{
                        "subnet":{
                           "id":"<ID of the target Subnet ID>"
                        },
                        "id":"<ID of the target Network ID>",
                        "name":"<name of the target network>"
                     }
                  }
               ]
            }
         },
         "restore_type":"selective",
         "type":"openstack",
         "oneclickrestore":false
      }
   }
}

Inplace Restore

POST https://$(tvm_address):8780/v1/$(tenant_id)/snapshots/<snapshot_id>

Starts a restore according to the provided information

Path Parameters

Headers

HTTP/1.1 202 Accepted
Server: nginx/1.16.1
Date: Mon, 09 Nov 2020 12:53:03 GMT
Content-Type: application/json
Content-Length: 1341
Connection: keep-alive
X-Compute-Request-Id: req-311fa97e-0fd7-41ed-873b-482c149ee743

{
   "restore":{
      "id":"0bf96f46-b27b-425c-a10f-a861cc18b82a",
      "created_at":"2020-11-09T12:53:02.726757",
      "updated_at":"2020-11-09T12:53:02.726757",
      "finished_at":null,
      "user_id":"ccddc7e7a015487fa02920f4d4979779",
      "project_id":"c76b3355a164498aa95ddbc960adc238",
      "status":"restoring",
      "restore_type":"restore",
      "snapshot_id":"ed4f29e8-7544-4e1c-af8a-a76031211926",
      "links":[
         {
            "rel":"self",
            "href":"http://wlm_backend/v1/c76b3355a164498aa95ddbc960adc238/restores/0bf96f46-b27b-425c-a10f-a861cc18b82a"
         },
         {
            "rel":"bookmark",
            "href":"http://wlm_backend/c76b3355a164498aa95ddbc960adc238/restores/0bf96f46-b27b-425c-a10f-a861cc18b82a"
         }
      ],
      "name":"API",
      "description":"API description",
      "host":"",
      "size":0,
      "uploaded_size":0,
      "progress_percent":0,
      "progress_msg":null,
      "warning_msg":null,
      "error_msg":null,
      "time_taken":0,
      "restore_options":{
         "restore_type":"inplace",
         "type":"openstack",
         "oneclickrestore":false,
         "openstack":{
            "instances":[
               {
                  "restore_boot_disk":true,
                  "include":true,
                  "id":"7c1bb5d2-aa5a-44f7-abcd-2d76b819b4c8",
                  "vdisks":[
                     {
                        "restore_cinder_volume":true,
                        "id":"f6b3fef6-4b0e-487e-84b5-47a14da716ca"
                     }
                  ]
               },
               {
                  "restore_boot_disk":true,
                  "include":true,
                  "id":"08dab61c-6efd-44d3-a9ed-8e789d338c1b",
                  "vdisks":[
                     {
                        "restore_cinder_volume":true,
                        "id":"53204f34-019d-4ba8-ada1-e6ab7b8e5b43"
                     }
                  ]
               }
            ]
         }
      },
      "metadata":[
         
      ]
   }
}

Body Format

The One-Click restore requires a body to provide all necessary information in json format.

{
   "restore":{
      "name":"<restore-name>",
      "description":"<restore-description>",
      "options":{
         "restore_type":"inplace",
         "type":"openstack",
         "oneclickrestore":false,
         "openstack":{
            "instances":[
               {
                  "restore_boot_disk":<Boolean>,
                  "include":<Boolean>,
                  "id":"<ID of the instance the volumes are attached to>",
                  "vdisks":[
                     {
                        "restore_cinder_volume":<boolean>,
                        "id":"<ID of the Volume to restore>"
                     }
                  ]
               }
            ]
         }
      }
   }
}

Workload Policies

List Policies

GET https://$(tvm_address):8780/v1/$(tenant_id)/workload_policy

Requests the list of available Workload Policies

Path Parameters

Name

Type

Description

Headers

Name

Type

Description

HTTP/1.1 200 OK
Server: nginx/1.16.1
Date: Fri, 13 Nov 2020 13:56:08 GMT
Content-Type: application/json
Content-Length: 1399
Connection: keep-alive
X-Compute-Request-Id: req-4618161e-64e4-489a-b8fc-f3cb21d94096

{
   "policy_list":[
      {
         "id":"b79aa5f3-405b-4da4-96e2-893abf7cb5fd",
         "created_at":"2020-10-26T12:52:22.000000",
         "updated_at":"2020-10-26T12:52:22.000000",
         "status":"available",
         "name":"Gold",
         "description":"",
         "metadata":[
            
         ],
         "field_values":[
            {
               "created_at":"2020-10-26T12:52:22.000000",
               "updated_at":null,
               "deleted_at":null,
               "deleted":false,
               "version":"4.0.115",
               "id":"0201f8b4-482d-4ec1-9b92-8cf3092abcc2",
               "policy_id":"b79aa5f3-405b-4da4-96e2-893abf7cb5fd",
               "policy_field_name":"retention_policy_value",
               "value":"10"
            },
            {
               "created_at":"2020-10-26T12:52:22.000000",
               "updated_at":null,
               "deleted_at":null,
               "deleted":false,
               "version":"4.0.115",
               "id":"48cc7007-e221-44de-bd4e-6a66841bdee0",
               "policy_id":"b79aa5f3-405b-4da4-96e2-893abf7cb5fd",
               "policy_field_name":"interval",
               "value":"5"
            },
            {
               "created_at":"2020-10-26T12:52:22.000000",
               "updated_at":null,
               "deleted_at":null,
               "deleted":false,
               "version":"4.0.115",
               "id":"79070c67-9021-4220-8a79-648ffeebc144",
               "policy_id":"b79aa5f3-405b-4da4-96e2-893abf7cb5fd",
               "policy_field_name":"retention_policy_type",
               "value":"Number of Snapshots to Keep"
            },
            {
               "created_at":"2020-10-26T12:52:22.000000",
               "updated_at":null,
               "deleted_at":null,
               "deleted":false,
               "version":"4.0.115",
               "id":"9fec205a-9528-45ea-a118-ffb64d8c7d9d",
               "policy_id":"b79aa5f3-405b-4da4-96e2-893abf7cb5fd",
               "policy_field_name":"fullbackup_interval",
               "value":"-1"
            }
         ]
      }
   ]
}

Show Policy

GET https://$(tvm_address):8780/v1/$(tenant_id)/workload_policy/<policy_id>

Requests the details of a given policy

Path Parameters

Headers

HTTP/1.1 200 OK
Server: nginx/1.16.1
Date: Fri, 13 Nov 2020 14:18:42 GMT
Content-Type: application/json
Content-Length: 2160
Connection: keep-alive
X-Compute-Request-Id: req-0583fc35-0f80-4746-b280-c17b32cc4b25

{
   "policy":{
      "id":"b79aa5f3-405b-4da4-96e2-893abf7cb5fd",
      "created_at":"2020-10-26T12:52:22.000000",
      "updated_at":"2020-10-26T12:52:22.000000",
      "user_id":"adfa32d7746a4341b27377d6f7c61adb",
      "project_id":"4dfe98a43bfa404785a812020066b4d6",
      "status":"available",
      "name":"Gold",
      "description":"",
      "field_values":[
         {
            "created_at":"2020-10-26T12:52:22.000000",
            "updated_at":null,
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"0201f8b4-482d-4ec1-9b92-8cf3092abcc2",
            "policy_id":"b79aa5f3-405b-4da4-96e2-893abf7cb5fd",
            "policy_field_name":"retention_policy_value",
            "value":"10"
         },
         {
            "created_at":"2020-10-26T12:52:22.000000",
            "updated_at":null,
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"48cc7007-e221-44de-bd4e-6a66841bdee0",
            "policy_id":"b79aa5f3-405b-4da4-96e2-893abf7cb5fd",
            "policy_field_name":"interval",
            "value":"5"
         },
         {
            "created_at":"2020-10-26T12:52:22.000000",
            "updated_at":null,
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"79070c67-9021-4220-8a79-648ffeebc144",
            "policy_id":"b79aa5f3-405b-4da4-96e2-893abf7cb5fd",
            "policy_field_name":"retention_policy_type",
            "value":"Number of Snapshots to Keep"
         },
         {
            "created_at":"2020-10-26T12:52:22.000000",
            "updated_at":null,
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"9fec205a-9528-45ea-a118-ffb64d8c7d9d",
            "policy_id":"b79aa5f3-405b-4da4-96e2-893abf7cb5fd",
            "policy_field_name":"fullbackup_interval",
            "value":"-1"
         }
      ],
      "metadata":[
         
      ],
      "policy_assignments":[
         {
            "created_at":"2020-10-26T12:53:01.000000",
            "updated_at":null,
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"3e3f1b12-1b1f-452b-a9d2-b6e5fbf2ab18",
            "policy_id":"b79aa5f3-405b-4da4-96e2-893abf7cb5fd",
            "project_id":"4dfe98a43bfa404785a812020066b4d6",
            "policy_name":"Gold",
            "project_name":"admin"
         },
         {
            "created_at":"2020-10-29T15:39:13.000000",
            "updated_at":null,
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"8b4a6236-63f1-4e2d-b8d1-23b37f4b4346",
            "policy_id":"b79aa5f3-405b-4da4-96e2-893abf7cb5fd",
            "project_id":"c76b3355a164498aa95ddbc960adc238",
            "policy_name":"Gold",
            "project_name":"robert"
         }
      ]
   }
}

list assigned Policies

GET https://$(tvm_address):8780/v1/$(tenant_id)/workload_policy/assigned/<project_id>

Requests the lists of Policies assigned to a Project.

Path Parameters

Headers

HTTP/1.1 200 OK
Server: nginx/1.16.1
Date: Tue, 17 Nov 2020 09:14:01 GMT
Content-Type: application/json
Content-Length: 338
Connection: keep-alive
X-Compute-Request-Id: req-57175488-d267-4dcb-90b5-f239d8b02fe2

{
   "policies":[
      {
         "created_at":"2020-10-29T15:39:13.000000",
         "updated_at":null,
         "deleted_at":null,
         "deleted":false,
         "version":"4.0.115",
         "id":"8b4a6236-63f1-4e2d-b8d1-23b37f4b4346",
         "policy_id":"b79aa5f3-405b-4da4-96e2-893abf7cb5fd",
         "project_id":"c76b3355a164498aa95ddbc960adc238",
         "policy_name":"Gold",
         "project_name":"robert"
      }
   ]
}

Create Policy

POST https://$(tvm_address):8780/v1/$(tenant_id)/workload_policy

Creates a Policy with the given parameters

Path Parameters

Headers

HTTP/1.1 200 OK
Server: nginx/1.16.1
Date: Tue, 17 Nov 2020 09:24:03 GMT
Content-Type: application/json
Content-Length: 1413
Connection: keep-alive
X-Compute-Request-Id: req-05e05333-b967-4d4e-9c9b-561f1a7add5a

{
   "policy":{
      "id":"23176f20-9e9d-4fc3-9d3d-f10d2b184163",
      "created_at":"2020-11-17T09:24:01.000000",
      "updated_at":"2020-11-17T09:24:01.000000",
      "status":"available",
      "name":"CLI created",
      "description":"CLI created",
      "metadata":[
         
      ],
      "field_values":[
         {
            "created_at":"2020-11-17T09:24:01.000000",
            "updated_at":null,
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"767ae42d-caf0-4d36-963c-9b0e50991711",
            "policy_id":"23176f20-9e9d-4fc3-9d3d-f10d2b184163",
            "policy_field_name":"interval",
            "value":"4 hr"
         },
         {
            "created_at":"2020-11-17T09:24:01.000000",
            "updated_at":null,
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"7e34ce5c-3de0-408e-8294-cc091bee281f",
            "policy_id":"23176f20-9e9d-4fc3-9d3d-f10d2b184163",
            "policy_field_name":"retention_policy_value",
            "value":"10"
         },
         {
            "created_at":"2020-11-17T09:24:01.000000",
            "updated_at":null,
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"95537f7c-e59a-4365-b1e9-7fa2ed49c677",
            "policy_id":"23176f20-9e9d-4fc3-9d3d-f10d2b184163",
            "policy_field_name":"retention_policy_type",
            "value":"Number of Snapshots to Keep"
         },
         {
            "created_at":"2020-11-17T09:24:01.000000",
            "updated_at":null,
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"f635bece-be61-4e72-bce4-bc72a6f549e3",
            "policy_id":"23176f20-9e9d-4fc3-9d3d-f10d2b184163",
            "policy_field_name":"fullbackup_interval",
            "value":"-1"
         }
      ]
   }
}

Body Format

{
   "workload_policy":{
      "field_values":{
         "fullbackup_interval":"<-1 for never / 0 for always / Integer>",
         "retention_policy_type":"<Number of Snapshots to Keep/Number of days to retain Snapshots>",
         "interval":"<Integer hr>",
         "retention_policy_value":"<Integer>"
      },
      "display_name":"<String>",
      "display_description":"<String>",
      "metadata":{
         <key>:<value>
      }
   }
}

Update Policy

PUT https://$(tvm_address):8780/v1/$(tenant_id)/workload_policy/<policy-id>

Updates a Policy with the given information

Path Parameters

Headers

HTTP/1.1 200 OK
Server: nginx/1.16.1
Date: Tue, 17 Nov 2020 09:32:13 GMT
Content-Type: application/json
Content-Length: 1515
Connection: keep-alive
X-Compute-Request-Id: req-9104cf1c-4025-48f5-be92-1a6b7117bf95

{
   "policy":{
      "id":"23176f20-9e9d-4fc3-9d3d-f10d2b184163",
      "created_at":"2020-11-17T09:24:01.000000",
      "updated_at":"2020-11-17T09:24:01.000000",
      "status":"available",
      "name":"API created",
      "description":"API created",
      "metadata":[
         
      ],
      "field_values":[
         {
            "created_at":"2020-11-17T09:24:01.000000",
            "updated_at":"2020-11-17T09:31:45.000000",
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"767ae42d-caf0-4d36-963c-9b0e50991711",
            "policy_id":"23176f20-9e9d-4fc3-9d3d-f10d2b184163",
            "policy_field_name":"interval",
            "value":"8 hr"
         },
         {
            "created_at":"2020-11-17T09:24:01.000000",
            "updated_at":"2020-11-17T09:31:45.000000",
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"7e34ce5c-3de0-408e-8294-cc091bee281f",
            "policy_id":"23176f20-9e9d-4fc3-9d3d-f10d2b184163",
            "policy_field_name":"retention_policy_value",
            "value":"20"
         },
         {
            "created_at":"2020-11-17T09:24:01.000000",
            "updated_at":"2020-11-17T09:31:45.000000",
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"95537f7c-e59a-4365-b1e9-7fa2ed49c677",
            "policy_id":"23176f20-9e9d-4fc3-9d3d-f10d2b184163",
            "policy_field_name":"retention_policy_type",
            "value":"Number of days to retain Snapshots"
         },
         {
            "created_at":"2020-11-17T09:24:01.000000",
            "updated_at":"2020-11-17T09:31:45.000000",
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"f635bece-be61-4e72-bce4-bc72a6f549e3",
            "policy_id":"23176f20-9e9d-4fc3-9d3d-f10d2b184163",
            "policy_field_name":"fullbackup_interval",
            "value":"7"
         }
      ]
   }
}

Body Format

{
   "policy":{
      "field_values":{
         "fullbackup_interval":"<-1 for never / 0 for always / Integer>",
         "retention_policy_type":"<Number of Snapshots to Keep/Number of days to retain Snapshots>",
         "interval":"<Integer hr>",
         "retention_policy_value":"<Integer>"
      },
      "display_name":"String",
      "display_description":"String",
      "metadata":{
         <key>:<value>
      }
   }
}

Assign Policy

POST https://$(tvm_address):8780/v1/$(tenant_id)/workload_policy/<policy-id>

Updates a Policy with the given information

Path Parameters

Headers

HTTP/1.1 200 OK
Server: nginx/1.16.1
Date: Tue, 17 Nov 2020 09:46:23 GMT
Content-Type: application/json
Content-Length: 2318
Connection: keep-alive
X-Compute-Request-Id: req-169a53e4-b1c9-4bd1-bf68-3416d177d868

{
   "policy":{
      "id":"23176f20-9e9d-4fc3-9d3d-f10d2b184163",
      "created_at":"2020-11-17T09:24:01.000000",
      "updated_at":"2020-11-17T09:24:01.000000",
      "user_id":"adfa32d7746a4341b27377d6f7c61adb",
      "project_id":"4dfe98a43bfa404785a812020066b4d6",
      "status":"available",
      "name":"API created",
      "description":"API created",
      "field_values":[
         {
            "created_at":"2020-11-17T09:24:01.000000",
            "updated_at":"2020-11-17T09:31:45.000000",
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"767ae42d-caf0-4d36-963c-9b0e50991711",
            "policy_id":"23176f20-9e9d-4fc3-9d3d-f10d2b184163",
            "policy_field_name":"interval",
            "value":"8 hr"
         },
         {
            "created_at":"2020-11-17T09:24:01.000000",
            "updated_at":"2020-11-17T09:31:45.000000",
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"7e34ce5c-3de0-408e-8294-cc091bee281f",
            "policy_id":"23176f20-9e9d-4fc3-9d3d-f10d2b184163",
            "policy_field_name":"retention_policy_value",
            "value":"20"
         },
         {
            "created_at":"2020-11-17T09:24:01.000000",
            "updated_at":"2020-11-17T09:31:45.000000",
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"95537f7c-e59a-4365-b1e9-7fa2ed49c677",
            "policy_id":"23176f20-9e9d-4fc3-9d3d-f10d2b184163",
            "policy_field_name":"retention_policy_type",
            "value":"Number of days to retain Snapshots"
         },
         {
            "created_at":"2020-11-17T09:24:01.000000",
            "updated_at":"2020-11-17T09:31:45.000000",
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"f635bece-be61-4e72-bce4-bc72a6f549e3",
            "policy_id":"23176f20-9e9d-4fc3-9d3d-f10d2b184163",
            "policy_field_name":"fullbackup_interval",
            "value":"7"
         }
      ],
      "metadata":[
         
      ],
      "policy_assignments":[
         {
            "created_at":"2020-11-17T09:46:22.000000",
            "updated_at":null,
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"4794ed95-d8d1-4572-93e8-cebd6d4df48f",
            "policy_id":"23176f20-9e9d-4fc3-9d3d-f10d2b184163",
            "project_id":"cbad43105e404c86a1cd07c48a737f9c",
            "policy_name":"API created",
            "project_name":"services"
         },
         {
            "created_at":"2020-11-17T09:46:22.000000",
            "updated_at":null,
            "deleted_at":null,
            "deleted":false,
            "version":"4.0.115",
            "id":"68f187a6-3526-4a35-8b2d-cb0e9f497dd8",
            "policy_id":"23176f20-9e9d-4fc3-9d3d-f10d2b184163",
            "project_id":"c76b3355a164498aa95ddbc960adc238",
            "policy_name":"API created",
            "project_name":"robert"
         }
      ]
   },
   "failed_ids":[
      
   ]
}

Body Format

{
   "policy":{
      "remove_projects":[
         "<project_id>"
      ],
      "add_projects":[
         "<project_id>",
      ]
   }
}

Delete Policy

DELETE https://$(tvm_address):8780/v1/$(tenant_id)/workload_policy/<policy_id>

Deletes a given Policy

Path Parameters

Headers

HTTP/1.1 202 Accepted
Server: nginx/1.16.1
Date: Tue, 17 Nov 2020 09:56:03 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Connection: keep-alive