Trilio, by TrilioData, is a native OpenStack service that provides policy-based comprehensive backup and recovery for OpenStack workloads. The solution captures point-in-time workloads (Application, OS, Compute, Network, Configurations, Data and Metadata of an environment) as full or incremental snapshots. These snapshots can be held in a variety of storage environments including NFS AWS S3 compatible storage. With Trilio and its single click recovery, organizations can improve Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO). With Trilio, IT departments are enabled to fully deploy OpenStack solutions and provide business assurance through enhanced data retention, protection and integrity.

With the use of Trilio’s VAST (Virtual Snapshot Technology), Enterprise IT and Cloud Service Providers can now deploy backup and disaster recovery as a service to prevent data loss or data corruption through point-in-time snapshots and seamless one-click recovery. Trilio takes point-in-time backup of the entire workload consisting of compute resources, network configurations and storage data as one unit. It also takes incremental backups that only captures the changes that were made since last backup. Incremental snapshots save time and storage space as the backup only includes changes since the last backup. The benefits of using VAST for backup and restore could be summarized as below:

1. Efficient capture and storage of snapshots. Since our full backups only include data that is committed to storage volume and the incremental backups only include changed blocks of data since last backup, our backup processes are efficient and storages backup images efficiently on the backup media

2. Faster and reliable recovery. When your applications become complex that snap multiple VMs and storage volumes, our efficient recovery process will bring your application from zero to operational with just click of button

3. Easy migration of workloads between clouds. Trilio captures all the details of your application and hence our migration includes your entire application stack without leaving any thing for guess work.

4. Through policy and automation lower the Total Cost of Ownership. Our tenant driven backup process and automation eliminates the need for dedicated backup administrators, there by improves your total cost of ownership.

Once the Trilio VM or the Cluster of Trilio VMs has been spun, the actual installation process can begin. This process contains of the following steps:

1. Install the Trilio dm-api service on the control plane

2. Install the Trilio datamover service on the compute plane

3. Install the Trilio Horizon plugin into the Horizon service

How these steps look in detail is depending on the Openstack distribution Trilio is installed in. Each supported Openstack distribution has its own deployment tools. Trilio is integrating into these deployment tools to provide a native integration from the beginning to the end.

• Test text

It is recommended to think about the following elements prior to the installation of Trilio for Openstack.

Tenant Quotas

Trilio uses Cinder snapshots for calculating full and incremental backups. For full backups, Trilio creates Cinder snapshots for all the volumes in the backup job. It then leaves these Cinder snapshots behind for calculating the incremental backup image during next backup. During an incremental backup operation it creates new Cinder snapshots, calculates the changed blocks between the new snapshots and the old snapshots that were left behind during full/previous backups. It then deletes the old snapshots but leaves the newly created snapshots behind. So, it is important that each tenant who is availing Trilio backup functionality has sufficient Cinder snapshot quotas to accommodate these additional snapshots. The guideline is to add 2 snapshots for every volume that is added to backups to volume snapshot quotas for that tenant. You may also increase the volume quotas for the tenant by the same amount because Trilio briefly creates a volume from snapshot to read data from the snapshot for backup purposes. During a restore process, Trilio creates additional instances and Cinder volumes. To accommodate restore operations, a tenant should have sufficient quota for Nova instances and Cinder volumes. Otherwise restore operations will result in failures.

AWS S3 eventual consistency¶

AWS S3 object consistency model includes:

1. Read-after-write

2. Read-after-update

3. Read-after-delete

Each of them describes how an object will reach its consistent state after an object is created/updated or deleted. None of them provides strong consistency and there is a lag time for an object to reach the consistent state. Though Trilio employed mechanisms to work around the limitations of eventual consistency of AWS S3, when an object reach its consistency state is not deterministic. There is no official statement from AWS on how long it takes for an object to reach consistent state. However read-after-write has a shorter time to reach consistency compared to other IO patterns. Our solution is designed to maximize read-after-write IO pattern. The time in which an object reaches eventual consistency also depends on the AWS region. For example, aws-standard region does not have strong consistency model compared to us-east or us-west. We suggest to use these regions when creating s3 buckets for Trilio. Though read-after-update IO pattern is hard to avoid completely, we employed ample delays in accessing objects to accommodate larger durations for objects to get into consistent state. However in rare occasions, backups may still fail and need to restarted.

Trilio Cluster¶

Trilio can be deployed as a single node or a three node cluster. It is highly recommended that Trilio is deployed as three node cluster for fault tolerance and load balancing. Starting with 3.0 release, Trilio requires additional IP for cluster and is required for both single node and three node deployments. Cluster ip a.k.a virtual ip is used for managing cluster and is used to register Trilio service endpoint in the keystone sevice catalog.

Ceph is the most common OpenSource solution to provide block storage through OpenStack Cinder.

Ceph is a very flexible solution. The possibilities of Ceph require additional steps to the Trilio solution.

The uninstallation of Trilio is depending on the Openstack Distribution it is installed in. The high-level process is the same for all Distributions.

1. Uninstall the Horizon Plugin or the Trilio Horizon container

2. Uninstall the datamover-api container

3. Uninstall the datamover

4. Delete the Trilio Appliance Cluster

The Trilio Appliance can be reinitialized, which will delete all workload related values from the Trilio database.

To reinitialize the Trilio Appliance do:

• Login into the Trilio Dashboard

• Click on "admin" in the upper right corner to open the submenu

• Choose "Reinitialize"

• Verify that you want to reinitialize the Trilio

Download Trilio logs

It is possible to download the Trilio logs directly through the Trilio web gui.

To download logs throught the Trilio web gui:

• Login into the Trilio web gui

• Go to "Logs"

• Choose the log to be downloaded

  • Each log for every Trilio Appliance can be downloaded seperatly

  • or a zip of all logfiles can be created and downloaded

Backup-as-a-Service

Trilio is an add on service to OpenStack cloud infrastructure and provides backup and disaster recovery functions for tenant workloads. Trilio is very similar to other OpenStack services including nova, cinder, glance, etc and adheres to all tenets of OpenStack. It is a stateless service that scales with your cloud.

Main Components

Trilio has four main software components:

1. Trilio ships as a QCOW2 image. User can instantiate one or more VMs from the QCOW2 image on a standalone KVM boxes.

2. Trilio API is a python module that is installed on all OpenStack controller nodes where the nova-api service is running.

3. Trilio Datamover is a python module that is installed on every OpenStack compute nodes

4. Trilio horizon plugin is installed as an add on to horizon servers. This module is installed on every server that runs horizon service.

Service Endpoints

Trilio is both a provider and consumer into OpenStack ecosystem. It uses other OpenStack services such as nova, cinder, glance, neutron, and keystone and provides its own service to OpenStack tenants. To accomodate all possible OpenStack deployments, Trilio can be configured to use either public or internal URLs of services. Likewise Trilio provides its own public, internal and admin URLs.

Network Topology

This figure represents a typical network topology. Trilio exposes its public URL endpoint on public network and Trilio virtual appliances and data movers typically use either internal network or dedicated backup network for storing and retrieving backup images from backup store.

Trilio has four main software components:

1. Trilio ships as a QCOW2 image. User can instantiate one or more VMs from the QCOW2 image on a standalone KVM boxes.

2. Trilio API is a python module that is an extension to nova api service. This module is installed on all OpenStack controller nodes

3. Trilio Datamover is a python module that is installed on every OpenStack compute nodes

4. Trilio horizon plugin is installed as an add on to horizon servers. This module is installed on every server that runs horizon service.

System requirements Trilio Appliance

The Trilio Appliance gets delivered as a qcow2 image, which gets attached to a virtual machine.

Trilio supports KVM-based hypervisors on x86 architectures, with the following properties:

The recommended size of the VM for the Trilio Appliance is:

The qcow2 image itself defines the 40GB disk size of the VM.

Software Requirements

In addition to the Trilio Appliance does Trilio contain components, which are installed directly into the Openstack itself.

Each Openstack distribution comes with a set of supported operating systems. Please check the to see, which Openstack Distribution is supported with which Operating System.

Additionally, it is necessary to have the nfs-common packages installed on the compute nodes in case of using the NFS protocol for the backup target.

OpenStack Barbican for encryption

Since Trilio for OpenStack 4.2 is T4O capable of providing encrypted backups

The Trilio for OpenStack solution is leveraging the OpenStack Barbican service to provide encryption capabilities for its backups.

To be precise, T4O is using secrets provided by Barbican to encrypt and decrypt backups. Barbican is therefore required to utilize this feature.

For example: RHOSP16.1 to RHOSP16.2 upgrade

1] Finish upgrade process for RHOSP cloud

Without any changes in TrilioVault, customer needs to complete RHOSP upgrade process first.

2] Now we need to upgrade TrilioVault to match with new RHOSP release

2.1] Identify correct TrilioVault release for new RHOSP release

Check if current TrilioVault release which is deployed on the RHOSP cloud supports new RHOSP release If it does not support new RHOSP release, identify the next TrilioVault release which supports this new RHOSP release.

2.2] Install TrilioVault using devops code compatible with new RHOSP release

Install TrilioVault release identified in step 2.1, Refer TrilioVault install for RHOSP Get the correct deployment document as per Triliovault release from Trilio support team.

Follow all the steps.

2.3] Done

The Trilio appliance can be reconfigured at any time to adjust the Trilio cluster to any changes in the Openstack environment or the general backup solution.

To reconfigure the Trilio Cluster go to the "Configure". The configure page shows the current configuration of the TriloVault cluster.

The configuration page also gives access to the ansible playbooks of the last successful configuration.

To start the reconfiguration of the Trilio Cluster click "Reconfigure" at the end of the table.

Follow the guide afterwards.

The Trilio database is following an older OpenStack database schema. This schema is not forgetting anything and is instead setting the deleted flag for any data that is no longer valid/deleted.

This leads to ever-growing databases, which can lead to performance degradation.

To counter this a database clean-up and optimization tool has been made available.

Change Trilio Dashboard password

To change the Trilio GUI password do:

• Login into the Trilio Dashboard

• Click on "admin" in the upper right corner to open the submenu

• Choose "Reset Password"

• Set the new Trilio password

In case of the Trilio Dashboard being lost it can be resetted as long as SSH access to the appliance is available.

To reset the password to its default do the following:

[root@TVM1 ~]# source /home/stack/myansible/bin/activate
(myansible) [root@TVM1 ~]# cd /home/stack/myansible/lib/python3.6/site-packages/tvault_configurator
(myansible) [root@TVM1 tvault_configurator]# python recreate_conf.py
(myansible) [root@TVM1 tvault_configurator]# systemctl restart tvault-config

The dashboard login will be reset to:

Username: admin
Password: password

To use the workloadmgr CLI tool on the Trilio appliance it is only necessary to activate the virtual environment of the workloadmgr

source /home/stack/myansible/bin/activate

Definition

Trilio can notify users via E-Mail upon the completion of backup and restore jobs.

The E-Mail will be sent to the owner of the Workload.

Requirements to activate E-Mail Notifications

To use the E-mail notifications, two requirements need to be met.

User E-Mail assigned

As the E-Mail will be sent to the owner of the Workload does the Openstack User, who created the workload, require to have an E-Mail address associated.

Trilio E-Mail Server configured

Trilio needs to know which E-Mail server to use, to send the E-mail notifications. Backup Administrators can do this in the "Backup Admin" area.

Activate/Deactivate the E-Mail Notifications

E-Mail notifications are activated tenant wide. To activate the E-Mail notification feature for a tenant follow these steps:

1. Login to Horizon

2. Navigate to the Backups

3. Navigate to Settings

4. Check/Uncheck the box for "Enable Email Alerts"

Example E-Mails

The following screenshots show example E-mails send by Trilio.

Trilio integrates natively with Openstack. This includes that Trilio communicates completely through APIs using the Openstack Endpoints. Trilio is also generating its own Openstack endpoints. In addition, is the Trilio appliance and the compute nodes writing to and reading from the backup target. These points affect the network planning for the Trilio installation.

Existing endpoints in Openstack

Openstack knows 3 types of endpoints:

• Public Endpoints

• Internal Endpoints

• Admin Endpoints

Each of these endpoint types is designed for a specific purpose. Public endpoints are meant to be used by the Openstack end-users to work with Openstack. Internal endpoints are meant to be used by the Openstack services to communicate with each other. Admin endpoints are meant to be used by Openstack administrators.

Out of those 3 endpoint types does only the admin endpoint sometimes contain APIs which are not available on any other endpoint type.

To learn more about Openstack endpoints please visit the official Openstack documentation.

Openstack endpoints required by Trilio

Trilio is communicating with all services of Openstack on a defined endpoint type. Which endpoint type Trilio is using to communicate with Openstack is decided during the configuration of the Trilio appliance.

The following network requirement can be identified this way:

• Trilio appliance needs access to the Keystone admin endpoint on the admin endpoint network

• Trilio appliance needs access to all endpoints of one type

Recommendation: Provide access to all Openstack Endpoint types

Trilio is recommending providing full access to all Openstack endpoints to the Trilio appliance to follow the Openstack standards and best practices.

Trilio is generating its own endpoints as well. These endpoints are pointing towards the Trilio Appliance directly. This means that using those endpoints will not send the API calls towards the Openstack Controller nodes first, but directly to the Trilio VM.

Following the Openstack standards and best practices, it is therefore recommended to put the Trilio endpoints on the same networks as the already existing Openstack endpoints. This allows to extend the purpose of each endpoint type to the Trilio service:

• The public endpoint to be used by Openstack users when using Trilio CLI or API

• The internal endpoint to communicate with the Openstack services

• The admin endpoint to use the required admin only APIs of Keystone

Backup target access required by Trilio

The Trilio solution is using backup target storage to securely place the backup data. Trilio is dividing its backup data into two parts:

1. Metadata

2. Volume Disk Data

The first type of data is generated by the Trilio appliance through communicating with the Openstack Endpoints. All metadata that is stored together with a backup is written by the Trilio Appliance to the backup target in the json format.

The second type of data is generated by the Trilio Datamover service running on the compute nodes. The Datamover service is reading the Volume Data from the Cinder or Nova storage and transferring this data as qcow2 image to the backup target. Each Datamover service is hereby responsible for the VMs running on its compute node.

The network requirements are therefor:

• The Trilio appliance needs access to the backup target

• Every compute node needs access to the backup target

Example of a typical Trilio network integration

Most Trilio customers are following the Openstack standards and best practices to have the public, internal, and admin endpoints on separate networks. They also typically don't have any network yet, which can access the desired backup target.

The starting network configuration typically looks like this:

Following the Openstack standards and Trilio's recommendation will the Trilio Appliance be placed on all those 3 networks. Further is the access to the backup target required by Trilio Appliance and Compute nodes. Here done by adding a 4th network.

The resulting network configuration would look like this:

It is of course possible to combine networks as necessary. As long as the required network access is available will Trilio work.

Other examples of Trilio network integrations

Each Openstack installation is different and so is the network configuration. There are endless possibilities of how to configure the Openstack network and how to implement the Trilio appliance into this network. The following three examples have been seen in production:

The first example is from a manufacturing company, which wanted to split the networks by function and decided to put the Trilio backup target on the internal network as the backup and recovery function was identified as an Openstack internal solution. This example looks complex but integrates Trilio just as recommended.

The second example is from a financial institute that wanted to be sure that the Openstack Users have no direct uncontrolled network access to the Openstack infrastructure. Following this example requires additional work as the internal HA-Proxy needs to be configured to correctly translates the API calls towards the Trilio

The third example is from a service company that was forced to treat Trilio as an external 3rd party solution, as we require a virtual machine running outside of Openstack. This kind of network configuration requires good planning on the Trilio endpoints and firewall rules.

After the Trilio VM has been configured and all components are installed can the license be applied.

The license can be applied either through the admin-tab in Horizon or the CLI

Apply license through Horizon

To apply the license through Horizon follow these steps:

1. Login to Horizon using admin user.

2. Click on Admin Tab.

3. Navigate to Backups-Admin

4. Navigate to Trilio

5. Navigate to License

6. Click "Update License"

7. Click "Choose File"

8. choose license-file on client system

9. click "Apply"

Apply license through CLI

• <license_file> ➡️ path to the license file

It is possible to configure Cinder to have multiple configurations and keyrings for CEPH.

In this case, the Trilio Datamover file needs to be extended with the CEPH information.

For Trilio to be able to work in such an environment it is required to put copies of each of these configurations and keyrings into a separate directory, which is then made known to the Trilio Datamover inside a [ceph] block in the tvault-contego.conf.

A tvault-contego.conf file with the extended [ceph] block would look like this.

It is possible to configure Cinder and Ceph to use different Ceph users for different Ceph pools and Cinder volume types. Or to have the nova boot volumes and cinder block volumes controlled by different users.

In the case of multiple Ceph users, it is required to adopt the keyring extension in the tvault-contego.conf inside the Ceph block.

The following example will try all files with the extension keyring that are located inside /etc/ceph to access the Ceph cluster for a Trilio related task.

Trilio is using a base64 hash for the mount point of NFS Backup targets. This hash makes sure, that multiple NFS Shares can be used with the same Trilio installation.

This base64 hash is part of the Trilio incremental backups as an absolute path of the backing files. This requires the usage of mount bind during a DR scenario or quick migration scenario.

In the case that there is time for a thorough migration there is another option to change the backing file and make the Trilio backups available on a different NFS Share. This option is updating the backing file to the new NFS Share mount point.

Backing file change script

Trilio is providing a shell script for the purpose of changing the backing file. This script is used after the Trilio appliance has been reconfigured to use the new NFS share.

Downloading the shell script

The Shell script is publicly available at:

Pre-Requisites

The following requirements need to be met before the change of the backing file can be attempted.

• The Trilio Appliance has been reconfigured with the new NFS Share

  • Please check for reconfiguring the Trilio Appliance

• The Openstack environment has been reconfigured with the new NFS Share

  • Please check for Red Hat Openstack Platform

  • Please check for Canonical Openstack

  • Please check for Kolla Ansible Openstack

  • Please check for Ansible Openstack

• The workloads are available on the new NFS Share

• The workloads are owned by nova:nova user

Usage

The shell script is changing one workload at a time.

Run the following command:

with

• /var/triliovault-mounts/<base64>/ being the new NFS mount path

• workload_<workload_id> being the workload to rebase

Logging of the procedure

The shell script is generating the following log file at the following location:

The log file will not get overwritten when the script is run multiple times. Each run of the script will append the available log file.

The Trilio Appliance Dashboard gives an overview of the running services and their Status inside the Cluster. The dashboard can be accessed using the virtual IP.

The Trilio Appliance Dashboard gives an overview of the running services and their Status inside the Cluster. This dashboard is accessible through the virtual IP.

It shows for each Trilio Appliance the Status of the following Trilio services:

• wlm-workloads

• wlm-scheduler

• wlm-api

• wlm-cron

To give administrators an overview of the HA status, does the dashboard also show the service status for:

• Pacemaker

• RabbitMQ

• MySQL Galera Cluster

The Trilio solution is using the qcow2 backing file to provide full synthetic backups.

Especially when the NFS backup target is used, there are scenarios at which this backing file needs to be updated to a new mount path.

To make this process easier and streamlined Trilio is providing the following rebase tool.

The Trilio Appliance is delivered as qcow2 image and runs as VM on top of a KVM Hypervisor.

Creating the cloud-init image

The Trilio appliance is utilizing cloud-init to provide the initial network and user configuration.

Cloud-init is reading it's information either from a metadata server or from a provided cd image. Trilio is utilizing the cd image.

Needed tools

To create the cloud-init image it is required to have genisoimage available.

#For RHEL and centos
yum install genisoimage
#For Ubuntu 
apt-get install genisoimage

Providing the Metadata

Cloud-init is using two files for it's metadata.

The first file is called meta-data and contains the information about the network configuration. Below is an example of this file.

[root@kvm]# cat meta-data
instance-id: triliovault
network-interfaces: |
   auto eth0
   iface eth0 inet static
   address 158.69.170.20
   netmask 255.255.255.0
   gateway 158.69.170.30

   dns-nameservers 11.11.0.51
local-hostname: localhost

The second file is called user-data and contains little scripts and information to set up for example the user passwords. Below is an example of this file.

[root@kvm]# cat user-data
#cloud-config
chpasswd:
  list: |
    root:password1
    stack:password2
  expire: False

creating the image file

The image is getting created using genisoimage follwing this general command:

genisoimage -output <name>.iso -volid cidata -joliet -rock </path/user-data> </path/meta-data>

An example of this command is shown below.

genisoimage  -output tvault-firstboot-config.iso -volid cidata -joliet -rock user-data meta-data

Spining up the Trilio appliance

After the cloud-init image has been created the TriloVault appliance can be spun up on the desired KVM server.

Extract the Trilio QCOW2 tar file using the following command :

tar Jxvf TrilioVault_file.tar.xz

See below an example command, how to spin up the Trilio appliance using virsh and the created iso image.

virt-install -n triliovault-vm  --memory 24576 --vcpus 8 \
--os-type linux \ 
--disk tvault-appliance-os-3.0.154.qcow2,device=disk,bus=virtio,size=40 \
--network bridge=virbr0,model=virtio \
--network bridge=virbr1,model=virtio \
--graphics none \
--import \
--disk path=tvault-firstboot-config.iso,device=cdrom

Uninstalling cloud-init after first boot

Once the Trilio appliance is up and running with it's initial configuration is it recommended to uninstall cloud-init.

To uninstall cloud-init, follow the example below.

sudo yum remove cloud-init

[RHOSP, TripleO and Kolla only] Verify nova UID/GID for nova user on the Appliance

Red Hat OpenStack, TripleO, and Kolla Ansible Openstack are using the nova UID/GID of 42436 inside their containers instead of 162:162 which is the standard in other Openstack environments.

Please verify that the nova UID/GID on the Trilio Appliance is still 42436,

[root@TVM1 ~]# id nova
uid=42436(nova) gid=42436(nova) groups=42436(nova),990(libvirt),36(kvm)

In case of the UID/GID being changed back to 162:162 follow these steps to set it back to 42436:42436.

1. Download the shell script that will change the user id

2. Assign executable permissions

3. Execute the script

4. Verify that the nova user and group id have changed to '42436'

## Download the shell script
$ curl -O https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/master/common/nova_userid.sh

## Assign executable permissions
$ chmod +x nova_userid.sh

## Execute the shell script to change 'nova' user and group id to '42436'
$ ./nova_userid.sh

## Ignore any errors and verify that 'nova' user and group id has changed to '42436'
$ id nova
   uid=42436(nova) gid=42436(nova) groups=42436(nova),990(libvirt),36(kvm)

Updating the appliance to the latest minor version

It is recommended to directly update the Trilio appliance to the latest version.

To do so follow the minor update guide provided here:

• Online update Trilio Appliance

• Offline update Trilio Appliance

Starting Trilio for Openstack 4.0 does Trilio for Openstack allow in-place upgrades.

The following versions can be upgraded to each other:

The upgrade process contains upgrading the Trilio appliance and the Openstack components and is dependent on the underlying operating system.

Enable mount-bind for NFS

In T4O 4.2 and later releases, calculation of the mount point has been changed. It is necessary to set up the mount-bind to make T4O 4.1 or older backups available for T4O 4.2 and T4O 4.2 onwards releases. Please follow this documentation to set up the mount bind for Canonical OpenStack.

Trilio is using the Openstack Keystone Trust system which enables the Trilio service user to act in the name of another Openstack user.

This system is used during all backup and restore features.

Trusts can only be worked with via CLI

List all trusts

workloadmgr trust-list

Show a trust

workloadmgr trust-show <trust_id>

• <trust_id> ➡️ ID of the trust to show

Create a trust

workloadmgr trust-create [--is_cloud_trust {True,False}] <role_name>

• <role_name> ➡️Name of the role that trust is created for

• --is_cloud_trust {True,False} ➡️ Set to true if creating cloud admin trust. While creating cloud trust use same user and tenant which used to configure Trilio and keep the role admin.

Delete a trust

workloadmgr trust-delete <trust_id>

• <trust_id> ➡️ ID of the trust to be deleted

On the Trilio Nodes

The Trilio Cluster contains multiple log files.

The main log is workloadmgr-workloads.log, which contains all logs about ongoing and past Trilio backup and restore tasks. It can be found at:

/var/log/workloadmgr/workloadmgr-workloads.log

The next important log is the workloadmgr-api.log, which contains all logs about API calls received by the Trilio Cluster. It can be found at:

/var/log/workloadmgr/workloadmgr-api.log

The log for the third service is the workloadmgr-scheduler.log, which contains all logs about the internal job scheduling between Trilio nodes in the Trilio Cluster.

/var/log/workloadmgr/workloadmgr-scheduler.log

The last but not least service running on the Trilio Nodes is the wlm-cron service, which is controlling the scheduled automated backups.

/var/log/workloadmgr/workloadmgr-workloads.log

In the case of using S3 as a backup target is there also a log file that keeps track of the S3-Fuse plugin used to connect with the S3 storage.

/var/log/workloadmgr/s3vaultfuse.py.log

Trilio Datamover service logs on RHOSP

Datamover API log

The log for the Trilio Datamover API service is located on the nodes, typically controller, where the Trilio Datamover API container is running under:

/var/log/containers/trilio-datamover-api/dmapi.log

Datamover log

The log for the Trilio Datamover service is located on the nodes, typically compute, where the Trilio Datamover container is running under:

/var/log/containers/trilio-datamover/tvault-contego.log

In case of S3 being used in the log for the S3 Fuse plugin located on the same nodes under:

/var/log/containers/trilio-datamover/tvault-object-store.log

Trilio Datamover service logs on Kolla Openstack

Datamover API log

The log for the Trilio Datamover API service is located on the nodes, typically controller, where the Trilio Datamover API container is running under:

/var/log/kolla/trilio-datamover-api/dmapi.log

Datamover log

The log for the Trilio Datamover service is located on the nodes, typically compute, where the Trilio Datamover container is running under:

/var/log/kolla/triliovault-datamover/tvault-contego.log

In case of S3 being used in the log for the S3 Fuse plugin located on the same nodes under:

/var/log/kolla/trilio-datamover/tvault-object-store.log

Trilio Datamover service logs on Ansible Openstack

Datamover API log

The log for the Trilio Datamover API service is located on the nodes, typically controller, where the Trilio Datamover API container is running. Log into the dmapi container using lxc-attach command (example below).

lxc-attach -n controller_dmapi_container-a11984bf

The log file is then located under:

/var/log/dmapi/dmapi.log

Datamover log

The log for the Trilio Datamover service is typically located on the compute nodes and the logs can be found here:

/var/log/tvault-contego/tvault-contego.log

In case of S3 being used in the log for the S3 Fuse plugin located on the same nodes under:

/var/log/tvault-object-store/tvault-object-store.log

Troubleshooting inside a complex environment like Openstack can be very time-consuming. The following tipps will help to speed up the troubleshooting process to identify root causes.

What is happening where

Openstack and Trilio are divided into multiple services. Each service has a very specific purpose that is called during a backup or recovery procedure. Knowing which service is doing what helps to understand where the error is happening, allowing more focused troubleshooting.

Trilio cluster

The Trilio Cluster is the Controller of Trilio. It receives all Workload related requests from the users.

Every task of a backup or restore process is triggered and managed from here. This includes the creation of the directory structure and initial metadata files on the Backup Target.

During a backup process

During a backup process is the Trilio cluster also responsible to gather the metadata about the backed up VMs and networks from the Openstack environment. It is sending API calls towards the Openstack endpoints on the configured endpoint type to fetch this information. Once the metadata has been received does the Trilio Cluster write it as json files on the Backup Target.

The Trilio cluster is also sending the Cinder Snapshot command.

During a restore process

During restore process is the Trilio cluster reading the VM metadata from its Database and is using the metadata to create the Shell for the restore. It is sending API calls to the Openstack environment to create the necessary resources.

dmapi

The dmapi service is the connector between the Trilio cluster and the datamover running on the compute nodes.

The purpose of the dmapi service is to identify which compute node is responsible for the current backup or restore task. To do so is the dmapi service connecting to the nova api database requesting the compute hose of a provided VM.

Once the compute host has been identified is the dmapi forwarding the command from the Trilio Cluster to the datamover running on the identified compute host.

datamover

The datamover is the Trilio service running on the compute nodes.

Each datamover is responsible for the VMs running on top of its compute node. A datamover can not work with VMs running on a different compute node.

The datamover is controlling the freeze and thaw of VMs as well as the actual movement of the data.

Everything on the Backup Target is happening as user nova

Trilio is reading and writing on the Backup Target as nova:nova.

The POSIX user-id and group-id of nova:nova need to be aligned between the Trilio Cluster and all compute nodes. Otherwise backup or restores may fail with permission or file not found issues.

Alternativ ways to achieve the goal are possible, as long as all required nodes can fully write and read as nova:nova on the Backup Target.

It is recommended to verify the required permissions on the Backup Target in case of any errors during the data transfer phase or in case of any file permission errors.

Input/Output Error with Cohesity NFS

On Cohesity NFS if an Input/Output error is observed, then increase the timeo and retrans parameter values in your NFS options.

Timeout receiving packet error in multipath iscsi environment

Logging inside all datamover containers and add uxsock_timeout with value as 60000 which is equals to 60 sec inside /etc/multipath.conf.Restart datamover container

Trilio Trustee Role

Trilio is using RBAC to allow the usage of Trilio features to users.

This trustee role is absolutely required and can not be overwritten using the admin role.

It is recommended to verify the assignment of the Trilio Trustee Role in case of any permission errors from Trilio during creation of Workloads, backups or restores.

Openstack Quotas

Trilio is creating Cinder Snapshots and temporary Cinder Volumes. The Openstack Quotas need to allow that.

Every disk that is getting backed up requires one temporary Cinder Volumes.

Every Cinder Volume that is getting backup requires two Cinder Snapshots. The second Cinder Snapshot is temporary to calculate the incremental.

Trilio Configurator

Once Trilio is configured use virtual IP to access its dashboard. If service status panels on the dashboard page are not visible then access the virtual IP on port 3001 (https://<T4O-VIP>:3001/) and accept the SSL exception, and then refresh the dashboard page.

\

1. Can Trilio for OpenStack restore instance UUIDs?

Answer: NO

Trilio for OpenStack does not restore Instance UUIDs (also known as Instance IDs). The only scenario where we do not modify the Instance UUID is during an Inplace Restore, where we only recover the data without creating new instances.

When Trilio for OpenStack restores virtual machines (VMs), it effectively creates new instances. This means that new Virtual Machine Instance UUIDs are generated for the restored VMs. We achieve this by orchestrating a call to Nova, which creates new VMs with new UUIDs.

By following this approach, we maintain the principles of OpenStack and auditing. We do not update or modify existing database entries when objects are deleted and subsequently recovered. Instead, all deletions are marked as such, and new instances, including the recovered ones, are created as new objects in the Nova tables. This ensures compliance and preserves the integrity of the OpenStack environment.

2. Can Trilio for OpenStack restore MAC addresses?

Answer: YES

Trilio can restore the VMs MAC address, however, there is a caveat when restoring a virtual machine (VM) to a different IP address: a new MAC address will be assigned to the VM.

In the case of a One-Click Restore, the original MAC addresses and IP addresses will be recovered, but the VM will be created with a new UUID, as mentioned in question #1.

When performing a Selective Restore, you have the option to recover the original MAC address. To do so, you need to select the original IP address from the available dropdown menu during the recovery process.

By choosing the original IP address, Trilio for OpenStack will ensure that the VM is restored with its original MAC address, providing more flexibility and customization in the restoration process.

Example of Selective Restore with original MAC (and IP address):

1. In this example, we have taken a Trilio backup of a VM called prod-1.

1. The VM is deleted and we perform a Selective Restore of a VM called prod-1, selecting the IP address it was originally assigned from the drop-down menu:

1. Trilio then restores the VM with the original MAC address:

1. If you left the option as "Choose next available IP address", it will assign a new MAC to the VM instead as Neutron maps all MAC addresses to IP addresses on the Subnet - so logically a new IP will result in a new MAC address.

Trilio Workloads are designed to allow a Desaster Recovery without the need to backup the Trilio database.

As long as the Trilio Workloads are existing on the Backup Target Storage and a Trilio installation has access to them, it is possible to restore the Workloads.

Disaster Recovery Process

1. and Trilio for the target cloud

4. Notify users to of Workloads being available

Mount-paths

Trilio incremental Snapshots involve a backing file to the prior backup taken, which makes every Trilio incremental backup a synthetic full backup.

Trilio is using qcow2 backing files for this feature:

As can be seen in the example is the backing file an absolute path, which makes it necessary, that this path exists so the backing files can be accessed.

Trilio is using the base64 hashing algorithm for the NFS mount-paths, to allow the configuration of multiple NFS Volumes at the same time. The hash value is calculated using the provided NFS path.

When the path of the backing file is not available on the Trilio appliance and Compute nodes, will the restores of incremental backups fail.

The tested and recommended method to make the backing files available is creating the required directory path and using mount --bind to make the path available for the backups.

Running the mount --bind command will make the necessary path available until the next reboot. If it is required to have access to the path beyond a reboot is it necessary to edit the fstab.

Uninstall Trilio Services

The Trilio Ansible OpenStack playbook can be run to uninstall the Trilio services.

cd /opt/openstack-ansible/playbooks
openstack-ansible os-tvault-install.yml --tags "tvault-all-uninstall"

Destroy Trilio Datamover API container

To cleanly remove the Trilio Datamover API container run the following Ansible playbook.

cd /opt/openstack-ansible/playbooks
openstack-ansible lxc-containers-destroy.yml --limit "DMPAI CONTAINER_NAME"

Clean openstack_user_config.yml

Remove the tvault-dmapi_hosts and tvault_compute_hosts entries from /etc/openstack_deploy/openstack_user_config.yml

#tvault-dmapi
tvault-dmapi_hosts:
  infra-1:
    ip: 172.26.0.3
  infra-2:
    ip: 172.26.0.4
    
#tvault-datamover
tvault_compute_hosts:
  infra-1:
    ip: 172.26.0.7
  infra-2:
    ip: 172.26.0.8

Remove Trilio haproxy settings in user_variables.yml

Remove Trilio Datamover API settings from /etc/openstack_deploy/user_variables.yml

# Datamover haproxy setting
haproxy_extra_services:
  - service:
      haproxy_service_name: datamover_service
      haproxy_backend_nodes: "{{ groups['dmapi_all'] | default([]) }}"
      haproxy_ssl: "{{ haproxy_ssl }}"
      haproxy_port: 8784
      haproxy_balance_type: http
      haproxy_backend_options:
        - "httpchk GET / HTTP/1.0\\r\\nUser-agent:\\ osa-haproxy-healthcheck"

Remove Trilio Datamover API inventory file

rm /opt/openstack-ansible/inventory/env.d/tvault-dmapi.yml

Remove Trilio Datamover API service endpoints

 source cloudadmin.rc
 openstack endpoint delete "internal datamover service endpoint_id"
 openstack endpoint delete "public datamover service endpoint_id"
 openstack endpoint delete "admin datamover service endpoint_id"

Delete Trilio Datamover API database and user

• Go inside galera container.

• Login as root user in mysql database engine.

• Drop dmapi database.

• Drop dmapi user

lxc-attach -n "GALERA CONTAINER NAME"
mysql -u root -p "root password"
DROP DATABASE dmapi;
DROP USER dmapi;

Remove dmapi rabbitmq user from rabbitmq container

• Go inside rabbitmq container.

• Delete dmapi user.

• Delete dmapi vhost.

lxc-attach -n "RABBITMQ CONTAINER NAME"
rabbitmqctl delete_user dmapi
rabbitmqctl delete_vhost /dmapi

Clean haproxy

Remove /etc/haproxy/conf.d/datamover_service file.

rm  /etc/haproxy/conf.d/datamover_service

Remove HAproxy configuration entry from /etc/haproxy/haproxy.cfg file.

frontend datamover_service-front-1
    bind ussuriubuntu.triliodata.demo:8784 ssl crt /etc/ssl/private/haproxy.pem ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
    option httplog
    option forwardfor except 127.0.0.0/8
    reqadd X-Forwarded-Proto:\ https
    mode http
    default_backend datamover_service-back

frontend datamover_service-front-2
    bind 172.26.1.2:8784
    option httplog
    option forwardfor except 127.0.0.0/8
    mode http
    default_backend datamover_service-back


backend datamover_service-back
    mode http
    balance leastconn
    stick store-request src
    stick-table type ip size 256k expire 30m
    option forwardfor
    option httplog
    option httpchk GET / HTTP/1.0\r\nUser-agent:\ osa-haproxy-healthcheck


    server controller_dmapi_container-bf17d5b3 172.26.1.75:8784 check port 8784 inter 12000 rise 1 fall 1

Restart the HAproxy service.

systemctl restart haproxy

Remove certificates from Compute nodes

rm -rf /opt/config-certs/rabbitmq
rm -rf /opt/config-certs/s3

Destroy the Trilio VM Cluster

List all VMs running on the KVM node

virsh list

Destroy the Trilio VMs

virsh destroy <Trilio VM Name or ID>

Undefine the Trilio VMs

virsh undefine <Trilio VM name>

Delete the TrlioVault VM disk from KVM Host storage

To gracefully shutdown/restart the Trilio cluster the following steps are recommended.

Verify no snapshots or restores are running

It is recommended to verify that no snapshots or restores are running on the Trilio Cluster.

This can be verified using the following two commands:

workloadmgr snapshot-list --all=True
workloadmgr restore-list

Identify the master node for the VIP(s) and wlm-cron service

The Trilio cluster is using the pacemaker service for setting the VIP(s) of the cluster and controlling the active node for the wlm-cron service. The identified node will be the last to shut down in case that the whole cluster gets shut down.

This can be checked using the following command:

pcs status

In the following example is the master node the tvm1

pcs status
Cluster name: triliovault

WARNINGS:
Corosync and pacemaker node names do not match (IPs used in setup?)

Stack: corosync
Current DC: tvm3 (version 1.1.23-1.el7_9.1-9acf116022) - partition with quorum
Last updated: Thu Aug 26 12:10:32 2021
Last change: Thu Aug 26 08:02:51 2021 by root via crm_resource on tvm1

3 nodes configured
8 resource instances configured

Online: [ tvm1 tvm2 tvm3 ]

Full list of resources:

 virtual_ip     (ocf::heartbeat:IPaddr2):       Started tvm1
 virtual_ip_public      (ocf::heartbeat:IPaddr2):       Started tvm1
 virtual_ip_admin       (ocf::heartbeat:IPaddr2):       Started tvm1
 virtual_ip_internal    (ocf::heartbeat:IPaddr2):       Started tvm1
 wlm-cron       (systemd:wlm-cron):     Started tvm1
 Clone Set: lb_nginx-clone [lb_nginx]
     Started: [ tvm1 ]
     Stopped: [ tvm2 tvm3 ]

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

Shutdown/Restart of a single node in the cluster

A single node in the cluster can be shut down or restarted without issues. All services will come up and the RabbitMQ and Galeera service will rejoin the remaining cluster.

When the master node gets shutdown or restarted the VIP(s) and the wlm-cron service will switch to one of the remaining cluster nodes.

Stop the services on the node

To speed up the shutdown/restart process it is recommended to stop the Trilio services, the RabbitMQ service, and the MariaDB service on the node.

systemctl stop wlm-api
systemctl stop wlm-scheduler
systemctl stop wlm-workloads
systemctl stop mysqld
rabbitmqctl stop

Shutdown/Restart the node

After the services have been stopped the node can be restarted or shut down using standard Linux commands.

reboot
shutdown

Restarting the complete cluster node by node

Restarting the whole cluster node by node follows the same procedure as restarting a single node, with the difference that each restarted node needs to be fully started again before the next node can be restarted.

Shutdown/Restart the complete cluster as a whole

When the complete cluster needs to get stopped and restarted at the same time the following procedure needs to be completed.

The procedure on a high level is:

• Shutdown the two slave nodes

• Shutdown the master node

• Start the master node

• Enable the Galeera cluster

• Start the two slave nodes

Shutdown the two slave nodes

Before shutting down the two slave nodes it is recommended to stop running Trilio services, the RabbitMQ server, and the MariaDB on the nodes.

systemctl stop wlm-api
systemctl stop wlm-scheduler
systemctl stop wlm-workloads
systemctl stop mysqld
rabbitmqctl stop

Afterward, the nodes can be shut down.

shutdown

Shutdown the master node

Before shutting down the master node it is recommended to stop running Trilio services, the RabbitmQ server, the MariaDB, the wlm-cron and the VIP(s) resource in Pacemaker.

systemctl stop wlm-api
systemctl stop wlm-scheduler
systemctl stop wlm-workloads
systemctl stop mysqld
rabbitmqctl stop
pcs resource stop wlm-cron
pcs resource stop lb_nginx-clone

Afterward, the node can be shut down.

shutdown

Start the master node

The first server that is getting booted will be the master node. It is highly recommended that the old master node will be booted first again.

Enable the Galeera cluster

Login into the freshly started master node and run the following command. This will restart the Galeera cluster with this node as master.

galera_new_cluster

Start the slave nodes

After the master node has been booted and the Galeera cluster started the remaining nodes can be started and will automatically rejoin the Trilio cluster.

By default is the Trilio GUI available on all NICs on port 443.

To limit this to only one IP the following steps need to be applied.

Network Setup

The Trilio Appliance provides by default the possibility of 4 VIPs.

• A general VIP which can be used for everything

• A public VIP for the public endpoint

• An internal VIP for the internal endpoint

• An admin VIP for the admin endpoint

Should an additional VIP be required to restrict the access of the Trilio Dashboard to this VIP the new VIP needs to be created as a new resource inside the PCS cluster.

pcs resource create dashboard_ip ocf:heartbeat:IPaddr2 ip=<new_vip> cidr_netmask=<netmask> nic=<new_nw_interface> op monitor interval=30s
pcs constraint colocation add dashboard_ip virtual_ip

Nginx setup

When the new dashboard_ip has been created or decided, then the next step is to set up the proxy forwarding inside Nginx, which will make the Trilio GUI available through port 8000.

1. Create new conf file at /etc/nginx/conf.d/tvault-dashboard.conf. Replace variables dashboard_ip and virtual_ip as configured or decided.

   Copy

   server {
       listen <dashboard_ip>:8000 ssl ;
       ssl_certificate "/opt/stack/data/cert/workloadmgr.cert";
       ssl_certificate_key "/opt/stack/data/cert/workloadmgr.key";
       keepalive_timeout 65;
       proxy_read_timeout 1800;
       access_log on;
       location / {
           proxy_set_header Host $host:$server_port;
           proxy_set_header X-Real-IP $remote_addr;
           proxy_pass https://<virtual_ip>:443;
       }
   }
   server {
       listen <dashboard_ip>:3001 ssl ;
       ssl_certificate "/opt/stack/data/cert/workloadmgr.cert";
       ssl_certificate_key "/opt/stack/data/cert/workloadmgr.key";
       keepalive_timeout 65;
       proxy_read_timeout 1800;
       access_log on;
       location / {
           proxy_set_header Host $host:$server_port;
           proxy_set_header X-Real-IP $remote_addr;
           proxy_pass https://<virtual_ip>:3001;
       }
   }
   

2. edit /etc/nginx/nginx.conf and uncomment line #include /etc/nginx/conf.d/*.conf;

3. check nginx syntax: nginx -t

4. reload nginx conf: nginx -s reload

5. Verify if the new cluster resource is visible or not using pcs resource command and by accessing the dashboard_ip.

Limit the access of the Dashboard

The configured dashboard_ip will always end on the nginx service on port 8000 and will then be forwarded to the local dashboard service on port 443.

This configuration limits the required access to the local dashboard service to the Trilio appliance cluster itself. All other connections on port 443 can be dropped.

The following commands will set the required iptable rules.

iptables -A INPUT -p tcp -s tvm1,tvm2,tvm3 --dport 80 -j ACCEPT
iptables -A INPUT -p tcp -s tvm1,tvm2,tvm3 --dport 443 -j ACCEPT
iptables -A INPUT -p tcp --dport 80 -j DROP
iptables -A INPUT -p tcp --dport 443 -j DROP

Verify the accessibility as required

At this point is the Trilio GUI only reachable on the dashboard_ip on port 8000. Accessing the Trilio GUI through any other IP or on port 443 is not allowed.

https://<dashboard_ip>:8000

Same cloud - different owner

Migration within the same cloud to a different owner Cloud A — Domain A — Project A — User A => Cloud A — Domain A — Project A — User B Cloud A — Domain A — Project A — User A => Cloud A — Domain A — Project B — User B Cloud A — Domain A — Project A — User A =>Cloud A — Domain B — Project B — User B

Steps used:

1. Create a secret for Project A in Domain A via User A.

2. Create encrypted workload in Project A in Domain A via User A. Take snapshot.

3. Reassign workload to new owner

4. Load rc file of User A & provide read only rights through acl to the new owner

   openstack acl user add --user <userB_id> <secret_href> --insecure

Different cloud

Migration between clouds Cloud A — Domain A — Project A — User A => Cloud B — Domain B — Project B — User B

Steps used:

1. Create a secret for Project A in Domain A via User A.

2. Create an encrypted workload in Project A in Domain A via User A. Trigger snapshot.

3. Reassign workload to Cloud B - Domain B — Project B — User B

4. Load RC file of User B.

5. Create a secret for Project B in Domain B via User B with the same payload used in Cloud A.

6. Create token via “openstack token issue --insecure”

7. Add migrated workload's metadata to the new secret (provide issued token to Auth-Token & workload id to matadata as below)

curl -i -X PUT \
   -H "X-Auth-Token:gAAAAABh0ttjiKRPpVNPBjRjZywzsgVton2HbMHUFrbTXDhVL1w2zCHF61erouo4ZUjGyHVoIQMG-NyGLdR7nexmgOmG7ed66LJ3IMVul1LC6CPzqmIaEIM48H0kc-BGvhV0pvX8VMZiozgFdiFnqYHPDvnLRdh7cK6_X5dw4FHx_XPmkhx7PsQ" \
   -H "Content-Type:application/json" \
   -d \
'{
  "metadata": {
      "workload_id": "c13243a3-74c8-4f23-b3ac-771460d76130",
      "workload_name": "workload-c13243a3-74c8-4f23-b3ac-771460d76130"
    }
}' \
 'https://kolla-victoria-ubuntu20-1.triliodata.demo:9311/v1/secrets/f3b2fce0-3c7b-4728-b178-7eb8b8ebc966/metadata'
 
 
curl -i -X GET \
   -H "X-Auth-Token:gAAAAABh0ttjiKRPpVNPBjRjZywzsgVton2HbMHUFrbTXDhVL1w2zCHF61erouo4ZUjGyHVoIQMG-NyGLdR7nexmgOmG7ed66LJ3IMVul1LC6CPzqmIaEIM48H0kc-BGvhV0pvX8VMZiozgFdiFnqYHPDvnLRdh7cK6_X5dw4FHx_XPmkhx7PsQ" \
 'https://kolla-victoria-ubuntu20-1.triliodata.demo:9311/v1/secrets/f3b2fce0-3c7b-4728-b178-7eb8b8ebc966/metadata'

Trilio and Canonical have started a partnership to ensure a native deployment of Trilio using JuJu Charms.

Those JuJu Charms are publicly available as Open Source Charms.

The documentation of the charms can be found here:

Juju charms for OpenStack Yoga release onwards

Juju charms for other supported OpenStack releases upto wallaby

Prerequisite

Have a canonical OpenStack base setup deployed for a required release like Jammy Zed/Yoga, Focal yoga/Wallaby/Victoria/Ussuri, or Bionic Ussuri/Queens.

Steps to install the Trilio charms

1. Export the OpenStack base bundle

juju export-bundle --filename openstack_base_file.yaml

2. Create a Trilio overlay bundle as per the OpenStack setup release using the charms given above.

Some sample Trilio overlay bundles can be found here.

3. If file search functionality is required, provision any additional node(s) that will be required for deploying the Trilio Workload manager (trilio-wlm) as a VM instead of lxd container(s).

4. Commission the additional node from MAAS UI.

5. Do a dry run to check if the Trilio bundle is working

juju deploy --dry-run ./openstack_base_file.yaml --overlay <Trilio bundle path>

6. Do the deployment

juju deploy ./openstack_base_file.yaml --overlay <Trilio bundle path>

7. Wait till all the Trilio units are deployed successfully. Check the status via juju status command.

8. Once the deployment is complete, perform the below operations:

1. Create cloud admin trust

juju run-action --wait trilio-wlm/leader create-cloud-admin-trust password=<openstack admin password>

juju run --wait trilio-wlm/leader create-cloud-admin-trust password=<openstack admin password>

1. Add license

juju attach-resource trilio-wlm license=<Path to trilio license file>

juju run-action --wait trilio-wlm/leader create-license

juju run --wait trilio-wlm/leader create-license

Note: Reach out to the Trilio support team for the license file.

For multipath enabled environments, perform the following actions

1. log into each nova compute node

2. add uxsock_timeout with value as 60000 (i.e. 60 sec) in /etc/multipath.conf

3. restart tvault-contego service

Sample Trilio overlay bundles

A few Sample overlay bundles for different OpenStack versions can be found HERE.

Trilio for OpenStack is generating a base64 hash value for every NFS backup target connected to the T4O solution. This enables T4O to mount multiple NFS backup targets to the same T4O installation.

The mountpoints are generated utilizing a hash value inside the mountpoint, providing a unique mount for every NFS backup target.

This mountpoint is then used inside the incremental backups to point to the qcow2 backing files. The backing file path is required as a full path and can not be set as a relative path. This is a limitation of the qcow2 format.

T4O 4.2 has changed how the hash value gets calculated. T4O 4.1 and prior calculated the hash value out of the complete NFS path provided as shown in the example below.

# echo -n 10.10.2.20:/Trilio_Backup | base64
MTAuMTAuMi4yMDovVHJpbGlvX0JhY2t1cA==

T4O 4.2 is now only considering the NFS directory part for the hash value as shown below.

# echo -n /Trilio_Backup | base64
L1RyaWxpb19CYWNrdXA=

It is therefore necessary to make older backups taken by T4O 4.1 or prior available for TVO4.2 to restore.

This can be done by one of two methods:

1. Rebase all backups and change the backing file path

2. Make the old mount point available again and point it to the new one using mount bind

Rebase all backups

Trilio is providing a script, that is taking care of the rebase procedure. This script can be downloaded from the following location.

Copy and use the script from the Trilio appliance as the nova user.

The script is requiring the complete path to the workload directory on the backup target.

./backing_file_update.sh /var/triliovault-mounts/<base64>/workload_<workload_id>

This needs to be repeated until all workloads have been rebased.

Use mount bind with the old mount point

This method is generating a second mount point based on the old hash value calculation and then mounting the new mount point to the old one. By doing this mount bind both mount points will be available and point to the same backup target.

To use this method the following information needs to be available:

• old mount path

• new mount path

Examples of how to calculate the base64 hash value are shown below.

Old mountpoint hash value:

# echo -n 10.10.2.20:/Trilio_Backup | base64
MTAuMTAuMi4yMDovVHJpbGlvX0JhY2t1cA==

New mountpoint hash value:

# echo -n /Trilio_Backup | base64
L1RyaWxpb19CYWNrdXA=

The mount bind needs to be done for all Trilio appliances and Datamover services.

Trilio Appliance

To enable the mount bind on the Trilio appliance follow these steps:

1. Create the old mountpoint directory mkdir -p old_mount_path

2. run mount --bind command mount --bind <new_mount_path> <old_mount_path>

3. set permissions for mountpoint chmod 777 <old_mount_path>

An example is given below.

mkdir /var/triliovault-mounts/MTkyLjE2OC4xLjM1Oi9tbnQvdHZhdWx0L3R2bTQ=/
mount --bind /var/triliovault-mounts/L21udC90dmF1bHQvdHZtNA==/ /var/triliovault-mounts/MTkyLjE2OC4xLjM1Oi9tbnQvdHZhdWx0L3R2bTQ=
chmod 777 /var/triliovault-mounts/MTkyLjE2OC4xLjM1Oi9tbnQvdHZhdWx0L3R2bTQ=/

RHOSP/TripleO Datamover

The following steps need to be done on the overcloud compute node. They don't need to be done inside any container.

To enable the mount bind on the Trilio appliance follow these steps:

1. Create the old mountpoint directory mkdir -p old_mount_path

2. run mount --bind command mount --bind <new_mount_path> <old_mount_path>

3. set permissions for mountpoint chmod 777 <old_mount_path>

mkdir /var/lib/nova/triliovault-mounts/MTkyLjE2OC4xLjM1Oi9tbnQvdHZhdWx0L3R2bTQ=/
mount --bind /var/lib/nova/triliovault-mounts/L21udC90dmF1bHQvdHZtNA==/ /var/lib/nova/triliovault-mounts/MTkyLjE2OC4xLjM1Oi9tbnQvdHZhdWx0L3R2bTQ=
chmod 777 /var/lib/nova/triliovault-mounts/MTkyLjE2OC4xLjM1Oi9tbnQvdHZhdWx0L3R2bTQ=/

Kolla Ansible OpenStack Datamover

The following steps need to be done on the overcloud compute node. They don't need to be done inside any container.

To enable the mount bind on the Trilio appliance follow these steps:

1. Create the old mountpoint directory mkdir -p old_mount_path

2. run mount --bind command mount --bind <new_mount_path> <old_mount_path>

3. set permissions for mountpoint chmod 777 <old_mount_path>

mkdir /var/trilio/triliovault-mounts/MTkyLjE2OC4xLjM1Oi9tbnQvdHZhdWx0L3R2bTQ=/
mount --bind /var/trilio/triliovault-mounts/L21udC90dmF1bHQvdHZtNA==/ /var/trilio/triliovault-mounts/MTkyLjE2OC4xLjM1Oi9tbnQvdHZhdWx0L3R2bTQ=
chmod 777 /var/trilio/triliovault-mounts/MTkyLjE2OC4xLjM1Oi9tbnQvdHZhdWx0L3R2bTQ=/

Ansible OpenStack Datamover

The following steps need to be done on the overcloud compute node. They don't need to be done inside any container.

To enable the mount bind on the Trilio appliance follow these steps:

1. Create the old mountpoint directory mkdir -p old_mount_path

2. run mount --bind command mount --bind <new_mount_path> <old_mount_path>

3. set permissions for mountpoint chmod 777 <old_mount_path>

mkdir /var/triliovault-mounts/MTkyLjE2OC4xLjM1Oi9tbnQvdHZhdWx0L3R2bTQ=/
mount --bind /var/triliovault-mounts/L21udC90dmF1bHQvdHZtNA==/ /var/trilio/triliovault-mounts/MTkyLjE2OC4xLjM1Oi9tbnQvdHZhdWx0L3R2bTQ=
chmod 777 /var/triliovault-mounts/MTkyLjE2OC4xLjM1Oi9tbnQvdHZhdWx0L3R2bTQ=/

Canonical Openstack WLM & Datamover containers

Canonical OpenStack does the creation of the mountpoint and the mount bind through JuJu using the following commands.

To create the mountpoint, if it doesn't already exist:

juju exec [-m <model>] --application trilio-data-mover "sudo -u nova mkdir /var/triliovault-mounts/MTkyLjE2OC4xLjM1Oi9tbnQvdHZhdWx0L3R2bTQ=/"
juju exec [-m <model>] --application trilio-wlm "sudo -u nova mkdir /var/triliovault-mounts/MTkyLjE2OC4xLjM1Oi9tbnQvdHZhdWx0L3R2bTQ=/

To create the mount bind

juju exec [-m <model>] --application trilio-data-mover "sudo mount --bind /var/triliovault-mounts/L21udC90dmF1bHQvdHZtNA==/ /var/triliovault-mounts/MTkyLjE2OC4xLjM1Oi9tbnQvdHZhdWx0L3R2bTQ=/"
juju exec [-m <model>] --application trilio-wlm "sudo mount --bind /var/triliovault-mounts/L21udC90dmF1bHQvdHZtNA==/ /var/triliovault-mounts/MTkyLjE2OC4xLjM1Oi9tbnQvdHZhdWx0L3R2bTQ=/"

Introduction

Filename and location:

This file is used only when the user wants to configure 'multiple IP/endpoints based NFS share' as a backup target for Trilio. In all other cases like single IP NFS, S3 this file does not get used. Follow regular install doc.

If the user is using multiple-IP/endpoints based NFS share as a backup target for triliovault then triliovault mounts any one IP/endpoint on a given compute node for a given NFS share. Users can distribute NFS share IPs/endpoints across compute nodes.

'triliovault_nfs_map_input.yml' file is a tool that users can use to distribute/load balance NFS share endpoints across compute nodes in a given cloud.

Examples

Using hostnames

Here, the User has ‘one’ NFS share exposed with three IP addresses. 192.168.1.34, 192.168.1.35, 192.168.1.33 Share directory path is: /var/share1

So, this NFS share supports the following full paths that clients can mount:

There are 32 compute nodes in the OpenStack cloud. 30 node hostnames have the following naming pattern

The remaining 2 node hostnames do not follow any format/pattern.

Now the mapping file will look like this

Using IPs

Compute node IP range used here: 172.30.3.11-40 and 172.30.4.40, 172.30.4.50 Total of 32 compute nodes

Getting the correct compute hostnames/IPs

RHOSP or TripleO

Use the following command to get compute hostnames. Check the ‘Name' column. Use these exact hostnames in 'triliovault_nfs_map_input.yml' file.

In the following command output, ‘overcloudtrain1-novacompute-0' and ‘overcloudtrain1-novacompute-1' are correct hostnames.

Kolla-Ansible OpenStack

Compute hostnames/IPs should match in kolla-ansible inventory file and triliovault_nfs_map_input.yml.

If IP addresses are sued in kolla-ansible inventory file then the same IP addresses should be used in ‘triliovault_nfs_map_input.yml' file too. If the kolla-ansible deploy command looks like the following,

kolla-ansible -i multinode deploy

then, the inventory file is 'multinode'. Generally, it is available at “/root/multinode“.

OpenStack Ansible

Compute hostnames/IPs should match in the Openstack Ansible inventory file and triliovault_nfs_map_input.yml

If IP addresses have been used in the Openstack-ansible inventory file then you should use the same IP addresses in the ‘triliovault_nfs_map_input.yml' file too.

Generally inventory file available at /etc/openstack_deploy/openstack_user_config.yml

Openstack Ansible deploy command looks like following,

openstack-ansible os-tvault-install.yml

Openstack Ansible automatically picks up values that the user has set in the file openstack_user_config.yml

Clean triliovault_datamover_api container

The container needs to be cleaned on all nodes where the triliovault_datamover_api container is running. The Kolla Openstack inventory file helps to identify the nodes with the service.

Following steps need to be done to clean the triliovault_datamover_api container:

Stop the triliovault_datamover_api container.

Remove the triliovault_datamover_api container.

Clean /etc/kolla/triliovault-datamover-api directory.

Clean log directory of triliovault_datamover_api container.

Clean triliovault_datamover container

The container needs to be cleaned on all nodes where the triliovault_datamover container is running. The Kolla Openstack inventory file helps to identify the nodes with the service.

Following steps need to be done to clean the triliovault_datamover container:

Stop the triliovault_datamover container.

Remove the triliovault_datamover container.

Clean /etc/kolla/triliovault-datamover directory.

Clean log directory of triliovault_datamover container.

Clean haproxy of Trilio Datamover API

The Trilio Datamover API entries need to be cleaned on all haproxy nodes. The Kolla Openstack inventory file helps to identify the nodes with the service.

Following steps need to be done to clean the haproxy container:

Clean Kolla Ansible deployment procedure

Delete all Trilio related entries from:

Trilio entries can be found in:

• /usr/local/share/kolla-ansible/ansible/roles/ ➡️ There is a role triliovault

• /etc/kolla/globals.yml➡️ Trilio entries had been appended at the end of the file

• /etc/kolla/passwords.yml➡️ Trilio entries had been appended at the end of the file

• /usr/local/share/kolla-ansible/ansible/site.yml ➡️ Trilio entries had been appended at the end of the file

• /root/multinode ➡️ Trilio entries had been appended at the end of this example inventory file

Revert to original Horizon container

Run deploy command to replace the Trilio Horizon container with original Kolla Ansible Horizon container.

Clean Keystone resources

Trilio created a dmapi service with dmapi user.

Clean Trilio database resources

Trilio Datamover API service has its own database in the Openstack database.

Login to Openstack database as root user or user with similar priviliges.

Delete dmapi database and user.

Destroy the Trilio VM Cluster

List all VMs running on the KVM node

Destroy the Trilio VMs

Undefine the Trilio VMs

Delete the TrlioVault VM disk from KVM Host storage

Definition

Every Workload has its own schedule. Those schedules can be activated, deactivated and modified.

A schedule is defined by:

• Status (Enabled/Disabled)

• Start Day/Time

• End Day

• Hrs between 2 snapshots

Disable a schedule

Using Horizon

To disable the scheduler of a single Workload in Horizon do the following steps:

1. Login to the Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload to be modified

5. Click the small arrow next to "Create Snapshot" to open the sub-menu

6. Click "Edit Workload"

7. Navigate to the tab "Schedule"

8. Uncheck "Enabled"

9. Click "Update"

Using CLI

• --workloadid <workloadid> ➡️ Requires at least one workloadid, Specify an ID of the workload whose scheduler disables. Specify option multiple times to include multiple workloads. --workloadids <workloadid> --workloadids <workloadid>

Enable a schedule

Using Horizon

To disable the scheduler of a single Workload in Horizon do the following steps:

1. Login to the Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload to be modified

5. Click the small arrow next to "Create Snapshot" to open the sub-menu

6. Click "Edit Workload"

7. Navigate to the tab "Schedule"

8. check "Enabled"

9. Click "Update"

Using CLI

• --workloadid <workloadid> ➡️ Requires at least one workloadid, Specify an ID of the workload whose scheduler disables. Specify option multiple times to include multiple workloads. --workloadids <workloadid> --workloadids <workloadid>

Modify a schedule

To modify a schedule the workload itself needs to be modified.

Please follow this procedure to .

Verify the scheduler trust is working

Trilio is using the which enables the Trilio service user to act in the name of another Openstack user.

This system is used during all backup and restore features.

Using Horizon

As a trust is bound to a specific user for each Workload does the Trilio Horizon plugin show the status of the Scheduler on the Workload list page.

Using CLI

• <workload_id> ➡️ ID of the workload to validate

Unmount old mount point

The first step is to remove the datamover container and to unmount the old mounts. This is necessary to make sure, that the new datamover container with the new backend target is not getting any interference from the old backup target.

#check current mount point
[root@compute ~]# df -h
Filesystem                      Size  Used Avail Use% Mounted on
devtmpfs                        7.8G     0  7.8G   0% /dev
tmpfs                           7.8G     0  7.8G   0% /dev/shm
tmpfs                           7.8G   26M  7.8G   1% /run
tmpfs                           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/mapper/cl-root             280G   12G  269G   5% /
/dev/sda1                       976M  197M  713M  22% /boot
192.168.1.34:/mnt/tvault/42436  2.5T 1005G  1.5T  41% /var/trilio/triliovault-mounts/MTkyLjE2OC4xLjM0Oi9tbnQvdHZhdWx0LzQyNDM2

#Stop triliovault_datamover
[root@compute ~]# docker stop triliovault_datamover
triliovault_datamover
[root@compute ~]#

#Delete triliovault_datamover
[root@compute ~]# docker rm triliovault_datamover
triliovault_datamover
[root@compute ~]#

#unmount mount point
[root@compute ~]# umount /var/trilio/triliovault-mounts/MTkyLjE2OC4xLjM0Oi9tbnQvdHZhdWx0LzQyNDM2

#check mount point is unmounted successfully 
[root@compute ~]# df -h
Filesystem           Size  Used Avail Use% Mounted on
devtmpfs             7.8G     0  7.8G   0% /dev
tmpfs                7.8G     0  7.8G   0% /dev/shm
tmpfs                7.8G   26M  7.8G   1% /run
tmpfs                7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/mapper/cl-root  280G   12G  269G   5% /
/dev/sda1            976M  197M  713M  22% /boot
[root@compute ~]#

#Delete mounted dir from compute node
[root@compute trilio]# rm -rf /var/trilio/triliovault-mounts/MTkyLjE2OC4xLjM0Oi9tbnQvdHZhdWx0LzQyNDM2

Update globals.yml

Edit the globals.yml file to contain the new backup target.

Follow the installation documentation to learn about the globals.yml Trilio variables.

Deploy Trilio components with new backup target

root@controller:~# kolla-ansible -i multinode deploy

Verify the successful change in backup target

##Check that all Containers are up and running
#Controller node
root@controller:~# docker ps -a | grep trilio
583b8d42ab42        trilio/ubuntu-binary-trilio-datamover-api:4.1.36-ussuri    "dumb-init --single-…"   3 days ago          Up 3 days                               openstack-nova-api-triliodata-plugin
3be25d3819ac        trilio/ubuntu-binary-trilio-horizon-plugin:4.1.36-ussuri   "dumb-init --single-…"   4 days ago          Up 4 days                               horizon

#Compute node
root@compute:~# docker ps -a | grep trilio
bf52face23fb        trilio/ubuntu-binary-trilio-datamover:4.1.36-ussuri    "dumb-init --single-…"   3 days ago          Up 3 days                               trilio-datamover

## Verify the backup target has been changed successfully
# In case of switch to NFS
[root@compute ~]# df -h
Filesystem                      Size  Used Avail Use% Mounted on
devtmpfs                        7.8G     0  7.8G   0% /dev
tmpfs                           7.8G     0  7.8G   0% /dev/shm
tmpfs                           7.8G   26M  7.8G   1% /run
tmpfs                           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/mapper/cl-root             280G   12G  269G   5% /
/dev/sda1                       976M  197M  713M  22% /boot
192.168.1.34:/mnt/tvault/42436  2.5T 1005G  1.5T  41% /var/trilio/triliovault-mounts/MTkyLjE2OC4xLjM0Oi9tbnQvdHZhdWx0LzQyNDM2

#In case of switch to S3
[root@compute ~]# df -h
Filesystem           Size  Used Avail Use% Mounted on
devtmpfs             7.8G     0  7.8G   0% /dev
tmpfs                7.8G     0  7.8G   0% /dev/shm
tmpfs                7.8G   34M  7.8G   1% /run
tmpfs                7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/mapper/cl-root  280G   12G  269G   5% /
/dev/sda1            976M  197M  713M  22% /boot
Trilio             -     -  0.0K    - /var/trilio/triliovault-mounts

##Reverify in the triliovault_datamover containers
[root@compute ~]# docker exec -it triliovault_datamover bash
(triliovault-datamover)[nova@compute /]$ df -h
Filesystem           Size  Used Avail Use% Mounted on
overlay              280G   12G  269G   5% /
tmpfs                7.8G     0  7.8G   0% /sys/fs/cgroup
devtmpfs             7.8G     0  7.8G   0% /dev
tmpfs                7.8G     0  7.8G   0% /dev/shm
/dev/mapper/cl-root  280G   12G  269G   5% /etc/iscsi
tmpfs                6.3G     0  6.3G   0% /var/triliovault/tmpfs
Trilio             -     -  0.0K    - /var/trilio/triliovault-mounts

Reconfigure the Trilio Appliance

Follow the documentation to reconfigure the Trilio appliance with the new backup target.

Each Trilio Workload has a dedicated owner. The ownership of a Workload is defined by:

• OpenStack User - The OpenStack User-ID is assigned to a Workload

• OpenStack Project - The OpenStack Project-ID is assigned to a Workload

• OpenStack Cloud - The Trilio Serviceuser-ID is assigned to a Workload

This ownership ensures that only the owners of a Workload are able to work with it.

OpenStack Administrators can reassign Workloads or reimport Workloads from older Trilio installations.

Import workloads

Workload import allows a user to import Workloads directly from the Backup Target into the Trilio database.

To get a list of importable Workloads use the following CLI command:

workloadmgr workload-get-importworkloads-list [--project_id <project_id>]

• --project_id <project_id> ➡️ List workloads belongs to given project only.

To import Workloads into the Trilio database use the following CLI command:

workloadmgr workload-importworkloads [--workloadids <workloadid>]

• --workloadids <workloadid> ➡️ Specify workload ids to import only specified workloads. Repeat option for multiple workloads.

For every workload-importworkloads command triggered, the system will generate a jobid which will be displayed after successful workload-importworkloads command execution.

Respective import process can further be tracked using this jobid in a CLI provided starting T4O-4.3 release. User can add all workloads (which are to be imported) with a single workload-importworkloads command OR can trigger multiple such commands. For every command trigger, system will generate one unique jobid.

CLI details below:

workloadmgr workload-get-importworkloads-progres --jobid <jobId>

• --jobid <jobId> ➡️ The ID returned by workload-importworkloads CLI command.

Orphaned Workloads

The definition of an orphaned Workload is from the perspective of a specific Trilio installation. Any workload that is located on the Backup Target Storage, but not known to the TrilioVualt installation is considered orphaned.

Further is to divide between Workloads that were previously owned by Projects/Users in the same cloud or are migrated from a different cloud.

The following CLI command provides the list of orphaned workloads:

workloadmgr workload-get-orphaned-workloads-list [--migrate_cloud {True,False}]
                                                 [--generate_yaml {True,False}]

• --migrate_cloud {True,False} ➡️ Set to True if you want to list workloads from other clouds as well. Default is False.

• --generate_yaml {True,False} ➡️ Set to True if want to generate output file in yaml format, which would be further used as input for workload reassign API.

Reassigning Workloads

OpenStack administrators are able to reassign a Workload to a new owner. This involves the possibility to migrate a Workload from one cloud to another or between projects.

Use the following CLI command to reassign a Workload:

workloadmgr workload-reassign-workloads
                                        [--old_tenant_ids <old_tenant_id>]
                                        [--new_tenant_id <new_tenant_id>]
                                        [--workload_ids <workload_id>]
                                        [--user_id <user_id>]
                                        [--migrate_cloud {True,False}]
                                        [--map_file <map_file>]

• --old_tenant_ids <old_tenant_id>➡️ Specify old tenant ids from which workloads need to reassign to new tenant. Specify multiple times to choose Workloads from multiple tenants.

• --new_tenant_id <new_tenant_id> ➡️ Specify new tenant id to which workloads need to reassign from old tenant. Only one target tenant can be specified.

• --workload_ids <workload_id>➡️ Specify workload_ids which need to reassign to new tenant. If not provided then all the workloads from old tenant will get reassigned to new tenant. Specifiy multiple times for multiple workloads.

• --user_id <user_id>➡️ Specify user id to which workloads need to reassign from old tenant. only one target user can be specified.

• --migrate_cloud {True,False}➡️ Set to True if want to reassign workloads from other clouds as well. Default if False

• --map_file➡️ Provide file path(relative or absolute) including file name of reassign map file. Provide list of old workloads mapped to new tenants. Format for this file is YAML.

A sample mapping file with explanations is shown below:

reassign_mappings:
   - old_tenant_ids: [] #user can provide list of old_tenant_ids or workload_ids
     new_tenant_id: new_tenant_id
     user_id: user_id
     workload_ids: [] #user can provide list of old_tenant_ids or workload_ids
     migrate_cloud: True/False #Set to True if want to reassign workloads from
                  # other clouds as well. Default is False

   - old_tenant_ids: [] #user can provide list of old_tenant_ids or workload_ids
     new_tenant_id: new_tenant_id
     user_id: user_id
     workload_ids: [] #user can provide list of old_tenant_ids or workload_ids
     migrate_cloud: True/False #Set to True if want to reassign workloads from
                  # other clouds as well. Default is False

Definition

The file search functionality allows the user to search for files and folders located on a chosen VM in a workload in one or more Backups.

Navigating to the file search tab in Horizon

The file search tab is part of every workload overview. To reach it follow these steps:

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload a file search shall be done in

5. Click the workload name to enter the Workload overview

6. Click File Search to enter the file search tab

Configuring and starting a file search Horizon

A file search runs against a single virtual machine for a chosen subset of backups using a provided search string.

To run a file search the following elements need to be decided and configured

Choose the VM the file search shall run against

Under VM Name/ID choose the VM that the search is done upon. The drop down menu provides a list of all VMs that are part of any Snapshot in the Workload.

Set the File Path

The File Path defines the search string that is run against the chosen VM and Snapshots. This search string does support basic RegEx.

Example File Path for all files inside /etc : /etc/*

Define the Snapshots to search in

"Filter Snapshots by" is the third and last component that needs to be set. This defines which Snapshots are going to be searched.

There are 3 possibilities for a pre-filtering:

1. All Snapshots - Lists all Snapshots that contain the chosen VM from all available Snapshots

2. Last Snapshots - Choose between the last 10, 25, 50, or custom Snapshots and click Apply to get the list of the available Snapshots for the chosen VM that match the criteria.

3. Date Range - Set a start and end date and click apply to get the list of all available Snapshots for the chosen VM within the set dates.

After the pre-filtering is done all matching Snapshots are automatic prechosen. Uncheck any Snapshot that shall not be searched.

Start the File Search and retrieve the results in Horizon

To start a File Search the following elements need to be set:

• A VM to search in has to be chosen

• A valid File Path provided

• At least one Snapshot to search in selected

Once those have been set click "Search" to start the file search.

After a short time the results will be presented. The results are presented in a tabular format grouped by Snapshots and Volumes inside the Snapshot.

For each found file or folder the following information are provided:

• POSIX permissions

• Amount of links pointing to the file or folder

• User ID who owns the file or folder

• Group ID assigned to the file or folder

• Actual size in Bytes of the file or folder

• Time of creation

• Time of last modification

• Time of last access

• Full path to the found file or folder

Once the Snapshot of interest has been identified it is possible to go directly to the Snapshot using the "View Snapshot" option at the top of the table. It is also possible to directly mount the Snapshot using the "Mound Snapshot" Button at the end of the table.

Doing a CLI File Search

workloadmgr filepath-search [--snapshotids <snapshotid>]
                            [--end_filter <end_filter>]
                            [--start_filter <start_filter>]
                            [--date_from <date_from>]
                            [--date_to <date_to>]
                            <vm_id> <file_path>

• <vm_id> ➡️ ID of the VM to be searched

• <file_path> ➡️ Path of the file to search for

• --snapshotids <snapshotid> ➡️ Search only in specified snapshot ids snapshot-id: include the instance with this UUID

• --end_filter <end_filter> ➡️Displays last snapshots, example , last 10 snapshots, default 0 means displays all snapshots

• --start_filter <start_filter> ➡️Displays snapshots starting from , example , snapshot starting from 5, default 0 means starts from first snapshot

• --date_from <date_from> ➡️ From date in format 'YYYY-MM-DDTHH:MM:SS' eg 2016-10-10T00:00:00, If time isn't specified then it takes 00:00 by default

• --date_to <date_to> ➡️ To date in format 'YYYY-MM-DDTHH:MM:SS'(defult is current day),Specify HH:MM:SS to get snapshots within same day inclusive/exclusive results for date_from and date_to

Trilio enables Openstack administrators to set Project Quotas against the usage of Trilio.

The following Quotas can be set:

• Number of Workloads a Project is allowed to have

• Number of Snapshots a Project is allowed to have

• Number of VMs a Project is allowed to protect

• Amount of Storage a Project is allowed to use on the Backup Target

Work with Workload Quotas via Horizon

Workload Quotas are managed like any other Project Quotas.

1. Login into Horizon as user with admin role

2. Navigate to Identity

3. Navigate to Projects

4. Identify the Project to modify or show the quotas on

5. Use the small arrow next to "Manage Members" to open the submenu

6. Choose "Modify Quotas"

7. Navigate to "Workload Manager"

8. Edit Quotas as desired

9. Click "Save"

Work with Workload Quotas via CLI

List available Quota Types

Trilio is providing several different Quotas. The following command allows listing those.

workloadmgr project-quota-type-list

Show Quota Type Details

The following command will show the details of a provided Quota Type.

workloadmgr project-quota-type-show <quota_type_id>

• <quota_type_id> ➡️ID of the Quota Type to show

Create a Quota

The following command will create a Quota for a given project and set the provided value.

workloadmgr project-allowed-quota-create --quota-type-id quota_type_id
                                         --allowed-value allowed_value 
                                         --high-watermark high_watermark 
                                         --project-id project_id

• <quota_type_id> ➡️ID of the Quota Type to be created

• <allowed_value>➡️ Value to set for this Quota Type

• <high_watermark>➡️ Value to set for High Watermark warnings

• <project_id>➡️ Project to assign the quota to

List allowed Quotas

The following command lists all Trilio Quotas set for a given project.

workloadmgr project-allowed-quota-list <project_id>

• <project_id>➡️ Project to list the Quotas from

Show allowed Quota

The following command shows the details about a provided allowed Quota.

workloadmgr project-allowed-quota-show <allowed_quota_id>

• <allowed_quota_id> ➡️ID of the allowed Quota to show.

Update allowed Quota

The following command shows how to update the value of an already existing allowed Quota.

workloadmgr project-allowed-quota-update [--allowed-value <allowed_value>]
                                         [--high-watermark <high_watermark>]
                                         [--project-id <project_id>]
                                         <allowed_quota_id>

• <allowed_value>➡️ Value to set for this Quota Type

• <high_watermark>➡️ Value to set for High Watermark warnings

• <project_id>➡️ Project to assign the quota to

• <allowed_quota_id> ➡️ID of the allowed Quota to update

Delete allowed Quota

The following command will delete an allowed Quota and sets the value of the connected Quota Type back to unlimited for the affected project.

workloadmgr project-allowed-quota-delete <allowed_quota_id>

• <allowed_quota_id> ➡️ID of the allowed Quota to delete

Generic Prerequisites

• Please ensure to complete the upgrade of all the Trilio components on the Openstack controller & compute nodes before starting the rolling upgrade of Trilio.

• The mentioned Gemfury repository should be accessible from Trilio VM.

• Either 4.2 GA OR any hotfix patch against 4.2 should be already deployed for performing upgrades mentioned in the current document.

• Please ensure the following points before starting the upgrade process:

  • No snapshot OR restore to be running.

  • Global job-scheduler should be disabled.

  • wlm-cron should be disabled and any lingering process should be killed.

Deactivating the wlm-cron service

The following sets of commands will disable the wlm-cron service and verify that it is has been completly shut-down.

Steps to upgrade Trilio having Internet access

1. Download the hf_upgrade.sh script on all T4O nodes where the upgrade is to be done.

You can check the usage of this script .

1. Run the following command to download and Install the upgraded packages

1. Run the following command to enable the wlm-cron service

Steps to upgrade Trilio having No Internet access

Here the packages need to be downloaded on a separate host which has internet access. The script ./hf_upgrade.sh can be used with the below-mentioned option to download the required package

1. Download the hf_upgrade.sh script and use the script to download the upgraded packages. You can check the usage of this script .\

2. Copy the downloaded 4.3-offlinePkgs.tar.gz package and hf_upgrade.sh script to all the Trilio nodes\

3. Now access the appliance VMs and install the copied offline packages on each of the Trilio appliances by running the below-mentioned command:\

4. Once the installation is done, run the following command to enable the wlm-cron service\

Post upgrade steps

1. Verify all wlm services on all Trilio nodes

2. Make sure all wlm services are up and running from python version 3.8.x

3. Check the mount point using “df -h” command

4. Grafana dashboard shows the correct wlm service status on all T4O nodes.

5. Enable Global Job Scheduler

   ---

Additional check for wlm-cron on the primary node

[RHOSP, TripleO and Kolla only] Verify nova UID/GID for nova user on the Appliance

Red Hat OpenStack, TripleO, and Kolla Ansible Openstack are using the nova UID/GID of 42436 inside their containers instead of 162:162 which is the standard in other Openstack environments.

Please verify that the nova UID/GID on the Trilio Appliance is still 42436, to do so follow the below document provided here:

Introduction

Integrating Trilio for OpenStack 4.2 with Barbican facilitates encryption support for the qcow2-data segment of Trilio backups. However, the JSON files housing the backed-up OpenStack metadata remain unencrypted.

This capability necessitates the presence of the OpenStack Barbican service. Absence of the Barbican service will result in the omission of encryption options for Workloads within the Horizon interface.

Trilio for OpenStack (T4O) 4.2 exclusively retrieves secrets from Barbican, without generating, modifying, or deleting any secrets within the Barbican platform.

Barbican secrets are indispensable for executing backups or restorations in encryption-enabled Workloads. The onus lies on the OpenStack project user to supply these secrets and guarantee their accurate availability.

In order to employ encrypted Workloads, the Trilio trustee role must be capable of interacting with the Barbican service to access and retrieve secrets from Barbican. By default, only the 'admin' and 'creator' roles are endowed with these permissions.

Encryption availability for a Workload is confined to the Workload creation stage. Post-creation, the encryption status of a Workload is irreversible; it cannot be altered or toggled.

The following secret configurations are supported for AES-256

By default Barbican will generate secrets of the following type:

• Alghorithm: AES-256 (All supported types utilize this algorithm)

• Mode: cbc

• content type: application/octet-stream

• payload-content-encoding: base64

• secret type: opaque

• payload: plaintext

Prerequisite

For encrypted workload Barbican should enabled on Openstack, then while configuring TVO-4.2, trustee role should be kept as creator .

Additionally every user who’ll be interacting with TrilioVault for any operator, should also have creator role assigned.

1. Secret creation

Secret can be created via OpenStack cli with following the below steps

1. Source the rc file of the user who is going to create the encrypted workload.

2. Create any supported type of secret and fetch the secret UUID using OpenStack secret cli.

   1. Refer to this guide for more information about

   2. Below is the example of symmetric secret type creation & fetch the secret UUID.

      1. Generate a new 256-bit key using order create

()[root@overcloudtrain1-controller-0 /]# openstack secret order create --name secret2 --algorithm aes --mode ctr --bit-length 256 --payload-content-type=application/octet-stream key +----------------+-------------------------------------------------------------------------------------------+ | Field | Value | +----------------+-------------------------------------------------------------------------------------------+ | Order href | https://overcloudtrain1.trilio.local:13311/v1/orders/641bac2c-b5b2-4a7a-9172-8fe0cf55425d | | Type | Key | | Container href | N/A | | Secret href | None | | Created | None | | Status | None | | Error code | None | | Error message | None | +----------------+-------------------------------------------------------------------------------------------+

b. View the details of the order to identify the location of the generated key, shown here as the Secret href value: ()[root@overcloudtrain1-controller-0 /]# openstack secret order get https://overcloudtrain1.trilio.local:13311/v1/orders/641bac2c-b5b2-4a7a-9172-8fe0cf55425d +----------------+--------------------------------------------------------------------------------------------+ | Field | Value | +----------------+--------------------------------------------------------------------------------------------+ | Order href | https://overcloudtrain1.trilio.local:13311/v1/orders/641bac2c-b5b2-4a7a-9172-8fe0cf55425d | | Type | Key | | Container href | N/A | | Secret href | https://overcloudtrain1.trilio.local:13311/v1/secrets/f2c96fe2-6ae7-4985-b98c-e571ba05403c | | Created | 2023-07-12T11:38:17+00:00 | | Status | ACTIVE | | Error code | None | | Error message | None | +----------------+--------------------------------------------------------------------------------------------+

c. Fetch the secret UUID via below command (use Secret href)

()[root@overcloudtrain1-controller-0 /]# openstack secret get https://overcloudtrain1.trilio.local:13311/v1/secrets/f2c96fe2-6ae7-4985-b98c-e571ba05403c +---------------+--------------------------------------------------------------------------------------------+ | Field | Value | +---------------+--------------------------------------------------------------------------------------------+ | Secret href | https://overcloudtrain1.trilio.local:13311/v1/secrets/f2c96fe2-6ae7-4985-b98c-e571ba05403c | | Name | secret2 | | Created | 2023-07-12T11:38:17+00:00 | | Status | ACTIVE | | Content types | {'default': 'application/octet-stream'} | | Algorithm | aes | | Bit length | 256 | | Secret type | symmetric | | Mode | ctr | | Expiration | None | +---------------+--------------------------------------------------------------------------------------------+ ()[root@overcloudtrain1-controller-0 /]#

Note down the last value from Secret href URL which is UUID (8-4-4-4-12 format).

For example secret UUID for the Secret href URL https://overcloudtrain1.trilio.local:13311/v1/secrets/f2c96fe2-6ae7-4985-b98c-e571ba05403c will be f2c96fe2-6ae7-4985-b98c-e571ba05403c

This UUID will be used further for creating the encrypted workload.

2. Workload creation

1. Login to openstack horizon dashboard with user who has created the secret & go to Project->Backup->workloads

2. click on create workload.

3. select the checkbox for Enable Encryption

4. Provide the UUID noted in above steps in Secret UUID text box

1. Follow the usual procedure for further tabs (Workload member, Schedule, policy & Option) & click on create.

2. Workload will be created and value for Encryption field will be True.

Encrypted workload migration

For migration of encrypted workload please follow:

Upgrade from older release to 4.2

For upgrade from older release to 4.2 please follow:

For Trilio configuration please follow:

To configure the banner shown upon accessing the Trilio Appliance GUI do the following.

1. Login into Trilio Appliance console

2. edit the banner.yaml at /etc/tvault-config/banner.yaml

3. restart tvault-config service

The content of the banner.yaml looks as follows and can be edited as required:

Clean Trilio Datamover API service

The following steps need to be run on all nodes, which have the Trilio Datamover API service running. Those nodes can be identified by checking the roles_data.yaml for the role that contains the entry OS::TripleO::Services::TrilioDatamoverApi.

Once the role that runs the Trilio Datamover API service has been identified will the following commands clean the nodes from the service.

Stop trilio_dmapi container.

# For RHOSP13
systemctl disable tripleo_trilio_dmapi.service
systemctl stop tripleo_trilio_dmapi.service
docker stop trilio_dmapi

# For RHOSP16 onwards
systemctl disable tripleo_trilio_dmapi.service
systemctl stop tripleo_trilio_dmapi.service
podman stop trilio_dmapi

Remove trilio_dmapi container.

# For RHOSP13
docker rm trilio_dmapi
docker rm trilio_datamover_api_init_log
docker rm trilio_datamover_api_db_sync

# For RHOSP16 onwards
podman rm trilio_dmapi
podman rm trilio_datamover_api_init_log
podman rm trilio_datamover_api_db_sync

## If present, remove below container as well
podman rm container-puppet-triliodmapi

Clean Trilio Datamover API service conf directory.

rm -rf /var/lib/config-data/puppet-generated/triliodmapi
rm /var/lib/config-data/puppet-generated/triliodmapi.md5sum
rm -rf /var/lib/config-data/triliodmapi*

Clean Trilio Datamover API service log directory.

rm -rf /var/log/containers/trilio-datamover-api/

Clean Trilio Datamover Service

The following steps need to be run on all nodes, which have the Trilio Datamover service running. Those nodes can be identified by checking the roles_data.yaml for the role that contains the entry OS::TripleO::Services::TrilioDatamover.

Once the role that runs the Trilio Datamover service has been identified will the following commands clean the nodes from the service.

Stop trilio_datamover container.

# For RHOSP13
docker stop trilio_datamover

# For RHOSP16 onwards
systemctl disable tripleo_trilio_datamover.service
systemctl stop tripleo_trilio_datamover.service
podman stop trilio_datamover

Remove trilio_datamover container.

# For RHOSP13
docker rm trilio_datamover

# For RHOSP16 onwards
podman rm trilio_datamover

## If present, remove below container as well
podman rm container-puppet-triliodmapi

Unmount Trilio Backup Target on compute host.

## Following steps applicable for all supported RHOSP releases.

# Check triliovault backup target mount point
mount | grep trilio

# Unmount it
-- If it's NFS	(COPY UUID_DIR from your compute host using above command)
umount /var/lib/nova/triliovault-mounts/<UUID_DIR>

-- If it's S3
umount /var/lib/nova/triliovault-mounts

# Verify that it's unmounted		
mount | grep trilio
	
df -h  | grep trilio

# Remove mount point directory after verifying that backup target unmounted successfully.
# Otherwise actual data from backup target may get cleaned.	

rm -rf /var/lib/nova/triliovault-mounts

Clean Trilio Datamover service conf directory.

rm -rf /var/lib/config-data/puppet-generated/triliodm/
rm /var/lib/config-data/puppet-generated/triliodm.md5sum
rm -rf /var/lib/config-data/triliodm*

Clean log directory of Trilio Datamover service.

rm -rf /var/log/containers/trilio-datamover/

Clean Trilio haproxy resources

The following steps need to be run on all nodes, which have the haproxy service running. Those nodes can be identified by checking the roles_data.yaml for the role that contains the entry OS::TripleO::Services::HAproxy.

Once the role that runs the haproxy service has been identified will the following commands clean the nodes from all Trilio resources.

Edit the following file inside the haproxy container and remove all Trilio entries.

/var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg

An example of these entries is given below.

listen trilio_datamover_api
  bind 172.25.3.60:13784 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 172.25.3.60:8784 transparent
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  http-request set-header X-Forwarded-Port %[dst_port]
  option httpchk
  option httplog
  server overcloud-controller-0.internalapi.localdomain 172.25.3.59:8784 check fall 5 inter 2000 rise 2

Restart the haproxy container once all edits have been done.

# For RHOSP13
docker restart haproxy-bundle-docker-0

# For RHOSP16 onwards
podman restart haproxy-bundle-podman-0

Clean Trilio Keystone resources

Trilio registers services and users in Keystone. Those need to be unregistered and deleted.

openstack service delete dmapi
openstack user delete dmapi

Clean Trilio database resources

Trilio creates a database for the dmapi service. This database needs to be cleaned.

Login into the database cluster

## On RHOSP13, run following command on node where database service runs
docker exec -ti -u root galera-bundle-docker-0 mysql -u root

## On RHOSP16
podman exec -it galera-bundle-podman-0 mysql -u root

Run the following SQL statements to clean the database.

## Clean database
DROP DATABASE dmapi;

## Clean dmapi user
=> List 'dmapi' user accounts
MariaDB [mysql]> select user, host from mysql.user where user='dmapi';
+-------+-------------+
| user  | host        |
+-------+-------------+
| dmapi | 172.25.2.10 |
| dmapi | 172.25.2.8  |
+-------+-------------+
2 rows in set (0.00 sec)

=> Delete those user accounts
MariaDB [mysql]> DROP USER [email protected];
Query OK, 0 rows affected (0.82 sec)

MariaDB [mysql]> DROP USER [email protected];
Query OK, 0 rows affected (0.05 sec)

=> Verify that dmapi user got cleaned
MariaDB [mysql]> select user, host from mysql.user where user='dmapi';
Empty set (0.00 sec)

Revert overcloud deploy command

Remove the following entries from roles_data.yaml used in the overcloud deploy command.

• OS::TripleO::Services::TrilioDatamoverApi

• OS::TripleO::Services::TrilioDatamover

Follow these steps to clean the overcloud deploy command from all Trilio entries.

1. Remove trilio_env.yaml entry

2. Remove trilio endpoint map file Replace with original map file if existing

Revert back to original RHOSP Horizon container

Run the cleaned overcloud deploy command.

Destroy the Trilio VM Cluster

List all VMs running on the KVM node

virsh list

Destroy the Trilio VMs

virsh destroy <Trilio VM Name or ID>

Undefine the Trilio VMs

virsh undefine <Trilio VM name>

Delete the TrlioVault VM disk from KVM Host storage

Trilio’s tenant driven backup service gives tenants control over backup policies. However, sometimes it may be too much control to tenants and the cloud admins may want to limit what policies are allowed by tenants. For example, a tenant may become overzealous and only uses full backups every 1 hr interval. If every tenant were to pursue this backup policy, it puts a severe strain on cloud infrastructure. Instead, if cloud admin can define predefined backup policies and each tenant is only limited to those policies then cloud administrators can exert better control over backup service.

Workload policy is similar to nova flavor where a tenant cannot create arbitrary instances. Instead, each tenant is only allowed to use the nova flavors published by the admin.

List and showing available Workload policies

Using Horizon

To see all available Workload policies in Horizon follow these steps:

1. Login to Horizon using admin user.

2. Click on Admin Tab.

3. Navigate to Backups-Admin

4. Navigate to Trilio

5. Navigate to Policy

The following information are shown in the policy tab for each available policy:

• Creation time

• name

• description

• status

• set interval

• set retention type

• set retention value

Using CLI

workloadmgr policy-list

workloadmgr policy-show <policy_id>

• <policy_id>➡️ Id of the policy to show

Create a policy

Using Horizon

To create a policy in Horizon follow these steps:

1. Login to Horizon using admin user.

2. Click on Admin Tab.

3. Navigate to Backups-Admin

4. Navigate to Trilio

5. Navigate to Policy

6. Click new policy

7. provide a policy name on the Details tab

8. provide a description on the Details tab

9. provide the RPO in the Policy tab

10. Choose the Snapshot Retention Type

11. provide the Retention value

12. Choose the Full Backup Interval

13. Click create

Using CLI

workloadmgr policy-create --policy-fields <key=key-name>
                          [--display-description <display_description>]
                          [--metadata <key=key-name>]
                          <display_name>

• --policy-fields <key=key-name> ➡️ Specify following key value pairs for policy fields Specify option multiple times to include multiple keys. 'interval' : '1 hr' 'retention_policy_type' : 'Number of Snapshots to Keep' or 'Number of days to retain Snapshots' 'retention_policy_value' : '30' 'fullbackup_interval' : '-1' (Enter Number of incremental snapshots to take Full Backup between 1 to 999, '-1' for 'NEVER' and '0' for 'ALWAYS')For example --policy-fields interval='1 hr' --policy-fields retention_policy_type='Number of Snapshots to Keep '--policy-fields retention_policy_value='30' --policy- fields fullbackup_interval='2'

• --display-description <display_description> ➡️ Optional policy description. (Default=No description)

• --metadata <key=keyname> ➡️ Specify a key value pairs to include in the workload_type metadata Specify option multiple times to include multiple keys. key=value

• <display_name> ➡️ the name the policy will get

Edit a policy

Using Horizon

To edit a policy in Horizon follow these steps:

1. Login to Horizon using admin user.

2. Click on Admin Tab.

3. Navigate to Backups-Admin

4. Navigate to Trilio

5. Navigate to Policy

6. identify the policy to edit

7. click on "Edit policy" at the end of the line of the chosen policy

8. edit the policy as desired - all values can be changed

9. Click "Update"

Using CLI

workloadmgr policy-update [--display-name <display-name>]
                          [--display-description <display-description>]
                          [--policy-fields <key=key-name>]
                          [--metadata <key=key-name>]
                          <policy_id>

• --display-name <display-name>➡️Name of the policy

• --display-description <display_description> ➡️ Optional policy description. (Default=No description)

• --policy-fields <key=key-name> ➡️ Specify following key value pairs for policy fields Specify option multiple times to include multiple keys. 'interval' : '1 hr' 'retention_policy_type' : 'Number of Snapshots to Keep' or 'Number of days to retain Snapshots' 'retention_policy_value' : '30' 'fullbackup_interval' : '-1' (Enter Number of incremental snapshots to take Full Backup between 1 to 999, '-1' for 'NEVER' and '0' for 'ALWAYS')For example --policy-fields interval='1 hr' --policy-fields retention_policy_type='Number of Snapshots to Keep '--policy-fields retention_policy_value='30' --policy- fields fullbackup_interval='2'

• --metadata <key=keyname> ➡️ Specify a key value pairs to include in the workload_type metadata Specify option multiple times to include multiple keys. key=value

• <policy_id> ➡️ the name the policy will get

Assign/Remove a policy

Using Horizon

To assign or remove a policy in Horizon follow these steps:

1. Login to Horizon using admin user.

2. Click on Admin Tab.

3. Navigate to Backups-Admin

4. Navigate to Trilio

5. Navigate to Policy

6. identify the policy to assign/remove

7. click on the small arrow at the end of the line of the chosen policy to open the submenu

8. click "Add/Remove Projects"

9. Choose projects to add or remove by using the plus/minus buttons

10. Click "Apply"

Using CLI

workloadmgr policy-assign [--add_project <project_id>]
                          [--remove_project <project_id>]
                          <policy_id>

• --add_project <project_id> ➡️ ID of the project to assign policy to. Use multiple times to assign multiple projects.

• --remove_project <project_id> ➡️ ID of the project to remove policy from. Use multiple times to remove multiple projects.

• <policy_id>➡️policy to be assigned or removed

Delete a policy

Using Horizon

To delete a policy in Horizon follow these steps:

1. Login to Horizon using admin user.

2. Click on Admin Tab.

3. Navigate to Backups-Admin

4. Navigate to Trilio

5. Navigate to Policy

6. identify the policy to assign/remove

7. click on the small arrow at the end of the line of the chosen policy to open the submenu

8. click "Delete Policy"

9. Confirm by clicking "Delete"

Using CLI

workloadmgr policy-delete <policy_id>

• <policy_id> ➡️ID of the policy to be deleted

Start File Search

POST https://$(tvm_address):8780/v1/$(tenant_id)/search

Starts a File Search with the given parameters

Path Parameters

Headers

HTTP/1.1 200 OK
Server: nginx/1.16.1
Date: Mon, 09 Nov 2020 13:23:25 GMT
Content-Type: application/json
Content-Length: 244
Connection: keep-alive
X-Compute-Request-Id: req-bdfd3fb8-5cbf-4108-885f-63160426b2fa

{
   "file_search":{
      "created_at":"2020-11-09T13:23:25.698534",
      "updated_at":null,
      "id":14,
      "deleted_at":null,
      "status":"executing",
      "error_msg":null,
      "filepath":"/etc/h*",
      "json_resp":null,
      "vm_id":"08dab61c-6efd-44d3-a9ed-8e789d338c1b"
   }
}

Body format

{
   "file_search":{
      "start":<Integer>,
      "end":<Integer>,
      "filepath":"<Reg-Ex String>",
      "date_from":<Date Format: YYYY-MM-DDTHH:MM:SS>,
      "date_to":<Date Format: YYYY-MM-DDTHH:MM:SS>,
      "snapshot_ids":[
         "<Snapshot-ID>"
      ],
      "vm_id":"<VM-ID>"
   }
}

Get File Search Results

POST https://$(tvm_address):8780/v1/$(tenant_id)/search/<search_id>

Starts a filesearch with the given parameters

Path Parameters

Headers

HTTP/1.1 200 OK
Server: nginx/1.16.1
Date: Mon, 09 Nov 2020 13:24:28 GMT
Content-Type: application/json
Content-Length: 819
Connection: keep-alive
X-Compute-Request-Id: req-d57bea9a-9968-4357-8743-e0b906466063

{
   "file_search":{
      "created_at":"2020-11-09T13:23:25.000000",
      "updated_at":"2020-11-09T13:23:48.000000",
      "id":14,
      "deleted_at":null,
      "status":"completed",
      "error_msg":null,
      "filepath":"/etc/h*",
      "json_resp":"[
                      {
                         "ed4f29e8-7544-4e1c-af8a-a76031211926":[
                            {
                               "/dev/vda1":[
                                  "/etc/hostname",
                                  "/etc/hosts"
                               ],
                               "/etc/hostname":{
                                  "dev":"2049",
                                  "ino":"32",
                                  "mode":"33204",
                                  "nlink":"1",
                                  "uid":"0",
                                  "gid":"0",
                                  "rdev":"0",
                                  "size":"1",
                                  "blksize":"1024",
                                  "blocks":"2",
                                  "atime":"1603455255",
                                  "mtime":"1603455255",
                                  "ctime":"1603455255"
                               },
                               "/etc/hosts":{
                                  "dev":"2049",
                                  "ino":"127",
                                  "mode":"33204",
                                  "nlink":"1",
                                  "uid":"0",
                                  "gid":"0",
                                  "rdev":"0",
                                  "size":"37",
                                  "blksize":"1024",
                                  "blocks":"2",
                                  "atime":"1603455257",
                                  "mtime":"1431011050",
                                  "ctime":"1431017172"
                               }
                            }
                         ]
                      }
                  ]",
      "vm_id":"08dab61c-6efd-44d3-a9ed-8e789d338c1b"
   }
}