Trilio for RHV, is a native RHV service that provides policy-based comprehensive backup and recovery for RHV workloads. The solution captures point-in-time workloads (Application, OS, Compute, Network, Configurations, Data, and Metadata of an environment) as full or incremental snapshots. A variety of storage environments can hold these Snapshots, including NFS and soon AWS S3 compatible storage. With TrilioVault and its single-click recovery, organizations can improve Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO). TrilioVault enables IT departments to fully deploy RHV solutions and provide business assurance through enhanced data retention, protection, and integrity.

With the use of Trilio’s VAST (Virtual Snapshot Technology), Enterprise IT and Cloud Service Providers can now deploy backup and disaster recovery as a service to prevent data loss or data corruption through point-in-time snapshots and seamless one-click recovery. Trilio takes point-in-time backup of the entire workload consisting of computing resources, network configurations, and storage data as one unit. It also takes incremental backups that only capture the changes made since the last backup. Incremental snapshots save time and storage space as the backup only includes changes since the last backup. The summarized benefits of using VAST for backup and restore are:

1. Efficient capture and storage of snapshots. Since our full backups only include data that is committed to storage volume and the incremental backups include changed blocks of data since the last backup, our backup processes are efficient and storages backup images efficiently on the backup media.

2. Faster and reliable recovery. When your applications become complex that snap multiple VMs and storage volumes, our efficient recovery process brings your application from zero to operational with the click of a button.

3. Reliable and smooth migration of workloads between environments. Trilio captures all the details of your application, and hence our migration includes your entire application stack without leaving anything for guesswork.

4. Through policy and automation, lower the Total Cost of Ownership. Our role driven backup process and automation eliminates the need for dedicated backup administrators, thereby improves your total cost of ownership.

TrilioVault for RHV 4.1 provides several Bug Fixes and the support of a new platform, Ovirt.

New supported Platform: OVirt

OVirt is the Upstream project of Red Hat Virtualization.

Trilio is happy to announce the full support for OVirt starting version 4.4.

The installation, configuration, and usage of TrilioVault inside OVirt are exactly the same as for RHV.

TrilioVault data transfer related logs

/var/log/ovirt_celery/worker<x>.log

The worker logs contain the status of the disk transfer from the RHV Host to the backup target. Useful if the data transfer process gets stuck or errors in between.

/var/log/ovirt-imageio-daemon/daemon.log

The daemon.log contains all informations about the actual connection between the RHV Host and the backup target. Useful to identify potential connection issues between the RHV Host and Backup target.

TrilioVault data transfer related logs

/var/log/ovirt-imageio-proxyimage-proxy.log

This log provides the used ticket number and RHV-Host for a backup transfer. It is helpful to identify the ticket numbers that are used in RHV to track a specific data transfer. It also shows any connection errors between the RHV-M and the RHV-Host.

Generally helpful RHV-Manager logs

TrilioVault is calling a lot of RHV APIs to read metadata, create RHV Snapshots and restore VMs. These tasks are done by RHV independently from TrilioVault and are logged in RHV logs.

/var/log/ovirt-engine/engine.log

This log is hard to read, but contains all tasks that the RHV-M is doing, including all Snapshot related tasks.

Additional logs that can be helpful during troubleshooting are:

/var/log/ovirt-engine/boot.log /var/log/ovirt-engine/console.log /var/log/ovirt-engine/ui.log

In case of the Trilio Dashboard being lost it can be resetted as long as SSH access to the appliance is available.

To reset the password to its default do the following:

[root@tvm ~]# source /home/rhv/myansible/bin/activate
(myansible) [root@tvm ~]# cd /opt/stack/workloadmgr/workloadmgr/tvault-config
(myansible) [root@tvm tvault-config]# python recreate_conf.py
(myansible) [root@tvm tvault-config]# systemctl restart tvault-config

The dashboard login will be reset to:

Username: admin
Password: password

The Admin Panel gives Adminstrators a centralized overview of:

• Snapshots

• Storage usage

• protected vs unprotected VMs

In addition does it contain the functionalities to control:

• Enable/Disable Global Job Scheduler

• Enable/Disable email alerts

• Update email alert email list

Accessing the Admin Panel

To access the Admin Panel the following steps need to be followed:

1. Login into RHV-Manager as a user with admin privileges

2. In the main menu click on Backup

3. In the Backup sub-menu click on Admin Panel

Admin Panel Overview tab

The Overview tab of the Admin Panel shows the following information:

• Graphical overview of how many Snapshots are available, errored or canceled

• Percentage of Snapshots that are Full Snapshots

• Graphical overview of used versus free Storage capacity

Admin Panel Enable/Disable Global Job Scheduler

To enable or disable the Global Job Scheduler follow these steps:

1. Login into RHV-Manager as a user with admin privileges

2. In the main menu click on Backup

3. In the Backup sub-menu click on Admin Panel

Uninstalling TrilioVault is done in 2 easy steps, which leave only the already created backups behind.

Step 1: Uninstall RHV ovirt-imageio extension

To uninstall the ovirt-imageio extension do the following:

To upgrade Trilio it is necessary to uninstall the old version of Trilio and install the new version.

This is done in a few easy steps.

Trilio 4.0 is the fourth release of Trilio for Red Hat Virtualization.

It is aimed to provide Backup and Recovery for Red Hat Virtualization 4.3.x & 4.4.x. The full requirements can be found .

Trilio for RHV 4.0 Features on a Glance

TrilioVault for RHV contains the possibility to send E-Mails to a defined list of E-Mail addresses for any succeeded or failed backup or recovery process.

Enabling the email alert function

To enable the email alert function the following steps need to be done:

TrilioVault for RHV is integrating into the RHV-Manager to provide a seamless experience for RHV Administrators and Users for all their Backup & Recovery needs inside RHV.

For this purpose is TrilioVault extending the RHV-Manager GUI with a new tab "Backup", which contains the sub-tabs Workloads, Admin Panel and Reporting as shown in figure 1.

TrilioVault Appliance logs used during configuration

The following logs contain all information gathered during the configuration of the TrilioVault Appliance

/var/log/workloadmgr/tvault-config.log

This log contains all information of pre-checks done, when filling out the configurator form.

TrilioVault for RHV 4.0 SP2 provides support for image-io 2.1 and several bug fixes. Further, have the names of the playbooks to install the TrilioVault components on RHV-Hosts and RHV-Managers to more reasonable names.

TrilioVault is a pure software solution and is composed of 4 elements:

1. TrilioVault Appliance (Virtual Machine)

2. TrilioVault RHV-M Web-GUI extension

3. TrilioVault ovirt-imageio-proxy extension

4. TrilioVault ovirt-imageio-daemon extension

System requirements TrilioVault Appliance

The TrilioVault Appliance gets delivered as a qcow2 image, which gets attached to a virtual machine.

Trilio supports only KVM based hypervisors and recommends to use the RHV Cluster as the hoster for the TrilioVault Appliance.

The recommended size of the VM for the TrilioVault Appliance is:

The qcow2 image itself defines the 40GB disk size of the VM.

System requirements TrilioVault ovirt-imageio extension

TrilioVault is extending the ovirt-imageio-proxy service running on the RHV-Manager and ovirt-imageio-daemon running on the RHV-Hosts.

These extensions do not have any hardware related requirements, but they require specific versions of the ovirt-imageio services.

Please check the for further information.

Definition

The file search functionality allows the user to search for files and folders located on a chosen VM in a workload in one or more Backups.

Navigating to the file search tab

The file search tab is part of every workload overview. To reach it follow these steps:

1. Login to the RHV-Manager

2. Navigate to the Backup Tab

3. Identify the workload a file search shall be done in

Configuring and starting a file search

A file search runs against a single virtual machine for a chosen subset of backups using a provided search string.

To run a file search the following elements need to be decided and configured

Choose the VM the file search shall run against

Under VM Name/ID choose the VM that the search is done upon. The drop down menu provides a list of all VMs that are part of any Snapshot in the Workload.

Set the File Path

The File Path defines the search string that is run against the chosen VM and Snapshots. This search string does support basic RegEx.

Example File Path for all files inside /etc : /etc/*

Define the Snapshots to search in

Filter Snapshots by is the third and last component that needs to be set. This defines which Snapshots are going to be searched.

There are 3 possibilities for a pre-filtering:

1. All Snapshots - Lists all Snapshots that contain the chosen VM from all available Snapshots

2. Last Snapshots - Choose between last 10, 25, 50, or custom Snapshots and click Apply to get the list of the available Snapshots for the chosen VM that match the criteria.

3. Date Range - Set a start and end date and click apply to get the list of all available Snapshots for the chosen VM within the set dates.

After the pre-filtering is done choose the Snapshots that shall be searched by clicking their checkbox or by clicking the global checkbox.

Start the File Search and retrieve the results

To start a File Search the following elements need to be set:

• A VM to search in has to be choosen

• A valid File Path provided

• At least one Snapshot to search in selected

Once those have been set click "Search" to start the file search.

After a short time the results will be presented. The results are presented in a tabular format grouped by Snapshots and Volumes inside the Snapshot.

For each found file or folder the following information are provided:

• POSIX permissions

• Amount of links pointing to the file or folder

• User ID who owns the file or folder

Once the Snapshot of interest has been identified it is possible to go directly to the Snapshot using the "View Snapshot" option.

TrilioVault extends the ovirt-imageio services running on the RHV-Manager and the RHV hosts, to provide the parallel download of disks from multiple RHV hosts.

The imageio extensions are getting installed automatically using Ansible playbooks provided on the TrilioVault Appliance.

Trilio is built on the same architectural principles as modern analytical platforms such as Hadoop and other big data platforms. These platforms offer infinite scale without compromising on performance. Trilio's other attributes include agentless, natively integrated with RHV GUI, horizontally scalable, nondisruptive, and open universal backup schema.

Agentless

Trilio offers image-level backup, which backs up a given virtual machine physical disks as one file. Irrespective of the complexity of a VM or the applications running inside the VM, the user does not require any custom code, called agents, inside VM for TrilioVault to take VM backups. Agentless solutions are one of the highly desirable features because any solution that requires custom code to run in VM creates an operational nightmare.

Self-Service/UI Integrated

Trilio comes with a GUI plugin for RHV Manager, which provides seamless integration of Trilio functionality adjacent to virtual machines management. Trilio service authenticates users with OpenID tokens, so any user who logged in to RHV can use Trilio functionality without any out-of-band user management.

Scalable, Linear, Infinite Scale

Most backup solutions are built on client/server architectures, and hence, they invariably create performance and scale bottlenecks when the RHV cluster grows. Traditional backup solutions require constant tweaking to keep up with RHV cluster growth. Trilio is built on the same architectural principles as the RHV platform; hence, it grows with the RHV cluster without introducing scale and performance bottlenecks.

Nondisruptive

Deploying Trilio is nondisruptive to the RHV cluster or the virtual machines. Similarly, uninstalling Trilio is nondisruptive as well.

Open Universal Backup Schema

In the current world of multi-cloud environments, the backup images must be platform and vendor-neutral, so they are easily portable between clouds. Trilio saves backup images as QCOW2 images. QCOW2 is a standard format in KVM/RHV environments for virtual disks, and Linux comes with numerous tools to create and manage them. A backup image stored in QCOW2 format gives the user enormous flexibility on how these are leveraged for various use cases, including restoring backup images without TVM.

QCOW2 images also come with two important attributes that also make them ideal for storing backup images.

1. QCOW2s are sparse friendly. As a regular practice, users overprovision virtual disks to VMs. These virtual disks may be thick or thin-provisioned, but at any given time, applications only use a fraction of the virtual disk capacity. When taking image-level backups of virtual disks, backup solutions should only save the blocks that are allocated and not save blocks that are not allocated or used. For example, your virtual disks maybe a 1TB in capacity, but the applications utilized only 10GB of disk space. Since QCOW2 images are spare friendly, Trilio only stores the data. In the above example, the size of the QCOW2 image is 10GB

2. KVM/RHV supports virtual disk snapshots by a construct called overlay files. KVM/RHV creates a new disk snapshot, it creates a new qcow2 file called overlay file and is overlaid on the original qcow2 file. Any new writes are applied to overlay file, and any reads to old data is read from the older qcow2 file. Trilio leverages the same mechanism to store incremental backups. Trilio incremental backups are overlay files that include the data that is modified between the current backup process and the last good backup. Since the TrilioVault backup image structure on the backup media reflects what KVM/RHV natively represents, our process of creating backups and restores are highly efficient in terms of the amount of backup storage used and the network bandwidth utilization.

Trilio architecture reflects these principles.

As you can see from above the architecture diagram, Trilio does not require any media servers. Traditionally, media servers performs numerous bookkeeping operations of backup images, including pruning older backups, synthesizing full backups from existing backups, cataloging backup images and other operations. These are usually data intensive operations and as your RHV cluster grows, media servers need to scale in capacity to keep up with RHV growth. Scaling media servers may include trial and errors approaches and very difficult to calibrate correctly. TrilioVault employs data movers that are deployed on each RHV host that can horizontally scale with RHV, hence, there is no tuning to do when RHV cluster grows. Instead of centralizing media server functionality in to one appliance, all bookkeeping operations are performed with in the data mover in the context of current backup job. TrilioVault enhances operational efficiency of backups and recoveries and by not tieing the users to hardware licenses it also significantly improves the ROI and TCO of your investments.

Trilio Appliance

The Trilio Appliance is the controller of Trilio, called TVM.

The TVM is running and managing all backup and recovery jobs.

During a backup job is the TVM:

• Gathering the Metadata information generated of the VMs that are getting protected

• Writing the Metadata information onto the Backup Target

• Generating the RHV Snapshot

The TVM is available as qcow2 image and runs as VM on top of a KVM Hypervisor.

It is supported and recommended to run the TVM in the same RHV environment as a VM that the TVM protects.

RHV GUI integration

Trilio is natively integrated into the available RHV GUI, provides a new tab "Backup".

All functionalities of Trilio are accessible through the RHV GUI.

The RHV-Manager GUI integration is getting installed using Ansible-playbooks together with the ovirt-imageio-proxy extension.

Ovirt-imageio extensions

Ovirt-imageio is an RHV internal python service that allows the upload and download of disks into and out of RHV.

The default ovirt-imageio services only allow to move the disks through the RHV-M via https.

Trilio extends the ovirt-imageio functionality to provide movement of the disk data through NFS over the RHV Hosts themselves.

The ovirt-imageio extensions are getting installed using Ansible-playbooks.

Backup Target

Trilio is writing all Backups over the network using the NFS protocol to a provided Backup Target.

Any system utilizing the NFSv3 protocol is usable.

TrilioVault for RHV does provide the capabilties to take application consistent backups by utilizing the Qemu-Guest-Agent.

The Qemu-Guest-Agent

The Qemu-Guest-Agent is a component of the qemu hypervisor, which is used by RHV. RHV automatically builds all VMs to be prepared to use the Qemu-Guest-Agent.

The Qemu-Guest-Agent provides many capabilities, including the possibility to freeze and thaw Virtual Machines Filesystems.

The Trilio Appliance is delivered as qcow2 image and runs as VM on top of a KVM Hypervisor.

TrilioVault for RHV provides the capabilities to generate reports, to gain insights into the usage of TrilioVault over time and the stability of the service.

Definition

A Snapshot is a single TrilioVault backup of a workload including all data and metadata. It contains the information of all VM's that are protected by the workload.

Definition

A workload is a backup job that protects one or more Virtual Machines according to a configured policy. There can be as many workloads as needed. But each VM can only be part of one Workload.

Create a Workload

To create a workload do the following steps:

1. Login to the RHV-Manager

2. Navigate to the Backup Tab

3. Click "Create Workload"

The created Workload will be available after a few seconds and starts to take backups according to the provided schedule and policy.

Workload Overview

A workload contains many information, which can be seen in the workload overview.

To enter the workload overview do the following steps:

1. Login to the RHV-Manager

2. Navigate to the Backup Tab

3. Identify the workload that shall create a Snapshot

Details Tab

The Workload Details tab provides you with the general most important information about the workload:

• Name

• Description

• List of protected VMs

Snapshots Tab

The Workload Snapshots Tab shows the list of all available Snapshots in the chosen Workload.

From here it is possible to work with the Snapshots, create Snapshots on demand and start Restores.

Policy Tab

The Workload Policy Tab shows gives an overview of the current configured scheduler and retention policy. The following elements are shown:

• Scheduler Enabled / Disabled

• Start Date / Time

• End Date / Time

Filesearch Tab

The Workload Filesearch Tab provides access to the power search engine, which allows to find files and folders on Snapshots without the need of a restore.

Misc. Tab

The Workload Miscellaneous Tab shows the remaining metadata of the Workload. The following information are provided:

• Creation time

• last update time

• Workload ID

Edit a Workload

Workloads can be modified in all components to match changing needs.

To edit a workload do the following steps:

1. Login to the RHV-Manager

2. Navigate to the Backup Tab

3. Identify the workload to be modified

Delete a Workload

Once a workload is no longer needed it can be safely deleted.

To delete a workload do the following steps:

1. Login to the RHV-Manager

2. Navigate to the Backup Tab

3. Identify the workload to be deleted

Reset a Workload

In rare cases it might be necessary to start a backup chain all over again, to ensure the quality of the created backups. To not recreate a Workload in such cases is it possible to reset a Workload.

The Workload reset will:

• Cancel all ongoing tasks

• Delete all existing RHV Snapshots from the protected VMs

• recalculate the next Snapshot time

To reset a Workload do the following steps:

1. Login to the RHV-Manager

2. Navigate to the Backup Tab

3. Identify the workload to be deleted

TrilioVault for RHV integrates tightly into the RHV environment itself. This integration requires preparation before the installation starts.

Installing Redis

TrilioVault is capable of parallel disk transfer from multiple RHV-Hosts at the same time.

This capability is requiring a task queue system, Python Celery.

Python Celery requires a message broker system like RabbitMQ or Redis. TrilioVault uses the Redis message broker.

RHV does not include Redis, so installation is necessary.

The following steps install Redis:

1. Add the Fedora EPEL Repository# yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

2. Install Redis # yum install redis

3. Start Redis # systemctl start redis.service

(CLI) Uploading the TrilioVault VM qcow2 disk to RHV

Trilio is providing a python script to upload the TrilioVault qcow2 image to RHV using the RHV Apis.

The provided script is an adaption of the opensource .

Trilio's changes are:

• compatibility for RHV 4.3 and RHV 4.4

• compatibility with Python 2.7 and Python 3.5+

The script can be downloaded here:

The recommended process for using this script is:

1. Upload the TrilioVault qcow2 image to the RHV-Manager VM

2. Upload the script to the RHV-Manager VM

3. Unpack the TrilioVault qcow2 image if necessary

Command to upload the disk using the script

For Python 2:

For Python 3:

(GUI) Enabling Disk Upload through RHV Manager

Trilio delivers the TrilioVault Appliance as a qcow2 image.

Trilio supports that the TrilioVault Appliance is running on the same RHV Cluster it protects.

Uploading the qcow2 to the RHV Datastore is easy, but depending on the RHV usage so far, it might require the installation of additional certificates.

Verify the connection to the ovirt-imageio-proxy

RHV is using the ovirt-imageio-proxy service to upload and download images, snapshots, and disks through the RHV Manager.

The following steps verify the connection to the ovirt-imageio-proxy service.

1. Log in to the administrative portal of the RHV Manager

2. Go to Storage ➡️ Disks

3. Go to Upload ➡️ Start

Install the ovirt-engine certificate

When the Test-Connection to the ovirt-imageio-proxy is failing, a usual reason is the client system mistrusting the RHV-M due to a missing certificate.

The RHV-M does have two certificates, which can both be required to access the ovirt-imageio-proxy.

The first certificate is directly available for download from the error message in the window. The below URL shows the general path to the certificate. The downloaded certificate is the root certificate for the certificates used by the ovirt-imageio-proxy.

https:///ovirt-engine/services/pki-resource?resource=ca-certificate&format=X509-PEM-CA

Download and install this certificate according to the clients operating system and browser used.

Test the connection to the ovirt-imageio-proxy again after installation.

The second certificate is the actual certificate of the ovirt-imageio-proxy shown to the client system upon connection. The download is only possible from the RHV-M host system directly. The usual location of the certificate is:

/etc/pki/ovirt-engine/certs/imageio-proxy.cer

Install the certificate according to the client's operating system and browser used.

Test the connection to the ovirt-imageio-proxy again after installation.

(GUI) Uploading the TrilioVault VM qcow2 disk to RHV

The TrilioVault VM qcow2 image is a full operating system disk, that is getting attached to a VM running on the RHV-Cluster.

To be able to spin up the TrilioVault VM, upload the qcow2 disk into the RHV datadomain.

The following procedure uploads the qcow2 disk:

1. Go to Storage➡️Disk

2. Go to Upload➡️Start

3. Fill out the presented form and choose the path to the qcow2 image on the client system

TrilioVault allows to directly mount a Snapshot through the RHV-Manager.

This feature provides the capability to download any file from any Snapshot through the RHV-Manager independent of size.

Mounting a Snapshot

To mount a Snapshot follow these steps:

1. Login to the RHV-Manager

2. Navigate to the Backup Tab

3. Identify the workload that contains the Snapshot to show

Navigating the mounted Snapshot

A mounted Snapshot can be navigated like in any browser by clicking on files and folders.

Clicking on a directory will open that directory.

Clicking on a file will provide an overview about the metadata of this file including:

• Name

• Size

• Last Modified

It is further possible to get a preview of the file directly from this overview or to download the file through the RHV-Manager.

Definition

A Restore is the workflow to bring back the backed up VMs from a TrilioVault Snapshot.

TrilioVault does offer 3 types of restores:

The Trilio Appliance requires configuration to work with the chosen RHV environment. A Web-UI provides access to the Trilio Appliance dashboard and configurator.

Accessing the Trilio Dashboard

Enter the Trilio IP or FQDN into the browser to reach the Trilio Appliance landing page.

User: admin Password: password

Details needed for the Trilio Appliance configurator

Upon login into the Trilio Appliance, the shown page is the configurator. The configurator requires some information about the Trilio Appliance, RHV and Backup Storage.

Trilio Nodes information

The Trilio Appliance needs to be integrated into an existing environment to be able to operate correctly. This block asks for the information about the Trilio Appliance operating details.

• Virtual IP Address

  • The Trilio Appliance uses this IP address for all communication with RHV.

  • Format: IP/Netmask

• TVM Appliance IP

  • The first interface in the interface list of the Trilio Appliance will get this IP address assigned. Further is the TrilioVault Appliance hostname set.

  • Format: IP=hostname

• Name Servers

  • The DNS server the Trilio appliance will use.

  • Format: Comma separated list of IPs

RHV Credentials information

The Trilio appliance integrates with one RHV environment. This block asks for the information required to access and connect with the RHV Cluster.

• RHV Engine URL

  • URL of the RHV-Manager used to authenticate

  • Format: URL (FQDN and IP supported)

• RHV Username

  • admin-user to authenticate against the RHV-Manager

  • Format: user@domain

Backup Storage Configuration information

This block asks for the necessary details to configure the Backup Storage.

• Backup Storage

  • Predefined as NFS

• NFS Export

  • Full path to the NFS Volume used as Backup Storage

  • Format: Comma separated list of NFS paths

Trilio Certificate information

Trilio is integrating into the RHV Cluster as an additional service, following the RHV communication paradigms. These require that the Trilio Appliance is using SSL and that the RHV-Manager does trust the Trilio Appliance.

Trilio offers to possibilities how these required certificates can be provided. Either Trilio generates a complete fresh self-signed certificate or a certificate is provided.

In both cases is the FQDN required, to which the certificate is pointing to.

Please see below example in case of a provided certificate.

• FQDN

  • FQDN to reach the Trilio Appliance

  • Format: FQDN

Trilio License

It is possible to directly provide the Trilio Appliance with the license file that is going to be used by it.

It is not necessary to provide the License file directly through the configurator. It is also possible to provide the license afterwards through the Trilio License tab in the Trilio dashboard.

The Trilio License tab can also be used to verify and update the currently installed license.

Submit and Configuration

After filling out every block of the configurator, hit the submit button to start the configuration.

The configurator asks one more time for confirmation before starting.

Stay patient during the configuration, as this may easily take a few more minutes.

After the configurator has succeeded or failed, is the Ansible playbook shown. Use the possibilities to expand and collapse each task for troubleshooting failed configurations.

After the installation and configuration of TrilioVault for RHV did succeed the following steps can be done to verify that the TrilioVault installation is healthy.

Verify the TrilioVault Appliance services are up

TrilioVault is using 3 main services on the TrilioVault Appliance: