TrilioVault for RHV, by Trilio Data, is a native RHV service that provides policy-based comprehensive backup and recovery for RHV workloads. The solution captures point-in-time workloads (Application, OS, Compute, Network, Configurations, Data, and Metadata of an environment) as full or incremental snapshots. A variety of storage environments can hold these Snapshots, including NFS and soon AWS S3 compatible storage. With TrilioVault and its single-click recovery, organizations can improve Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO). TrilioVault enables IT departments to fully deploy RHV solutions and provide business assurance through enhanced data retention, protection, and integrity.
With the use of TrilioVault’s VAST (Virtual Snapshot Technology), Enterprise IT and Cloud Service Providers can now deploy backup and disaster recovery as a service to prevent data loss or data corruption through point-in-time snapshots and seamless one-click recovery. TrilioVault takes point-in-time backup of the entire workload consisting of computing resources, network configurations, and storage data as one unit. It also takes incremental backups that only capture the changes made since the last backup. Incremental snapshots save time and storage space as the backup only includes changes since the last backup. The summarized benefits of using VAST for backup and restore are:
Efficient capture and storage of snapshots.
Since our full backups only include data that is committed to storage volume and the incremental backups include changed blocks of data since the last backup, our backup processes are efficient and storages backup images efficiently on the backup media.
Faster and reliable recovery.
When your applications become complex that snap multiple VMs and storage volumes, our efficient recovery process brings your application from zero to operational with the click of a button.
Reliable and smooth migration of workloads between environments.
TrilioVault captures all the details of your application, and hence our migration includes your entire application stack without leaving anything for guesswork.
Through policy and automation, lower the Total Cost of Ownership.
Our role driven backup process and automation eliminates the need for dedicated backup administrators, thereby improves your total cost of ownership.
TrilioVault for RHV Architecture
TrilioVault is built on the same architectural principles as modern analytical platforms such as Hadoop and other big data platforms. These platforms offer infinite scale without compromising on performance. TrilioVault's other attributes include agentless, natively integrated with RHV GUI, horizontally scalable, nondisruptive, and open universal backup schema.
Agentless
TrilioVault offers image-level backup, which backs up a given virtual machine physical disks as one file. Irrespective of the complexity of a VM or the applications running inside the VM, the user does not require any custom code, called agents, inside VM for TrilioVault to take VM backups. Agentless solutions are one of the highly desirable features because any solution that requires custom code to run in VM creates an operational nightmare.
TrilioVault 3.7 Support Matrix
iSCSI
1.5.1
1.5.2
1.5.3
RHV 4.3.9 contains a bug which highly impacts TrilioVault up to the point of being unfunctional.
A Red Hat Hotfix is available from TrilioVault Customer Success.
The patch will be included officially in RHV 4.3.10
RHV Version
ovirt-imageio version
Storage Domain
4.2.8
1.4.5
NFSv3
4.3.X
1.4.8
Self-Service/UI Integrated
TrilioVault comes with a GUI plugin for RHV Manager, which provides seamless integration of TrilioVault functionality adjacent to virtual machines management. TrilioVault service authenticates users with OpenID tokens, so any user who logged in to RHV can use TrilioVault functionality without any out-of-band user management.
Scalable, Linear, Infinite Scale
Most backup solutions are built on client/server architectures, and hence, they invariably create performance and scale bottlenecks when the RHV cluster grows. Traditional backup solutions require constant tweaking to keep up with RHV cluster growth. TrilioVault is built on the same architectural principles as the RHV platform; hence, it grows with the RHV cluster without introducing scale and performance bottlenecks.
Nondisruptive
Deploying TrilioVault is nondisruptive to the RHV cluster or the virtual machines. Similarly, uninstalling TrilioVault is nondisruptive as well.
Open Universal Backup Schema
In the current world of multi-cloud environments, the backup images must be platform and vendor-neutral, so they are easily portable between clouds. TrilioVault saves backup images as QCOW2 images. QCOW2 is a standard format in KVM/RHV environments for virtual disks, and Linux comes with numerous tools to create and manage them. A backup image stored in QCOW2 format gives the user enormous flexibility on how these are leveraged for various use cases, including restoring backup images without TVM.
QCOW2 images also come with two important attributes that also make them ideal for storing backup images.
QCOW2s are sparse friendly. As a regular practice, users overprovision virtual disks to VMs. These virtual disks may be thick or thin-provisioned, but at any given time, applications only use a fraction of the virtual disk capacity. When taking image-level backups of virtual disks, backup solutions should only save the blocks that are allocated and not save blocks that are not allocated or used. For example, your virtual disks maybe a 1TB in capacity, but the applications utilized only 10GB of disk space. Since QCOW2 images are spare friendly, TrilioVault only stores the data. In the above example, the size of the QCOW2 image is 10GB
KVM/RHV supports virtual disk snapshots by a construct called overlay files. KVM/RHV creates a new disk snapshot, it creates a new qcow2 file called overlay file and is overlaid on the original qcow2 file. Any new writes are applied to overlay file, and any reads to old data is read from the older qcow2 file. TrilioVault leverages the same mechanism to store incremental backups. TrilioVault incremental backups are overlay files that include the data that is modified between the current backup process and the last good backup. Since the TrilioVault backup image structure on the backup media reflects what KVM/RHV natively represents, our process of creating backups and restores are highly efficient in terms of the amount of backup storage used and the network bandwidth utilization.
TrilioVault architecture reflects these principles.
TrilioVault for RHV Architecture overview
As you can see from above the architecture diagram, TrilioVault does not require any media servers. Traditionally, media servers performs numerous bookkeeping operations of backup images, including pruning older backups, synthesizing full backups from existing backups, cataloging backup images and other operations. These are usually data intensive operations and as your RHV cluster grows, media servers need to scale in capacity to keep up with RHV growth. Scaling media servers may include trial and errors approaches and very difficult to calibrate correctly. TrilioVault employs data movers that are deployed on each RHV host that can horizontally scale with RHV, hence, there is no tuning to do when RHV cluster grows. Instead of centralizing media server functionality in to one appliance, all bookkeeping operations are performed with in the data mover in the context of current backup job. TrilioVault enhances operational efficiency of backups and recoveries and by not tieing the users to hardware licenses it also significantly improves the ROI and TCO of your investments.
TrilioVault Appliance
The TrilioVault Appliance is the controller of TrilioVault, called TVM.
The TVM is running and managing all backup and recovery jobs.
During a backup job is the TVM:
Gathering the Metadata information generated of the VMs that are getting protected
Writing the Metadata information onto the Backup Target
Generating the RHV Snapshot
Sending the data copy commands to the ovirt-imageio services
The TVM is available as qcow2 image and runs as VM on top of a KVM Hypervisor.
It is supported and recommended to run the TVM in the same RHV environment as a VM that the TVM protects.
RHV GUI integration
TrilioVault is natively integrated into the available RHV GUI, provides a new tab "Backup".
All functionalities of TrilioVault are accessible through the RHV GUI.
The RHV-Manager GUI integration is getting installed using Ansible-playbooks together with the ovirt-imageio-proxy extension.
Ovirt-imageio extensions
Ovirt-imageio is an RHV internal python service that allows the upload and download of disks into and out of RHV.
The default ovirt-imageio services only allow to move the disks through the RHV-M via https.
TrilioVault extends the ovirt-imageio functionality to provide movement of the disk data through NFS over the RHV Hosts themselves.
The ovirt-imageio extensions are getting installed using Ansible-playbooks.
Backup Target
TrilioVault is writing all Backups over the network using the NFS protocol to a provided Backup Target.
Any system utilizing the NFSv3 protocol is usable.
TrilioVault 3.7 Release Notes
TrilioVault 3.7 is the third release of TrilioVault for Red Hat Virtualization.
It is aimed to provide Backup and Recovery for Red Hat Virtualization 4.2.8 & 4.3.x.
The full requirements can be found here.
TrilioVault 3.7 Features on a Glance
New Feature: File Recovery
TrilioVault 3.7 introduces the File Recovery to TrilioVault for RHV.
It is now possible to recover single files from any backups by downloading the files through the RHV-M
This feature is provided in two ways:
For files smaller than 2GB it is possible to download the file directly through the
It is possible to and browse a Snapshot to go to a file location and download it directly without any size limits.
New Feature: RestFul API
TrilioVault 3.7 makes the RestFul API of the TrilioVault Appliance available to use.
This feature allows to integrate TrilioVault 3.7 into any orchestration tool capable of using a RestFul API.
Known limitations
Large Size backups failing with error - "Could not initialize session: Unable to verify proxy ticket"
RHV has a limitation of 10hours for a disk's transfer session, which cannot be altered. RHV setups on slower networks will face this issue.
Ansible imageio daemon/proxy script execution interruption can leave imageio unstable
In the case of the Ansible playbooks installing the ovirt-imageio extensions being interrupted is it possible that the ovirt-image io version are getting unstable and no longer usable.
Workaround: Uninstall the ovirt-imageio extension and install them again.
RHV Snapshot changing preallocated disktypes to thinprovisioned
RHV Snapshots are always thin-provisioned qcow2 images. These qcow2 images are the active images for the VM running and they are the qcow2 backing file capability to point to the original image. That's why the disktype is changing from preallocated to thinprovisioned upon snapshotting.
Manual deletion or cancelling of any Snapshot from TVM promotes next snapshot to be Full
Manual deletion of Snapshots is only expected to happen, when an error occurred that got not identified by TrilioVault during the backup process.
To prevent possible data loss is the next Snapshot upgraded to a Full Snapshot.
Workloads are getting created for unprotected hosts
The list of available VMs is taken from the RHV-Manager and every VM not part of a Snapshot can be included in a workload. It is not checked if the Virtual Machines are running on RHV-Hosts that do not have the ovirt-imageio-daemon extension installed.
TrilioVault requires all RHV-Hosts to have the ovirt-imageio-daemon extension to be installed.
File search is case sensitive
TrilioVault is providing filesearch through libguestfs and therefor bound by any libguestfs limitations.
Known issues
VMs being retsored from RHV 4.2 to 4.3 might need changing the custom compatibility version
TrilioVault is storing and restoring the VM with all its configurations, including the compatibility version tied to a RHV release.
Upon a restore might the VM fail to power up after the restore. The restore itself succeeds.
Workaround: Edit the VM configuration and update the compatibility version.
Import Workloads gives Gateway Timeout error on UI
Importing a bigger amount of Workloads with many Snapshots leads to the TrilioVault Dashboard refresh timing out before the import has finished. The import is continuing in the background as desired.
Recommendation: Verify from the RHV-M or via TrilioVault API the amount of available workloads.
Create Workload window does not disappear while creating large (>150 VMs) workloads
While the workload is being created on the TrilioVault Appliance is the RHV-M integration waiting for the workload creation successful signal to close the create workload window. The workload creation is continuing as desired.
TVM Reconfiguration requires manual restart of services
During reconfiguration it is possible that the configurator fails during nginx service restart or wlm-api restart.
Workaround: Restart nginx and wlm-api services manually before retrying the configuration.
Snapshots fail with 500 error "Server failed to performed the request" after server restart
After restarting the RHV-Hosts it is possible that the communication between Redis and ovirt_celery is broken.
Rerunning the Ansible-Playbooks for all RHV-Hosts fixes this issue.
Restarting the TrilioVault during a running Snapshot leads to Snapshot getting stuck in execution state
Restarting the TrilioVaut VM will stop any ongoing backup or restore processes on the TrilioVault Appliance. This can lead to the Snapshot status not being updated in the TrilioVault database, leaving them stuck in execution state without any task connected to them.
Further Snapshots are not getting affected.
Please contact Customer Success for help in moving stuck Snapshots into the error state.
Upgrade TrilioVault
To upgrade TrilioVault it is necessary to uninstall the old version of TrilioVault and install the new version.
During configuration check the box for workload import. This will automatically load all workloads hosted on the backup target into the fresh configured TrilioVault Appliance.
Global Job Scheduler
Definition
The Global Job Scheduler controls whether Workloads will automatically take backups according to their schedule or not.
It is used to prevent automated backups during maintenance or troubleshooting.
Disabling an Enabling of the Global Job Scheduler
To disable or enable the Global Job Scheduler follow these steps:
Login to the RHV-Manager
Navigate to the Backup Tab
Click "Scheduler Settings"
Already started backups will finish their backup process when the Global Job Scheduler gets deactivated after the backup job has been started.
Snapshot mount
TrilioVault allows to directly mount a Snapshot through the RHV-Manager.
This feature provides the capability to download any file from any Snapshot through the RHV-Manager independent of size.
It is not possible to download complete directories
Important RHV-Manager Logs
TrilioVault data transfer related logs
/var/log/ovirt-imageio-proxyimage-proxy.log
This log provides the used ticket number and RHV-Host for a backup transfer.
It is helpful to identify the ticket numbers that are used in RHV to track a specific data transfer.
It also shows any connection errors between the RHV-M and the RHV-Host.
Important RHV-Host Logs
TrilioVault data transfer related logs
/var/log/ovirt_celery/worker<x>.log
The worker logs contain the status of the disk transfer from the RHV Host to the backup target.
Useful if the data transfer process gets stuck or errors in between.
Choose between "Enable Scheduler" or "Disable Scheduler"
Click "Submit"
Mounting a Snapshot
To mount a Snapshot follow these steps:
Login to the RHV-Manager
Navigate to the Backup Tab
Identify the workload that contains the Snapshot to show
Click the workload name to enter the Workload overview
Navigate to the Snapshots tab
Identify the searched Snapshot in the Snapshot list
Click the Snapshot Name
Click File Manager
Click the VM to be mounted (this might take a minute)
Only VM can be mounted at the same time for the complete RHV environment.
Navigating the mounted Snapshot
A mounted Snapshot can be navigated like in any browser by clicking on files and folders.
Clicking on a directory will open that directory.
Clicking on a file will provide an overview about the metadata of this file including:
Name
Size
Last Modified
Last accessed
Owner
Owner Group
Permissions
It is further possible to get a preview of the file directly from this overview or to download the file through the RHV-Manager.
Generally helpful RHV-Manager logs
TrilioVault is calling a lot of RHV APIs to read metadata, create RHV Snapshots and restore VMs.
These tasks are done by RHV independently from TrilioVault and are logged in RHV logs.
/var/log/ovirt-engine/engine.log
This log is hard to read, but contains all tasks that the RHV-M is doing, including all Snapshot related tasks.
Additional logs that can be helpful during troubleshooting are:
The daemon.log contains all informations about the actual connection between the RHV Host and the backup target.
Useful to identify potential connection issues between the RHV Host and Backup target.
TrilioVault extends the ovirt-imageio services running on the RHV-Manager and the RHV hosts, to provide the parallel download of disks from multiple RHV hosts.
The imageio extensions are getting installed automatically using Ansible playbooks provided on the TrilioVault Appliance.
Every time the RHV environment gets updated or a new RHV host is getting added to the RHV Cluster it is necessary to rerun the installation of the ovirt-imageio extensions.
Preparing the inventory files
Ansible playbooks are working with inventory files. These inventory files contain the list of RHV-Hosts and RHV-Managers and how to access them.
To edit the inventory files, open the following files for the server type to add.
For the RHV hosts open: /opt/stack/imageio-ansible/inventories/production/daemon
For the RHV Manager open: /opt/stack/imageio-ansible/inventories/production/proxy
Using password authentication
The first supported method to allow Ansible to access the RHV hosts and the RHV Manager is the classic user password authentication.
To use password authentication edit the files using the following format:
<Server_IP> ansible_user=root password=xxxxx
One entry per RHV Host in daemon file and one entry per RHV Manager in the proxy file are required.
Using passwordless authentication (SSH keys)
The second supported method to Allow Ansible to access the RHV hosts and the RHV Manager is utilizing SSH keys to provide passwordless authentication.
For this method, it is necessary to prepare the TrilioVault Appliance and the RHV Cluster Nodes as well as the RHV Manager.
The recommended method from Trilio is:
Use ssh-keygen to generate a key pair
Add the private key to /root/.ssh/ on the TrilioVault Appliance
Add the public key to /root/.ssh/authorized_keys file on each RHV host and the RHV Manager
Once the TrilioVault Appliance can access the nodes without password, edit the inventory files using the following format:
<Server_IP> ansible_user=root
One entry per RHV Host in the daemon file and one entry per RHV Manager in the proxy file are required.
Starting the installation
To install the ovirt-imageio extensions go to:
/opt/stack/imageio-ansible
Depending on the method of authentication prepared in the inventory files, different commands need to be used to start the Ansible playbooks.
Using password authentication
To call the Ansible playbooks when the inventory files use password authentication run.
For RHV Hosts: ansible-playbook site.yml -i inventories/production/daemon --tags daemon
For RHV Manager: ansible-playbook site.yml -i inventories/production/proxy --tags proxy
Using passwordless authentication (SSH keys)
To call the Ansible playbooks when the inventory files use passwordless authentication run.
Ansible shows the output of the running playbook. Do not intervene until the playbook has finished.
Important TVM Logs
TrilioVault Appliance logs used during configuration
The following logs contain all information gathered during the configuration of the TrilioVault Appliance
/var/log/workloadmgr/tvault-config.log
This log contains all information of pre-checks done, when filling out the configurator form.
/var/log/workloadmgr/ansible-playbook.log
This log contains the complete Ansible output from the playbooks that run when the configurator is started.
With each configuration attempt a new ansible-playbook.log gets created. Old ansible-playbook.logs are renamed according to their creation time.
TrilioVault Appliance logs during any task after configuration
/var/log/workloadmgr/workloadmgr-api.log
This log tracks all API requests that have been received on the wlm-api service.
This log is helpful to verify that the TrilioVault VM is reachable from the RHV-M and authentication is working as expected.
/var/log/workloadmgr/workloadmgr-scheduler.log
This log tracks all jobs the wlm-scheduler is receiving from the wlm-api and sends them to the chosen wlm-workloads service.
This log is helpful, when the wlm-api doesn't throw any error, but no task like backup or restore is getting started.
/var/log/workloadmgr/workloadmgr-workloads.log
This log contains the complete output from the wlm-workloads service, which is controlling the actual backup and restore tasks.
This log is helpful to identify any errors that are happening on the TVM itself including RESTful api responses from the RHV-M.
Requirements
TrilioVault is a pure software solution and is composed of 4 elements:
TrilioVault Appliance (Virtual Machine)
TrilioVault RHV-M Web-GUI extension
TrilioVault ovirt-imageio-proxy extension
TrilioVault ovirt-imageio-daemon extension
System requirements TrilioVault Appliance
The TrilioVault Appliance gets delivered as a qcow2 image, which gets attached to a virtual machine.
Trilio supports only KVM based hypervisors and recommends to use the RHV Cluster as the hoster for the TrilioVault Appliance.
The recommended size of the VM for the TrilioVault Appliance is:
The qcow2 image itself defines the 40GB disk size of the VM.
In the rare case of the TrilioVault Appliance database or log files getting larger than 40GB disk, contact or open a ticket with Trilio Customer Success to attach another drive to the TrilioVault Appliance.
System requirements TrilioVault ovirt-imageio extension
TrilioVault is extending the ovirt-imageio-proxy service running on the RHV-Manager and ovirt-imageio-daemon running on the RHV-Hosts.
These extensions do not have any hardware related requirements, but they require specific versions of the ovirt-imageio services.
Please check the for further information.
The installed versions of the ovirt-imageio-proxy and the ovirt-imageio-daemon need to be the same.
Uninstall TrilioVault
Uninstalling TrilioVault is done in 2 easy steps, which leave only the already created backups behind.
Step 1: Uninstall RHV ovirt-imageio extension
To uninstall the ovirt-imageio extension do the following:
Login into the TrilioVault Appliance CLI
Verify the inventory files are still correct
/opt/stack/imageio-ansible/inventories/production/daemon
/opt/stack/imageio-ansible/inventories/production/proxy
Run the ansible playbooks with the clean tags
cd /opt/stack/imageio-ansible/
ansible-playbook site.yml -i inventories/production/daemon --tags clean-daemon
ansible-playbook site.yml -i inventories/production/proxy --tags clean-proxy
Step 2: Destroy the TrilioVault Appliance
This guide assumes you are running the TrilioVault Appliance in a RHV environment
To destroy the TrilioVault Appliance do the following:
Login into the RHV-Manager
Navigate to Compute➡️Virtual Machines
Mark the TrilioVault Appliance in the list of VMs
File Search
Definition
The file search functionality allows the user to search for files and folders located on a chosen VM in a workload in one or more Backups.
Click "Shutdown" or "Power Off"
Wait till the shutdown procedure finishes
Click "Remove" to destroy the TrilioVault Appliance
Navigating to the file search tab
The file search tab is part of every workload overview.
To reach it follow these steps:
Login to the RHV-Manager
Navigate to the Backup Tab
Identify the workload a file search shall be done in
Click the workload name to enter the Workload overview
Click File Search to enter the file search tab
Configuring and starting a file search
A file search runs against a single virtual machine for a chosen subset of backups using a provided search string.
To run a file search the following elements need to be decided and configured
Choose the VM the file search shall run against
Under VM Name/ID choose the VM that the search is done upon.
The drop down menu provides a list of all VMs that are part of any Snapshot in the Workload.
VMs that are no longer activly protected by the Workload but are still part of an existing Snapshot are listed in red.
Set the File Path
The File Path defines the search string that is run against the chosen VM and Snapshots.
This search string does support basic RegEx.
The File Path has to start with a '/'
Windows partitions are fully supported. Each partition is it's own Volume with it's own root.
Use '/Windows' instead of 'C:\Windows'
The file search does not go into deeper directories and always searches on the directory provided in the File Path
Example File Path for all files inside /etc : /etc/*
Define the Snapshots to search in
Filter Snapshots by is the third and last component that needs to be set.
This defines which Snapshots are going to be searched.
There are 3 possibilities for a pre-filtering:
All Snapshots - Lists all Snapshots that contain the chosen VM from all available Snapshots
Last Snapshots - Choose between last 10, 25, 50, or custom Snapshots and click Apply to get the list of the available Snapshots for the chosen VM that match the criteria.
Date Range - Set a start and end date and click apply to get the list of all available Snapshots for the chosen VM within the set dates.
After the pre-filtering is done choose the Snapshots that shall be searched by clicking their checkbox or by clicking the global checkbox.
When no Snapshot is chosen the file search will not start.
Start the File Search and retrieve the results
To start a File Search the following elements need to be set:
A VM to search in has to be choosen
A valid File Path provided
At least one Snapshot to search in selected
Once those have been set click "Search" to start the file search.
Do not navigate to any other RHV tab or website after starting the File Search.
Results are lost and the search has to be repeated to regain them.
After a short time the results will be presented.
The results are presented in a tabular format grouped by Snapshots and Volumes inside the Snapshot.
For each found file or folder the following information are provided:
POSIX permissions
Amount of links pointing to the file or folder
User ID who owns the file or folder
Group ID assigned to the file or folder
Actual size in Bytes of the file or folder
Time of creation
Time of last modification
Time of last access
Full path to the found file or folder
Once the Snapshot of interest has been identified it is possible to go directly to the Snapshot using the "View Snapshot" option.
The TrilioVault Appliance is delivered as qcow2 image and runs as VM on top of a KVM Hypervisor.
The TrilioVault VM qcow2 image must be an available disk on the RHV Storage before the creation of the TrilioVault Appliance is possible.
This guide shows the tested way to spin up the TrilioVault Appliance on a RHV Cluster. Please contact a RHV Administrator and Trilio Customer Success Agent in case of incompatibility with company standards.
Creation the TrilioVault VM
The creation of the TrilioVault VM works like for any other Virtual Machine inside RHV.
To create a new Virtual Machine, go to Compute ➡️ Virtual Machines.
The button "New" opens the window to define the VM.
The following instructions show the tested configuration for the TrilioVault Appliance.
After configuration, use the OK button to create the TrilioVault Appliance.
It is a required to activate the Advanced Options.
General Tab
Fill out the following details as necessary on the General tab:
Cluster - Choose the RHV Cluster to host the TrilioVault VM
Template - Blank
Operating System - The TrilioVault VM runs CentOS 7. Red Hat Enterprise 7.x x64 is a valid option.
Before moving to the next tab, attach the TrilioVault qcow2 image to the VM definition.
Click Attach under Instance Images.
Choose the TrilioVault qcow2 image
Check the box for OS
Without checking the box for OS, will the TrilioVault Appliance not boot, as the RHV VM is not utilizing the disk as the boot disk.
System Tab
Under the System tab set the following:
Memory size - 24576 MB / 24 GB
Maximum memory - 24576 MB / 24 GB (RHV automatically first sets four times the Memory size)
Physical Memory Guaranteed - 24576 MB / 24 GB (RHV automatically first sets the same value as Memory size)
It is possible to set the initial Memory size to 8GB. RHV is automatically setting the Maximum Memory to 4 times the Memory size value. The actual Memory size can be adjusted later as needed.
Note: Do not set the Physical Memory Guaranteed below 8GB.
Total virtual CPUs - 4
Nothing to set at the Advanced Parameters
Leave Hardware Clock Timer Offset at 0
Initial Run
Under the Initial Run tab set the following:
Check "Use Cloud-Init/Sysprep"
(Optional) Set VM Hostname
(Optional) Check and set "Configure Time Zone"
Further Tabs
There are no TrilioVault specific configurations necessary in any further tab.
Starting the TrilioVault Appliance
After the creation of the TrilioVault Appliance VM is the VM in a shutdown state.
Go to the overview of VMs in the RHV Manager (Compute ➡️ Virtual Machines), identify the TrilioVault Appliance VM in the list, mark it, and click the Run button to start it.
After the first boot of the TrilioVault Appliance is Cloud-Init getting disabled.
Snapshots
Definition
A Snapshot is a single TrilioVault backup of a workload including all data and metadata. It contains the information of all VM's that are protected by the workload.
Preparing the Installation
TrilioVault for RHV integrates tightly into the RHV environment itself.
This integration requires preparation before the installation starts.
Installing Redis
TrilioVault is extending the ovirt-imageio services running on the RHV Manager and the RHV Hosts to allow the parallel disk transfer from multiple RHV-Hosts at the same time.
This extension is using a task queue system, .
Instance Type - Custom
Optimized for - Server
Name - Provide a RHV internal name for the TrilioVault VM
Description - Provide a RHV internal description for the TrilioVault VM (optional)
Comment - Provide a RHV internal comment for the TrilioVault VM (Optional)
Activate Delete Protection
NICs - Choose the network the TrilioVault VM connects with. The plus and minus symbols add/delete NICs as necessary.
Leave custom serial policy unchecked
Open Authentication
Set User Name
Set Password or SSH authentication
Open Networks
Set "Cloud-Init Network Protocol" to "Openstack Metadata"
(optional) Set DNS Servers and DNS Search Domains
Check In-guest Network Interface Name
Set IPv4 configuration as necessary
Creating a Snapshot
Snapshots are automatically created by the TrilioVault scheduler.
If necessary or in case of deactivated scheduler is it possible to create a Snapshot on demand.
There are 2 possibilities to create a snapshot on demand.
Possibility 1: From the Backup overview
Login to the RHV-Manager
Navigate to the Backup Tab
Identify the workload that shall create a Snapshot
Click "Create Snapshot"
Provide a name and description for the Snapshot
Decide between Full and Incremental Snapshot
Click "Create"
Possibility 2: From the Workload Snapshot list
Login to the RHV-Manager
Navigate to the Backup Tab
Identify the workload that shall create a Snapshot
Click the workload name to enter the Workload overview
Navigate to the Snapshots tab
Click "Create Snapshot"
Provide a name and description for the Snapshot
Decide between Full and Incremental Snapshot
Click "Create"
Snapshot overview
Each Snapshot contains a lot of information about the backup. These information can be seen in the Snapshot overview.
To reach the Snapshot Overview follow these steps:
Login to the RHV-Manager
Navigate to the Backup Tab
Identify the workload that contains the Snapshot to show
Click the workload name to enter the Workload overview
Navigate to the Snapshots tab
Identify the searched Snapshot in the Snapshot list
Click the Snapshot Name
Details Tab
The Snapshot Details Tab shows the most important information about the Snapshot.
Snapshot Name / Description
Snapshot Type
Time Taken
Size
Which VMs are part of the Snapshot
for each VM in the Snapshot
Instance Info - Name & Status
Instance Type - vCPUs, Disk & RAM
Restores Tab
The Snapshot Restores Tab shows the list of Restores that have been started from the chosen Snapshot.
It is possible to start Restores from here.
Please refer to the Restores User Guide to learn more about Restores.
Misc. Tab
The Snapshot Miscellaneous Tab provides the remaining metadata information about the Snapshot.
Creation Time
Last Update time
Snapshot ID
Workload ID of the Workload containing the Snapshot
Delete Snapshots
Once a Snapshot is no longer needed, it can be safely deleted from a Workload.
The retention policy will automatically delete the oldest Snapshots according to the configure policy.
You have to delete all Snapshots to be able to delete a Workload.
Deleting a TrilioVault Snapshot will not delete any RHV Snapshots. Those need to be deleted separately if desired.
There are 2 possibilities to delete a Snapshot.
Possibility 1: Single Snapshot deletion through the submenu
To delete a single Snapshot through the submenu follow these steps:
Login to the RHV-Manager
Navigate to the Backup Tab
Identify the workload that contains the Snapshot to show
Click the workload name to enter the Workload overview
Navigate to the Snapshots tab
Identify the searched Snapshot in the Snapshot list
Click the small arrow in the line of the Snapshot next to "One Click Restore" to open the submenu
Click "Delete Snapshot"
Confirm by clicking "Delete"
Possibility 2: Multiple Snapshot deletion through checkbox in Snapshot overview
To delete one or more Snapshots through the Snapshot overview do the following:
Login to the RHV-Manager
Navigate to the Backup Tab
Identify the workload that contains the Snapshot to show
Click the workload name to enter the Workload overview
Navigate to the Snapshots tab
Identify the searched Snapshots in the Snapshot list
Check the checkbox for each Snapshot that shall be deleted
Click "Delete Snapshots"
Confirm by clicking "Delete"
Python Celery requires a message broker system like RabbitMQ or Redis.
TrilioVault uses the Redis message broker.
RHV does not include Redis, so installation is necessary.
Redis is not available from a Red Hat repository yet. The Fedora EPEL repository provides the needed packages.
The following steps install Redis:
Add the Fedora EPEL Repository# yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
Install Redis
# yum install redis
Start Redis
# systemctl start redis.service
Enable Redis to start on boot
# systemctl enable redis
Check Redis status
# systemctl status redis.service
Enabling Disk Upload through RHV Manager
Trilio delivers the TrilioVault Appliance as a qcow2 image.
Trilio supports that the TrilioVault Appliance is running on the same RHV Cluster it protects.
Uploading the qcow2 to the RHV Datastore is easy, but depending on the RHV usage so far, it might require the installation of additional certificates.
Verify the connection to the ovirt-imageio-proxy
RHV is using the ovirt-imageio-proxy service to upload and download images, snapshots, and disks through the RHV Manager.
The following steps verify the connection to the ovirt-imageio-proxy service.
Log in to the administrative portal of the RHV Manager
Go to Storage ➡️ Disks
Go to Upload ➡️ Start
Click Test Connection
When the connection test is unsuccessful, please proceed with the necessary steps to install the ovirt-engine-certificates.
When the connection test is successful, no further steps are required to upload the image.
Install the ovirt-engine certificate
When the Test-Connection to the ovirt-imageio-proxy is failing, a usual reason is the client system mistrusting the RHV-M due to a missing certificate.
The RHV-M does have two certificates, which can both be required to access the ovirt-imageio-proxy.
The first certificate is directly available for download from the error message in the window. The below URL shows the general path to the certificate. The downloaded certificate is the root certificate for the certificates used by the ovirt-imageio-proxy.
Download and install this certificate according to the clients operating system and browser used.
Test the connection to the ovirt-imageio-proxy again after installation.
Proceed to the second certificate only in case of the connection still failing.
The second certificate is the actual certificate of the ovirt-imageio-proxy shown to the client system upon connection. The download is only possible from the RHV-M host system directly. The usual location of the certificate is:
/etc/pki/ovirt-engine/certs/imageio-proxy.cer
Please visit /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf in case of the certificate being located elsewhere.
Install the certificate according to the client's operating system and browser used.
Test the connection to the ovirt-imageio-proxy again after installation.
Please contact your administrator when the connection still fails.
Uploading the TrilioVault VM qcow2 disk to RHV
The TrilioVault VM qcow2 image is a full operating system disk, that is getting attached to a VM running on the RHV-Cluster.
To be able to spin up the TrilioVault VM, upload the qcow2 disk into the RHV datadomain.
The following procedure uploads the qcow2 disk:
Go to Storage➡️Disk
Go to Upload➡️Start
Fill out the presented form and choose the path to the qcow2 image on the client system
Click OK to start the upload
In case of the upload not starting after several minutes, verify that the connection to the ovirt-imageio-proxy is possible.
Important note for VMs using iSCSI disks.
RHV is only creating a connection between the VM and the iSCSI disk when the VMs are running, as this connecting is achieved through a symlink on the RHV Host.
This behavior leads to RHV only being able to take RHV Snapshots through the VM when the VM is running.
In consequence TVR is only able to take backups while the VM is in a running state.
Attached Networks
Attached Volumes
Misc - Original ID of the VM
Definition
A workload is a backup job that protects one or more Virtual Machines according to a configured policy.
There can be as many workloads as needed.
But each VM can only be part of one Workload.
Create a Workload
To create a workload do the following steps:
Login to the RHV-Manager
Navigate to the Backup Tab
Click "Create Workload"
Provide Workload Name and Workload Description on the first tab "Details"
Choose between Serial or Parallel workload on the first tab "Details"
Choose the VMs to protect on the second Tab "Workload Members"
Decide for the schedule of the workload on the Tab "Schedule"
Provide the Retention policy on the Tab "Policy"
Choose the Full Backup Interval on the Tab "Policy"
Click create
The created Workload will be available after a few seconds and starts to take backups according to the provided schedule and policy.
Workload Overview
A workload contains many information, which can be seen in the workload overview.
To enter the workload overview do the following steps:
Login to the RHV-Manager
Navigate to the Backup Tab
Identify the workload that shall create a Snapshot
Click the workload name to enter the Workload overview
Details Tab
The Workload Details tab provides you with the general most important information about the workload:
Name
Description
List of protected VMs
It is possible to navigate to the protected VM directly from the list of protected VMs.
Snapshots Tab
The Workload Snapshots Tab shows the list of all available Snapshots in the chosen Workload.
From here it is possible to work with the Snapshots, create Snapshots on demand and start Restores.
Please refer to the Snapshot and Restore User Guide to learn more about those.
Policy Tab
The Workload Policy Tab shows gives an overview of the current configured scheduler and retention policy. The following elements are shown:
Scheduler Enabled / Disabled
Start Date / Time
End Date / Time
RPO
Time till next Snapshot run
Retention Policy and Value
Full Backup Interval policy and value
Filesearch Tab
The Workload Filesearch Tab provides access to the power search engine, which allows to find files and folders on Snapshots without the need of a restore.
Please refer to the File Search User Guide to learn more about this feature.
Misc. Tab
The Workload Miscellaneous Tab shows the remaining metadata of the Workload.
The following information are provided:
Creation time
last update time
Workload ID
Workload Type
Edit a Workload
Workloads can be modified in all components to match changing needs.
To edit a workload do the following steps:
Login to the RHV-Manager
Navigate to the Backup Tab
Identify the workload to be modified
Click the small arrow next to "Create Snapshot" to open the sub-menu
Click "Edit Workload"
Modify the workload as desired - All parameters can be changed
Click "Update"
Delete a Workload
Once a workload is no longer needed it can be safely deleted.
To delete a workload do the following steps:
All Snapshots need to be deleted before the workload gets deleted. Please refer to the Snapshots User Guide to learn how to delete Snapshots.
Login to the RHV-Manager
Navigate to the Backup Tab
Identify the workload to be deleted
Click the small arrow next to "Create Snapshot" to open the sub-menu
Click "Delete Workload"
Confirm by clicking "Delete Workload" yet again
Reset a Workload
In rare cases it might be necessary to start a backup chain all over again, to ensure the quality of the created backups.
To not recreate a Workload in such cases is it possible to reset a Workload.
The Workload reset will:
Cancel all ongoing tasks
Delete all existing RHV Snapshots from the protected VMs
recalculate the next Snapshot time
take a full backup at the next Snapshot
To reset a Workload do the following steps:
Login to the RHV-Manager
Navigate to the Backup Tab
Identify the workload to be deleted
Click the small arrow next to "Create Snapshot" to open the sub-menu
Click "Reset Workload"
Confirm by clicking "Reset Workload" yet again
Post Installation Health-Check
After the installation and configuration of TrilioVault for RHV did succeed the following steps can be done to verify that the TrilioVault installation is healthy.
Verify the TrilioVault Appliance services are up
TrilioVault is using 3 main services on the TrilioVault Appliance:
wlm-api
wlm-scheduler
wlm-workloads
Those can be verified to be up and running using the systemctl status command.
Check the TrilioVault pacemaker and nginx cluster
The second component to check the TrilioVault Appliance's health is the nginx and pacemaker cluster.
Verify API connectivity from the RHV-Manager
The RHV-Manager is doing all API calls towards the TrilioVault Appliance.
Therefore it is helpful to do a quick API connectivity check using curl.
The following curl command lists the available workload-types and verfifies that the connection is available and working:
Verify the ovirt-imageio services are up and running
TrilioVault is extending the already exiting ovirt-imageio services.
The installation of these extensions does check if the ovirt-services come up.
Still it is a good call to verify again afterwards:
On the RHV-Manager check the ovirt-imageio-proxy service:
On the RHV-Host check the ovirt-imageio-daemon service:
Verify the NFS Volume is correctly mounted
TrilioVault mounts the NFS Backup Target to the TrilioVault Appliance and RHV-Hosts.
To verify those are correctly mounted it is recommended to do the following checks.
First df -h looking for /var/triliovault-mounts/<hash-value>
Secondly do a read / write / delete test as the user vdsm:kvm (uid = 36 / gid = 36) from the TrilioVault Appliance and the RHV-Host.
Configure TrilioVault VM
The TrilioVault Appliance requires configuration to work with the chosen RHV environment.
A Web-UI provides access to the TrilioVault Appliance dashboard and configurator.
Recommended and tested browsers are: Chrome and Firefox.
Accessing the TrilioVault Dashboard
Enter the TrilioVault IP or FQDN into the browser to reach the TrilioVault Appliance landing page.
The user is: admin
The initial password is: password
After the first login into the TrilioVault Dashboard is it necessary to change the password.
Details needed for the TrilioVault Appliance configurator
Upon login into the TrilioVault Appliance, the shown page is the configurator.
The configurator requires some information about the TrilioVault Appliance, RHV and Backup Storage.
TrilioVault Nodes information
The TrilioVault Appliance needs to be integrated into an existing environment to be able to operate correctly.
This block asks for the information about the TrilioVault Appliance operating details.
Virtual IP Address
The TrilioVault Appliance uses this IP address for all communication with RHV.
Format: IP/Netmask
The TrilioVault Appliance for RHV does not yet support multi-node installations. It is an actively worked on feature, that gets integrated step by step.
TVM Appliance IP
The first interface in the interface list of the TrilioVault Appliance will get this IP address assigned. Further is the TrilioVault Appliance hostname set.
Format: IP=hostname
The Virtual IP and the TVM Appliance IP can not be the same address. The configuration fails upon using the same IPs for both values.
Name Servers
The DNS server the TrilioVault appliance will use.
Format: Comma separated list of IPs
RHV Credentials information
The TrilioVault appliance integrates with one RHV environment.
This block asks for the information required to access and connect with the RHV Cluster.
RHV Engine URL
URL of the RHV-Manager used to authenticate
Format: URL (FQDN and IP supported)
A preconfigured DNS Server is required, when using FQDN. The TrilioVault Appliance local host file gets overwritten during configuration. The configuration will fail when the FQDN is not resolvable by a DNS Server.
RHV Username
admin-user to authenticate against the RHV-Manager
Format: user@domain
The configurator verifies the credentials entered. The shown error is always Invalid Credentials in case of any error.
Backup Storage Configuration information
This block asks for the necessary details to configure the Backup Storage.
Backup Storage
Predefined as NFS
TrilioVault for RHV currently only supports NFS. The addition of S3 compatible Storage solutions gets delivered in a future version.
NFS Export
Full path to the NFS Volume used as Backup Storage
Format: Comma separated list of NFS paths
TrilioVault Certificate information
TrilioVault is integrating into the RHV Cluster as an additional service, following the RHV communication paradigms. These require that the TrilioVault Appliance is using SSL and that the RHV-Manager does trust the TrilioVault Appliance.
TrilioVault offers to possibilities how these required certificates can be provided.
Either TrilioVault generates a complete fresh self-signed certificate or a certificate is provided.
In both cases is the FQDN required, to which the certificate is pointing to.
Please see below example in case of a provided certificate.
FQDN
FQDN to reach the TrilioVault Appliance
Format: FQDN
TrilioVault License
It is possible to directly provide the TrilioVault Appliance with the license file that is going to be used by it.
TrilioVault will not create any workloads or backups without a valid license file.
It is not necessary to provide the License file directly through the configurator.
It is also possible to provide the license afterwards through the TrilioVault License tab in the TrilioVault dashboard.
The TrilioVault License tab can also be used to verify and update the currently installed license.
Submit and Configuration
After filling out every block of the configurator, hit the submit button to start the configuration.
The configurator asks one more time for confirmation before starting.
Stay patient during the configuration, as this may easily take a few more minutes.
After the configurator has succeeded or failed, is the Ansible playbook shown. Use the possibilities to expand and collapse each task for troubleshooting failed configurations.
Restores
Definition
A Restore is the workflow to bring back the backed up VMs from a TrilioVault Snapshot.
TrilioVault does offer 3 types of restores:
One Click restore
Selective restore
InPlace restore
One Click Restore
The One Click Restore will bring back all VMs from the Snapshot in the same state as they were backed up.
They will:
be located in the same cluster in the same datacenter
use the same storage domain
connect to the same network
The user can't change any Metadata.
The One Click Restore requires, that the original VM's that have been backed up are deleted or otherwise lost. If even one VM is still existing, will the One Click Restore fail.
The One Click Restore will automatically update the Workload to protect the restored VMs.
There are 2 possibilities to start a One Click Restore.
Possibility 1: From the Snapshot list
Login to the RHV-Manager
Navigate to the Backup Tab
Identify the workload that contains the Snapshot to be restored
Possibility 2: From the Snapshot overview
Login to the RHV-Manager
Navigate to the Backup Tab
Identify the workload that contains the Snapshot to be restored
Selective Restore
The Selective Restore is the most complex restore TrilioVault has to offer.
It allows to adapt the restored VMs to the exact needs of the User.
With the selective restore the following things can be changed:
Which VMs are getting restored
Name of the restored VMs
Which networks to connect with
The Selective Restore is always available and doesn't have any prerequirements.
There are 2 possibilities to start a Selective Restore.
Possibility 1: From the Snapshot list
Login to the RHV-Manager
Navigate to the Backup Tab
Identify the workload that contains the Snapshot to be restored
Possibility 2: From the Snapshot overview
Login to the RHV-Manager
Navigate to the Backup Tab
Identify the workload that contains the Snapshot to be restored
Inplace Restore
The Inplace Restore covers those use cases, where the VM and its Volumes are still available, but the data got corrupted or needs to a rollback for other reasons.
It allows the user to restore only the data of a selected Volume, which is part of a backup.
The Inplace Restore only works when the original VM and the original Volume are still available and connected. TrilioVault is checking this by the saved Object-ID.
The Inplace Restore will not create any new RHV resources. Please use one of the other restore options if new Volumes or VMs are required.
There are 2 possibilities to start an Inplace Restore.
Possibility 1: From the Snapshot list
Login to the RHV-Manager
Navigate to the Backup Tab
Identify the workload that contains the Snapshot to be restored
Possibility 2: From the Snapshot overview
Login to the RHV-Manager
Navigate to the Backup Tab
Identify the workload that contains the Snapshot to be restored
Example: 10.10.0.2/24
Example: 10.10.0.1=rhv-tvm
Example: 8.8.8.8,10.10.10.10
Domain Search Order
The domain the TrilioVault Appliance will use.
Format: Comma separated list of domain names
Example: trilio.demo,trilio.io
NTP Servers
NTP Servers the TrilioVault Appliance will use.
Format: Comma separated list of NTP Servers (FQDN and IP supported)
Example: 0.pool.ntp.org,10.10.10.10
Timezone
Timezone the TrilioVault will use.
Format: predefined list
Example: UTC
Example: https://rhv-manager.trilio.demo
Example: admin@internal
Password
The password to validate the RHV Username against the RHV-Manager
Format: String
Example: password
Example: 10.10.100.20:/rhv_backup
NFS Options
Options used by the TrilioVault NFS client to connect to the NFS Volume
Format: NFS Options
Example: nolock,soft,timeo=180,intr
Example: rhv-tvm.trilio.demo
Certificate
Certificate provided by the TrilioVault appliance upon request
Format: Certificate file
Example: rhv-tvm.crt
Private Key
Private Key used to verify the provided certificate
Format: private key file
Example: rhv-tvm.key
have the same flavor
Click the workload name to enter the Workload overview
Navigate to the Snapshots tab
Identify the Snapshot to be restored
Click "One Click Restore" in the same line as the identified Snapshot
(Optional) Provide a name / description
Click "Create"
Click the workload name to enter the Workload overview
Navigate to the Snapshots tab
Identify the Snapshot to be restored
Click the Snapshot Name
Navigate to the "Restores" tab
Click "One Click Restore"
(Optional) Provide a name / description
Click "Create"
Which Storage domain to use
Which DataCenter / Cluster to restore into
Which flavor the restored VMs will use
Click the workload name to enter the Workload overview
Navigate to the Snapshots tab
Identify the Snapshot to be restored
Click on the small arrow next to "One Click Restore" in the same line as the identified Snapshot
Click on "Selective Restore"
Configure the Selective Restore as desired
Click "Restore"
Click the workload name to enter the Workload overview
Navigate to the Snapshots tab
Identify the Snapshot to be restored
Click the Snapshot Name
Navigate to the "Restores" tab
Click "Selective Restore"
Configure the Selective Restore as desired
Click "Restore"
Click the workload name to enter the Workload overview
Navigate to the Snapshots tab
Identify the Snapshot to be restored
Click on the small arrow next to "One Click Restore" in the same line as the identified Snapshot
Click on "Inplace Restore"
Configure the Inplace Restore as desired
Click "Restore"
Click the workload name to enter the Workload overview
Navigate to the Snapshots tab
Identify the Snapshot to be restored
Click the Snapshot Name
Navigate to the "Restores" tab
Click "Inplace Restore"
Configure the Inplace Restore as desired
Click "Restore"
systemctl status wlm-api
######
● wlm-api.service - Cluster Controlled wlm-api
Loaded: loaded (/etc/systemd/system/wlm-api.service; disabled; vendor preset: disabled)
Drop-In: /run/systemd/system/wlm-api.service.d
└─50-pacemaker.conf
Active: active (running) since Wed 2020-04-22 09:17:05 UTC; 1 day 2h ago
Main PID: 21265 (python)
Tasks: 1
CGroup: /system.slice/wlm-api.service
└─21265 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-api --config-file=/etc/workloadmgr/workloadmgr.conf
systemctl status wlm-scheduler
######
● wlm-scheduler.service - Cluster Controlled wlm-scheduler
Loaded: loaded (/etc/systemd/system/wlm-scheduler.service; disabled; vendor preset: disabled)
Drop-In: /run/systemd/system/wlm-scheduler.service.d
└─50-pacemaker.conf
Active: active (running) since Wed 2020-04-22 09:17:17 UTC; 1 day 2h ago
Main PID: 21512 (python)
Tasks: 1
CGroup: /system.slice/wlm-scheduler.service
└─21512 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-scheduler --config-file=/etc/workloadmgr/workloadmgr.conf
systemctl status wlm-workloads
######
● wlm-workloads.service - workloadmanager workloads service
Loaded: loaded (/etc/systemd/system/wlm-workloads.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2020-04-22 09:15:43 UTC; 1 day 2h ago
Main PID: 20079 (python)
Tasks: 33
CGroup: /system.slice/wlm-workloads.service
├─20079 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf
├─20180 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf
[...]
├─20181 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf
├─20233 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf
├─20236 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf
└─20237 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf
pcs status
######
Cluster name: triliovault
WARNINGS:
Corosync and pacemaker node names do not match (IPs used in setup?)
Stack: corosync
Current DC: om_tvm (version 1.1.19-8.el7_6.1-c3c624ea3d) -
partition with quorum
Last updated: Wed Dec 5 12:25:02 2018
Last change: Wed Dec 5 09:20:08 2018 by root via cibadmin on om_tvm
1 node configured
4 resources configured
Online: [ om_tvm ]
Full list of resources:
virtual_ip (ocf::'heartbeat:IPaddr2): Started om_tvm
wlm-api (systemd:wlm-api): Started om_tvm
wlm-scheduler (systemd:wlm-scheduler): Started om_tvm
Clone Set: lb_nginx-clone [lb_nginx]
Started: [ om_tvm ]
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
