Backup-as-a-Service

Trilio is an add on service to OpenStack cloud infrastructure and provides backup and disaster recovery functions for tenant workloads. Trilio is very similar to other openstack services including nova, cinder, glance, etc and adheres to all tenets of OpenStack. It is a stateless service that scales with your cloud.

Main Components

Trilio has four main software components:

1. Trilio ships as a QCOW2 image. User can instantiate one or more VMs from the QCOW2 image on a standalone KVM boxes.

2. Trilio API is a python module that is an extension to nova api service. This module is installed on all OpenStack controller nodes

3. Trilio Datamover is a python module that is installed on every OpenStack compute nodes

4. Trilio horizon plugin is installed as an add on to horizon servers. This module is installed on every server that runs horizon service.

Service Endpoints

Trilio is both a provider and consumer into OpenStack ecosystem. It uses other OpenStack services such as nova, cinder, glance, neutron, and keystone and provides its own service to OpenStack tenants. To accomodate all possible OpenStack deployments, Trilio can be configured to use either public or internal URLs of services. Likewise Trilio provides its own public, internal and admin URLs.

Network Topology

This figure represents a typical network topology. Trilio exposes its public URL endpoint on public network and Trilio virtual appliances and data movers typically use either internal network or dedicated backup network for storing and retrieving backup images from backup store.

The Trilio Appliance is delivered as qcow2 image and runs as VM on top of a KVM Hypervisor.

Creating the cloud-init image

The Trilio appliance is utilizing cloud-init to provide the initial network and user configuration.

Cloud-init is reading it's information either from a metadata server or from a provided cd image. Trilio is utilizing the cd image.

Needed tools

To create the cloud-init image it is required to have genisoimage available.

#For RHEL and centos
yum install genisoimage
#For Ubuntu 
apt-get install genisoimage

Providing the Metadata

Cloud-init is using two files for it's metadata.

The first file is called meta-data and contains the information about the network configuration. Below is an example of this file.

[root@kvm]# cat meta-data
instance-id: triliovault
network-interfaces: |
   auto ens3
   iface ens3 inet static
   address 158.69.170.20
   netmask 255.255.255.0
   gateway 158.69.170.30

   dns-nameservers 11.11.0.51
local-hostname: tvault-controller

The second file is called user-data and contains little scripts and information to set up for example the user passwords. Below is an example of this file.

[root@kvm]# cat user-data
#cloud-config
chpasswd:
  list: |
    root:password1
    stack:password2
  expire: False

creating the image file

The image is getting created using genisoimage follwing this general command:

genisoimage -output <name>.iso -volid cidata -joliet -rock </path/user-data> </path/meta-data>

An example of this command is shown below.

genisoimage  -output tvault-firstboot-config.iso -volid cidata -joliet -rock user-data meta-data

Spining up the Trilio appliance

After the cloud-init image has been created the TriloVault appliance can be spun up on the desired KVM server.

See below an example command, how to spin up the Trilio appliance using virsh and the created iso image.

virt-install -n triliovault-vm  --memory 24576 --vcpus 8 \
--os-type linux \ 
--disk tvault-appliance-os-3.0.154.qcow2,device=disk,bus=virtio,size=40 \
--network bridge=virbr0,model=virtio \
--network bridge=virbr1,model=virtio \
--graphics none \
--import \
--disk path=tvault-firstboot-config.iso,device=cdrom

Uninstalling cloud-init after first boot

Once the Trilio appliance is up and running with it's initial configuration is it recommended to uninstall cloud-init.

To uninstall cloud-init, follow the example below.

sudo apt-get purge cloud-init

Once the Trilio VM or the Cluster of Trilio VMs has been spun, the actual installation process can begin. This process contains of the following steps:

1. Install the Trilio dm-api service on the control plane

2. Install the Trilio datamover service on the compute plane

3. Install the Trilio Horizon plugin into the Horizon service

How these steps look in detail is depending on the Openstack distribution Trilio is installed in. Each supported Openstack distribution has its own deployment tools. Trilio is integrating into these deployment tools to provide a native integration from the beginning to the end.

Trilio has four main software components:

1. Trilio ships as a QCOW2 image. User can instantiate one or more VMs from the QCOW2 image on a standalone KVM boxes.

2. Trilio API is a python module that is an extension to nova api service. This module is installed on all OpenStack controller nodes

3. Trilio Datamover is a python module that is installed on every OpenStack compute nodes

4. Trilio horizon plugin is installed as an add on to horizon servers. This module is installed on every server that runs horizon service.

System requirements Trilio Appliance

The Trilio Appliance gets delivered as a qcow2 image, which gets attached to a virtual machine.

Trilio supports KVM-based hypervisors on x86 architectures, with the following properties:

The recommended size of the VM for the Trilio Appliance is:

The qcow2 image itself defines the 40GB disk size of the VM.

2.3. Software Requirements¶

In addition to the Trilio Appliance does Trilio contain components, which are installed directly into the Openstack itself.

Each Openstack distribution comes with a set of supported operating systems. Please check the support matrix to see, which Openstack Distribution is supported with which Operating System.

Additional it is necessary to have the nfs-common packages installed on the compute nodes in case of using the NFS protocol for the backup target.

Trilio configuration process is using ansible scripts. Ansible, in the last few years, has grown in popularity as a preferred configuration management tool and Trilio uses ansible play books extensively to configure the Trilio cluster. To troubleshoot Trilio configuration issues, user should have basic understanding of ansible playbook output.

Ansible modules are inherently idempotent and hence Trilio configuration can run any number of times to change or reconfigure Trilio cluster.

Once the VM is booted, point your browser (Chrome or Firefox) to Trilio node IP address.

This will bring you to the Trilio Dashboard, which contains the Trilio configurator.

The user is: admin The default password is: password

Unlike previous versios of Trilio, the current version only requires you to configure the cluster once and the Trilio dashboard provides cluster wide management capability.

Details needed for the Trilio Appliance

Upon login into an unconfigured Trilio Appliance, the shown page is the configurator. The configurator requires some information about the Trilio Appliance, Openstack and Backup Storage.

Trilio Cluster information

The Trilio Cluster needs to be integrated into an existing environment to be able to operate correctly. This block asks for the information about the Trilio Cluster operating details.

• Controller Nodes

  • This is the list of Trilio virtual appliance IP addresses along with their hostnames.

  • Format: comma separated list with pairs combined through '='

  • Example: 172.20.4.151=tvault-104-1,172.20.4.152=tvault-104-2,172.20.4.153=tvault-104-3’

• Virtual IP Address

  • This is the Trilio cluster IP address which is mandatory

  • Format: IP/Subnet

  • Example: 172.20.4.150/24

• Name Server

  • List of nameservers, primarily used to resolve OpenStack service endpoints.

  • Format: comma separated list

  • example: 8.8.8.8,172.20.4.1

• Domain Search Order

  • The domain the Trilio Cluster will use.

  • Format: comma separated list

  • example: trilio.io,trilio.demo

• NTP Servers

  • NTP servers the Trilio Cluster will use

  • format: comma separated list

  • example: 0.pool.ntp.org,10.10.10.10

• Timezone

  • Timezone the Trilio Cluster will use internally

  • format: pre-populated list

  • example: UTC

Openstack Credentials informations

The Trilio appliance integrates with one RHV environment. This block asks for the information required to access and connect with the RHV Cluster.

• Keystone Admin URL

  • The Keystone admin endpoint mainly used during configuration

  • format: URL

  • example: https://keystone.trilio.io:35357/v3

• Keystone Public/Internal URL

  • The URL type defines which endpoint type will communicate with the Openstack endpoints

  • format: URL

  • example: https://internal.trilio.io:5000/v3

• Administrator

  • Username of an account with the domain admin role

  • format: String

  • example: admin

• Password

  • password for the user provided before

  • format: String

  • example: password

• Admin Tenant

  • The tenant to be used together with the provided user

• Region

  • Openstack Region the user and tenant are located in

  • format: String

  • example: RegionOne

• Domain ID

  • domain the provided user and tenant are located in

  • format: ID

  • exmaple: default

• Trustee Role

  • The Openstack role required to be able to use Trilio functionalities

Backup Storage Configuration information

This block is requesting the necessary information about the backup target that the Trilio installation will be used to store and read backups.

The very first field in this block decides the protocol used to connect with Backup Storage, NFS or S3.

Using the NFS protocol

• NFS Export

  • Path under which the NFS Volumes to be used can be found

  • format: comma separated list of NFS Volumes paths

  • example: 10.10.2.20:/upstream,10.10.5.100:/nfs2

• NFS Options

  • NFS options used by the Trilio Cluster when mounting the NFS Exports

  • format: NFS options

  • example: nolock,soft,timeo=180,intr,lookupcache=none

Using the S3 protocol

• S3 Compatible

  • Switch between Amazon and Ceph

  • format: predefined list

  • example: Amazon S3

• Access Key

  • Access Key necessary to login into the S3 storage

  • format: access key

  • example: SFHSAFHPFFSVVBSVBSZRF

• Secret Key

  • Secret Key necessary to login into the S3 storage

  • format: secret key

  • example: bfAEURFGHsnvd3435BdfeF

• Region

  • Configured Region for the S3 Bucket

  • format: String

  • example: us-east

• Bucket Name

  • Name of the bucket to be used as Backup target

  • format: string

  • example: Trilio-backup

• (CEPH S3 ONLY) Endpoint URL

  • URL to be used to reach and access the provided S3 compatible storage

  • format: URL

  • example: objects.trilio.io

Workload Import

If you are upgrading either from older versions of Trilio or reinstalling the appliance for maintenance reasons, please check this box during the configuration. Trilio is a stateless appliance and all the state is securely saved on the NFS/S3 storage. So, during the upgrade process, the user will need to import all backup job records back from the NFS/S3 storage to the appliance MySQL database. By checking this box, the configuration automatically imports all backup records.

TLS and certificates

Starting the configurator

Once all entries have been set and all validations are error free the configurator can be started.

• Click Finish

• Reconfirm in the pop-up that you want to start the configuration

• Wait for the configurator to finish

The configurator is using Ansible and a few Trilio internal API calls. After each configuration block or after the configurator finished it is possible to visit the ansible output.

At the end of a successful configuration are you forwarded to the VIP provided during configuration.

Installing Trilio Nova API Extension on Controller Node

The next step in the installation process is to install the Trilio Nova API extension on each of the Nova API controller nodes. The purpose of the API extension is to route RESTful API calls to a Trilio Data Mover (which will be discussed next).

To install the Trilio Nova Extension on each Nova controller node, follow these steps after logging into a controller node.

Once the script has been started does it prompt for the following information:

1. Trilio IP This IP will be used to download the Trilio Nova API Extension pip packages.

2. Request whether the script is running on a Controller or Compute Node To install the Trilio Nova API Extension choose Controller

The Trilio Nova API extension is a download from Trilio appliance and installed on the controller node. After installation, the Nova-API service is restarted to load the Trilio Nova API extension into Nova-API process so Nova-API now can handle Data Mover requests from the Trilio appliance.

Installing Trilio Data Mover on Computer Node

Now it is time to install the Trilio Data Mover service on the compute nodes where the Nova Compute service is running. The purpose of the Data Mover service is, as the name says, to move the actual disk data to and from the backup target.

To install the Trilio Data Mover on a Compute node, follow these steps after logging into a compute node.

At this point it is possible to run the installation either in an automated fashion or in an interactive mode.

Interactive install of the Trilio Data Mover

The following command will start the interactive installation.

Once the script has been started does it prompt for the following information:

1. Trilio IP This IP will be used to download the Trilio Nova API Extension pip packages.

2. Request whether the script is running on a Controller or Compute Node To install the Trilio Data Mover service choose Compute Node.

3. Request of the path to the compute.filters file Choose matching your distribution or provide the exact path.

4. Acceptance of Trilio changing the sudoers file for nova nova ALL = (root) NOPASSWD: /home/tvault/.virtenv/bin/privsep-helper *

5. Choose between NFS or S3 as backup target

6. In case of NFS

   1. Provide the NFS Volume path

   2. check and change the default NFS options if necessary

7. In case of S3

   1. Choose the S3 profile to be used

   2. provide access credentials according to the chosen profile

The Trilio Data Mover can be installed in an unattended way using a configuration file. The configuration file contains all the data that is required while the script is being executed.

The first step for the automated installation is to generate the answers file. A predefined file can be downloaded from any Trilio VM.

Edit tvault-contego-install.answers file as required for compute nodes. An example can be seen below:

Once the answersfile has been created can it be used to run the Data Mover installation unattended, which allows the automation of the installation process.

Start the Trilio Data Mover service

At the end of the installation is the Data Mover service deactivated.

To start the service using the installation script run:

The last component to install is the Trilio Horizon Plugin on the Controllers where the Openstack Horizon service is running. The purpose of the Horizon Plugin is, to provide the native integration of TrlioVault into the GUI of Openstack.

To install the Trilio Horizon Plugin on each Openstack Horizon controller node, follow these steps after logging into a controller node.

To upgrade the Trilio application to the new available version follow these steps.

1. Delete all instances of Trilio appliances

2. Spin up the latest Trilio Appliances

3. Install latest data mover on compute, nova api extension and Horizon plugin on controller nodes

4. Configure the Trilio Cluster

5. While configuring Trilio controller, check “import workloads” checkbox, this will import all old workload records from backup repository to new appliance.

After the installation and configuration of Trilio for Openstack did succeed the following steps can be done to verify that the Trilio installation is healthy.

Verify the Trilio Appliance services are up

Trilio is using 3 main services on the Trilio Appliance:

• wlm-api

• wlm-scheduler

• wlm-workloads

Those can be verified to be up and running using the systemctl status command.

systemctl status wlm-api
######
● wlm-api.service - Cluster Controlled wlm-api
   Loaded: loaded (/etc/systemd/system/wlm-api.service; disabled; vendor preset: disabled)
  Drop-In: /run/systemd/system/wlm-api.service.d
           └─50-pacemaker.conf
   Active: active (running) since Wed 2020-04-22 09:17:05 UTC; 1 day 2h ago
 Main PID: 21265 (python)
    Tasks: 1
   CGroup: /system.slice/wlm-api.service
           └─21265 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-api --config-file=/etc/workloadmgr/workloadmgr.conf

systemctl status wlm-scheduler
######
● wlm-scheduler.service - Cluster Controlled wlm-scheduler
   Loaded: loaded (/etc/systemd/system/wlm-scheduler.service; disabled; vendor preset: disabled)
  Drop-In: /run/systemd/system/wlm-scheduler.service.d
           └─50-pacemaker.conf
   Active: active (running) since Wed 2020-04-22 09:17:17 UTC; 1 day 2h ago
 Main PID: 21512 (python)
    Tasks: 1
   CGroup: /system.slice/wlm-scheduler.service
           └─21512 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-scheduler --config-file=/etc/workloadmgr/workloadmgr.conf

systemctl status wlm-workloads
######
● wlm-workloads.service - workloadmanager workloads service
   Loaded: loaded (/etc/systemd/system/wlm-workloads.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2020-04-22 09:15:43 UTC; 1 day 2h ago
 Main PID: 20079 (python)
    Tasks: 33
   CGroup: /system.slice/wlm-workloads.service
           ├─20079 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf
           ├─20180 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf
           [...]
           ├─20181 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf
           ├─20233 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf
           ├─20236 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf
           └─20237 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf

Check the Trilio pacemaker and nginx cluster

The second component to check the Trilio Appliance's health is the nginx and pacemaker cluster.

pcs status
######
Cluster name: triliovault

WARNINGS:
Corosync and pacemaker node names do not match (IPs used in setup?)
Stack: corosync
Current DC: om_tvm (version 1.1.19-8.el7_6.1-c3c624ea3d) -
partition with quorum
Last updated: Wed Dec 5 12:25:02 2018
Last change: Wed Dec 5 09:20:08 2018 by root via cibadmin on om_tvm
1 node configured
4 resources configured

Online: [ om_tvm ]
Full list of resources:
virtual_ip (ocf::'heartbeat:IPaddr2): Started om_tvm
wlm-api (systemd:wlm-api): Started om_tvm
wlm-scheduler (systemd:wlm-scheduler): Started om_tvm
Clone Set: lb_nginx-clone [lb_nginx]
Started: [ om_tvm ]
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled

Verify API connectivity of the Trilio Appliance

Checking the availability of the Trilio API on the chosen endpoints is recommended.

The following example curl command lists the available workload-types and verifies that the connection is available and working:

curl http://10.10.2.34:8780/v1/8e16700ae3614da4ba80a4e57d60cdb9/workload_types/detail -X GET -H "X-Auth-Project-Id: admin" -H "User-Agent: python-workloadmgrclient" -H "Accept: application/json" -H "X-Auth-Token: gAAAAABe40NVFEtJeePpk1F9QGGh1LiGnHJVLlgZx9t0HRrK9rC5vqKZJRkpAcW1oPH6Q9K9peuHiQrBHEs1-g75Na4xOEESR0LmQJUZP6n37fLfDL_D-hlnjHJZ68iNisIP1fkm9FGSyoyt6IqjO9E7_YVRCTCqNLJ67ZkqHuJh1CXwShvjvjw

Verify the tvault-contego services are up and running

The tvault-contego service is the Data Mover that got installed on all compute nodes nodes. It is recommended to check its status after the installation.

openstack compute service list
+----+----------------------+--------------------+----------+----------+-------+----------------------------+
| ID | Binary               | Host               | Zone     | Status   | State | Updated At                 |
+----+----------------------+--------------------+----------+----------+-------+----------------------------+
|  7 | nova-conductor       | upstreamcontroller | internal | enabled  | up    | 2020-06-12T09:13:55.000000 |
|  8 | nova-scheduler       | upstreamcontroller | internal | enabled  | up    | 2020-06-12T09:13:54.000000 |
|  9 | nova-consoleauth     | upstreamcontroller | internal | enabled  | up    | 2020-06-12T09:13:52.000000 |
| 10 | nova-compute         | upstreamcompute1   | US-East  | enabled  | up    | 2020-06-12T09:13:50.000000 |
| 11 | nova-compute         | upstreamcompute2   | US-West  | enabled  | up    | 2020-06-12T09:13:51.000000 |
| 12 | nova-contego_3.0.174 | upstreamcompute2   | internal | enabled  | up    | 2020-06-12T09:13:51.000000 |
| 13 | nova-contego_3.0.174 | upstreamcompute1   | internal | enabled  | up    | 2020-06-12T09:13:47.000000 |
+----+---------------------+--------------------+----------+----------+-------+----------------------------+

[root@upstreamcompute1 ~]# systemctl status tvault-contego.service
● tvault-contego.service - Tvault contego
   Loaded: loaded (/etc/systemd/system/tvault-contego.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2020-06-10 10:07:28 EDT; 1 day 19h ago
 Main PID: 10384 (python)
    Tasks: 21
   CGroup: /system.slice/tvault-contego.service
           └─10384 /usr/bin/python /usr/bin/tvault-contego --config-file=/etc...

Jun 12 03:15:33 upstreamcompute1 python[10384]: libvirt: QEMU Driver error :...d
Jun 12 03:15:33 upstreamcompute1 python[10384]: libvirt: QEMU Driver error :...d
Jun 12 03:16:11 upstreamcompute1 python[10384]: libvirt: QEMU Driver error :...d
Jun 12 03:16:31 upstreamcompute1 sudo[13977]:     nova : TTY=unknown ; PWD=/...n
Jun 12 03:16:33 upstreamcompute1 sudo[14004]:     nova : TTY=unknown ; PWD=/ ...
Jun 12 05:15:33 upstreamcompute1 python[10384]: libvirt: QEMU Driver error :...d
Jun 12 05:15:33 upstreamcompute1 python[10384]: libvirt: QEMU Driver error :...d
Jun 12 05:16:11 upstreamcompute1 python[10384]: libvirt: QEMU Driver error :...d
Jun 12 05:16:29 upstreamcompute1 sudo[23356]:     nova : TTY=unknown ; PWD=/...n
Jun 12 05:16:32 upstreamcompute1 sudo[23422]:     nova : TTY=unknown ; PWD=/ ...
Hint: Some lines were ellipsized, use -l to show in full.

Verify the NFS Volume is correctly mounted

Trilio mounts the NFS Backup Target to the Trilio Appliance and Compute nodes.

To verify those are correctly mounted it is recommended to do the following checks.

First df -h looking for /var/triliovault-mounts/<hash-value>

df -h
######
Filesystem                                      Size  Used Avail Use% Mounted on
devtmpfs                                         63G     0   63G   0% /dev
tmpfs                                            63G   16K   63G   1% /dev/shm
tmpfs                                            63G   35M   63G   1% /run
tmpfs                                            63G     0   63G   0% /sys/fs/cgroup
/dev/mapper/rhvh-rhvh--4.3.8.1--0.20200126.0+1  7.1T  3.7G  6.8T   1% /
/dev/sda2                                       976M  198M  712M  22% /boot
/dev/mapper/rhvh-var                             15G  1.9G   12G  14% /var
/dev/mapper/rhvh-home                           976M  2.6M  907M   1% /home
/dev/mapper/rhvh-tmp                            976M  2.6M  907M   1% /tmp
/dev/mapper/rhvh-var_log                        7.8G  230M  7.2G   4% /var/log
/dev/mapper/rhvh-var_log_audit                  2.0G   17M  1.8G   1% /var/log/audit
/dev/mapper/rhvh-var_crash                      9.8G   37M  9.2G   1% /var/crash
30.30.1.4:/rhv_backup                           2.0T  5.3G  1.9T   1% /var/triliovault-mounts/MzAuMzAuMS40Oi9yaHZfYmFja3Vw
30.30.1.4:/rhv_data                             2.0T   37G  2.0T   2% /rhev/data-center/mnt/30.30.1.4:_rhv__data
tmpfs                                            13G     0   13G   0% /run/user/0
30.30.1.4:/rhv_iso                              2.0T   37G  2.0T   2% /rhev/data-center/mnt/30.30.1.4:_rhv__iso

Secondly do a read / write / delete test as the user nova:nova (uid = 36 / gid = 36) from the Trilio Appliance and the RHV-Host.

su nova
######
[nova@tvm MTAuMTAuMi4yMDovdXBzdHJlYW0=]$ touch foo
[nova@tvm MTAuMTAuMi4yMDovdXBzdHJlYW0=]$ ll
total 24
drwxr-xr-x  3 nova nova 4096 Apr  2 17:27 contego_tasks
-rw-r--r--  1 nova nova    0 Apr 23 12:25 foo
drwxr-xr-x  2 nova nova 4096 Apr  2 15:38 test-cloud-id
drwxr-xr-x 10 nova nova 4096 Apr 22 11:00 workload_1540698c-8e22-4dd1-a898-8f49cd1a898c
drwxr-xr-x  9 nova nova 4096 Apr  8 15:21 workload_51517816-6d5a-4fce-9ac7-46ee1e09052c
drwxr-xr-x  6 nova nova 4096 Apr 22 11:30 workload_77fb42d2-8d34-4b8d-bfd5-4263397b636c
drwxr-xr-x  5 nova nova 4096 Apr 23 06:15 workload_85bf16ed-d4fd-49a6-a753-98c5ca6e906b
[nova@tvm MTAuMTAuMi4yMDovdXBzdHJlYW0=]$ rm foo
[nova@tvm MTAuMTAuMi4yMDovdXBzdHJlYW0=]$ ll
total 24
drwxr-xr-x  3 nova nova 4096 Apr  2 17:27 contego_tasks
drwxr-xr-x  2 nova nova 4096 Apr  2 15:38 test-cloud-id
drwxr-xr-x 10 nova nova 4096 Apr 22 11:00 workload_1540698c-8e22-4dd1-a898-8f49cd1a898c
drwxr-xr-x  9 nova nova 4096 Apr  8 15:21 workload_51517816-6d5a-4fce-9ac7-46ee1e09052c
drwxr-xr-x  6 nova nova 4096 Apr 22 11:30 workload_77fb42d2-8d34-4b8d-bfd5-4263397b636c
drwxr-xr-x  5 nova nova 4096 Apr 23 06:15 workload_85bf16ed-d4fd-49a6-a753-98c5ca6e906b

You may want to decommission Trilio as a service for your OpenStack. In this approach, you may not have the need to keep backups that were created and you would like to do complete uninstall of the product.

1. Delete all instances of Trilio.

2. On nova controller node, uninstall tvault-contego-api module.

3. On each nova compute node, uninstall tvault-contego data mover.

4. Uninstall tvault-horizon-plugin from the horizon dashboard node

5. Login as Admin to horizon dashboard and delete triliovault user.

6. You can also clear all the contents from the NFS share or S3 bucket that is configured as backup target for Trilio.

The Red Hat Openstack Platform Director is the supported and recommended method to deploy and maintain any RHOSP installation.

Trilio is integrating natively into the RHOSP Director. Manual deployment methods are not supported for RHOSP.

Prepare for deployment

Depending whether the RHOSP environment is already installed or is getting installed for the first time different steps are done to be able to deploy Trilio.

Overcloud is not yet deployed

If overcloud is not deployed already, in that case user should install trilio rpms on overcloud image before starting deployment. Trilio RPM packages are provided through yum repo hosted on the Trilio VM.

To inject the Trilio yum repository on the overcloud image do the following commands:

##Replace TrilioVault_VM_IP with actual Trilio vm IP
$ cat trilio.repo
[trilio]
name=Trilio Repository
baseurl=http://<TrilioVault_VM_IP>:8085/yum-repo/
enabled=1
gpgcheck=0

$ virt-customize --selinux-relabel -a overcloud-full.qcow2 --upload trilio.repo:/etc/yum.repos.d/
[  0.0] Examining the guest ...
[ 12.0] Setting a random seed
[ 12.0] Copying: opendaylight.repo to /etc/yum.repos.d/
[ 13.0] Finishing off

Afterwards will the overcloud image be created.

virt-customize --selinux-relabel / 
-a overcloud-full.qcow2 / 
--install puppet-triliovault, /
python-workloadmgrclient, /
tvault-contego, /
tvault-contego-api, /
tvault-horizon-plugin

Overcloud is already deployed

If overcloud is deployed already it is necessary to prepare the artifacts to install on the overcloud.

Firstly the github repository needs to be synced with the undercloud.

git clone https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts
git checkout stable/3.0
cd redhat-director-scripts/

Afterwards are those artifacts created and pushed to the overcloud nodes using the upload-swift-artifacts tool. This tool is provided on the undercloud and the prepare_artifacts.sh is created for it.

## Get help of script
(undercloud) [stack@undercloud redhat-director-scripts]$ ./prepare_artifacts.sh --help

Error: Script takes exactly two arguments, 1 provided
    First argument:    undercloud rc file path
    Second argument:    Trilio VM/cluster IP
For Example:
    ./prepare_artifacts.sh /home/stack/stackrc 192.168.122.201

##Run shell script
[root@redhat-undercloud redhat-director-scripts]# ./prepare_artifacts.sh /home/stack/stackrc 192.168.122.201

Update overcloud roles data file to include Trilio services

Trilio contains of multiple services. Add these services to your roles_data.yaml.

Add the following services to the roles_data.yaml

Trilio Datamover Api Service

This service needs to share the same role as the nova-api service. In case of the pre-defined roles will the nova-api service run on the role Controller. In case of custom defined roles, it is necessary to use the role the nova-api service is using.

Add the following line to the identified role:

'OS::TripleO::Services::TrilioDatamoverApi'

Trilio Horizon Plugin

This service needs to share the same role as the openstack horizon service. In case of the pre-defined roles will the openstack horizon service run on the role Controller. In case of custom defined roles, it is necessary to use the role the openstack horizon service is using.

Add the following line to the identified role:

'OS::TripleO::Services::TrilioHorizonPlugin'

Trilio Datamover Service

This service needs to share the same role as the nova-compute service. In case of the pre-defined roles will the nova-compute service run on the role Compute. In case of custom defined roles, it is necessary to use the role the nova-compute service is using.

Add the following line to the identified role:

'OS::TripleO::Services::TrilioDatamover' 

Provide environment details to trilio_env.yaml

Provide backup target details like NFS share, S3 bucket details and other necessary details in trilio_env.yaml environment file. This environment file will be used in overcloud deployment to configure trilio components.

Sample trilio_env.yaml

resource_registry:
  OS::TripleO::Services::TrilioDatamover: heat/trilio-datamover.yaml
  OS::TripleO::Services::TrilioDatamoverApi: heat/trilio-datamover-api.yaml
  OS::TripleO::Services::TrilioHorizonPlugin: heat/trilio-horizon-plugin.yaml


parameter_defaults:

   ##Edit following parameters only if you are using non-default locations for nova configuration files,
   ##Otherwise just keep them as it is
   NovaConfFile: '/etc/nova/nova.conf'
   NovaDistConfFile: '/usr/share/nova/nova-dist.conf'
   RedhatOpenstackVersion: '10'

   ##Backup target nfs/s3
   BackupTargetType: 'nfs'
    
   ##For backup target 'nfs'
   NfsShares: ''
   NfsOptions: 'nolock,soft,timeo=180,intr'
   
   ##For backup target 's3'
   #S3 type: amazon_s3/ceph_s3/minio_s3
   S3Type: 'amazon_s3'
   S3AccessKey: ''
   S3SecretKey: ''
   S3RegionName: ''
   S3Bucket: ''
   S3EndpointUrl: ''
   S3SslEnabled: false
   S3SignatureVersion: 's3v4'

   #Horizon directory on controller node, this location will be used to install trilio horizon plugin
   HorizonDir: '/usr/share/openstack-dashboard/'
   EnablePackageInstall: True

Deploy overcloud with trilio environment

Use the following heat environment file and roles data file in overcloud deploy command:

1. trilio_env.yaml

2. roles_data.yaml

To include new environment files use '-e' option and for roles data file use '-r' option. An example overcloud deploy command is shown below:

/openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/templates/network-environment.yaml \
-e /home/stack/templates/storage-environment.yaml \
-e ${basedir}/trilio_env.yaml \
-r ${basedir}/roles_data.yaml \
--control-scale 1 --compute-scale 1 --control-flavor control --compute-flavor compute \
--ntp-server 0.north-america.pool.ntp.org --neutron-network-type vxlan --neutron-tunnel-types vxlan \
--validation-errors-fatal --validation-warnings-fatal \
--log-file overcloud_deploy.log

Verify deployment

The following packages should be installed on nodes with the role that contains the nova-api:

# rpm -qa | grep -E 'tvault|workloadmgr'
tvault-contego-api-3.0.44-3.0.noarch

The following packages should be installed nodes with the role that contains the openstack horizon:

# rpm -qa | grep -E 'tvault|workloadmgr'
tvault-horizon-plugin-3.0.44-3.0.noarch
python-workloadmgrclient-3.0.44-3.0.noarch

The following packages should be installed on nodes with the role that contain nova-compute:

# rpm -qa | grep -E 'tvault|trilio'
tvault-contego-3.0.44-3.0.noarch
puppet-triliovault-3.0.44-3.0.noarch

On the same nodes that contain the nova-compute service a new systemd service tvault-contego should have been registered and running. Further should the following mount be visible: /var/triliovault-mounts/<hash>

[root@overcloud-compute-0 opt]# systemctl status tvault-contego
● tvault-contego.service - Tvault contego
   Loaded: loaded (/etc/systemd/system/tvault-contego.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2018-07-14 18:08:20 UTC; 1 day 16h ago
 Main PID: 373191 (python)
   CGroup: /system.slice/tvault-contego.service
           └─373191 /home/tvault/.virtenv/bin/python /home/tvault/.virtenv/bin/tvault-contego --config-file=/usr/share/nova/nova-dist.conf --config-file=/e...

Jul 14 18:08:24 overcloud-compute-0 python[373191]: Option "verbose" from group "DEFAULT" is deprecated for removal.  Its value may be silently ign... future.
Jul 14 18:08:24 overcloud-compute-0 python[373191]: Option "rpc_backend" from group "DEFAULT" is deprecated for removal.  Its value may be silently... future.
Jul 14 18:08:24 overcloud-compute-0 sudo[373247]:     nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/umount -l /var/triliovault-mounts/MTkyL...c2hhcmU=
Jul 14 18:08:24 overcloud-compute-0 python[373191]: umount: /var/triliovault-mounts/MTkyLjE2OC4xMjIuMjQwOi9uZnNmaWxlc2hhcmU=: not mounted
Jul 14 18:08:24 overcloud-compute-0 sudo[373249]:     nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/service tvault-object-store stop
Jul 14 18:08:25 overcloud-compute-0 python[373191]: Redirecting to /bin/systemctl stop tvault-object-store.service
Jul 14 18:08:25 overcloud-compute-0 python[373191]: Failed to stop tvault-object-store.service: Unit tvault-object-store.service not loaded.
Jul 14 18:08:25 overcloud-compute-0 sudo[373260]:     nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/mount -o nolock,soft,timeo=180,intr 192...c2hhcmU=
Jul 14 18:08:26 overcloud-compute-0 python[373191]: /home/tvault/.virtenv/lib/python2.7/site-packages/nova/rpc.py:175: FutureWarning: The access_policy arg...
Jul 14 18:08:26 overcloud-compute-0 python[373191]: serializer=serializer)
Hint: Some lines were ellipsized, use -l to show in full.

[root@overcloud-compute-0 opt]# df -h
Filesystem                     Size  Used Avail Use% Mounted on
/dev/vda2                      200G  5.5G  195G   3% /
devtmpfs                        12G     0   12G   0% /dev
tmpfs                           12G     0   12G   0% /dev/shm
tmpfs                           12G  2.1M   12G   1% /run
tmpfs                           12G     0   12G   0% /sys/fs/cgroup
tmpfs                          2.4G     0  2.4G   0% /run/user/981
tmpfs                          2.4G     0  2.4G   0% /run/user/1000
192.168.122.240:/nfsfileshare   20G  1.7G   19G   9% /var/triliovault-mounts/MTkyLjE2OC4xMjIuMjQwOi9uZnNmaWxlc2hhcmU=

Lastly login into the Horizon Dashboard as admin user. Two new tabs should be visible:

1. Backups

2. Backups-Admin

Troubleshooting for overcloud deployment failures

Trilio components will be deployed using puppet scripts.

In case of the overcloud deployment failing does the following command provide the list of errors:

openstack stack failures list overcloud

Further commands that can help identifying any errors.

heat stack-list --show-nested -f "status=FAILED"
heat resource-list --nested-depth 5 overcloud | grep FAILED

Cinder backend is Ceph - additional steps

Add Ceph details to configuration file

If Cinder backend is Ceph it is necessary to manually add the ceph details to tvault-contego.conf on all compute nodes.

The file can be found here: /var/lib/config-data/puppet-generated/triliodm/etc/tvault-contego/tvault-contego.conf

Add the following information:

[libvirt]
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = <rbd_user>

[ceph] 
keyring_ext = .<rbd_user>.keyring

Give nova user read permission

In case that the user nova does not have permission to read and use the ceph conf and keyring files, run the following commands to provide the necessary access:

setfacl -m u:nova:r /etc/ceph/<ceph_keyring_file_path>
setfacl -m u:nova:r /etc/ceph/<ceph_conf_file_path>

To change the Trilio GUI password do:

• Login into the Trilio Dashboard

• Click on "admin" in the upper right corner to open the submenu

• Choose "Reset Password"

• Set the new Trilio password

Trilio allows you to view or download a file from the snapshot. Any changes to the files or directories when snapshot is mounted are temporary and are discarded when the snapshot is unmounted. Mounting is a faster way to restore a single or multiple files. To mount a snapshot follow these steps.

Trilio ships with a ubuntu based file manager image. This image includes a web based file manager application that helps with browsing and download files when snapshot is mounted. Before you create a File Manager instance, File Manager (a.k.a Recovery Manager) image needs to be uploaded to the Glance. The cloud administrator can upload the image to Glance and mark the image public so that every tenant will have access to the image. File Manager image includes a qemu guest agent. Trilio data mover communicates with the qemu agent to map backup images and discover file systems in the backup images. Since the file manager image includes qemu guest agent, the image should be uploaded to glance with property hw_qemu_guest_agent=yes so Nova will create a qmp channel when creating an instance from this image. Trilio horizon plugin uses a special property tvault_recovery_manager, when set to yes it will filter out instances that don’t have this property. If a tenant has a huge number of instances, setting this property on the image will help Trilio horizon plug-in present only the instances whose image has this property. In this release we only support launching File Manager instance with virtio bus so please set hw_disk_bus to virtio. Use the following command to upload the image to Glance and mark it as public image and set the property.

openstack image create \
--file <File Manager Image Path> \
--container-format bare \
--disk-format qcow2 \
--public \
--property hw_qemu_guest_agent=yes \
--property tvault_recovery_manager=yes \
--property hw_disk_bus=virtio \
tvault-file-manager

The Trilio Appliance can be reinitialized, which will delete all workload related values from the Trilio database.

To reinitialize the Trilio Appliance do:

• Login into the Trilio Dashboard

• Click on "admin" in the upper right corner to open the submenu

• Choose "Reinitialize"

• Verify that you want to reinitialize the Trilio

Like all Openstack services is Trilio using a technical openstack user.

The password for this user can be changed through the Trilio web gui.

To change it use the Service Password option on the main menu.

Trilio release 3.0 introduces new features and capabilities including support for S3 storage targets, capturing tenant’s networking topology, expanded lifecycle cloud management support with Red Hat Director, and more.

Support for S3 storage targets

Since the introduction of Amazon S3, object storage is quickly becoming storage of choice for cloud platforms. Object storage offers very reliable, infinitely scalable storage using cheap hardware. Object storage has become storage of choice for archival, backup and disaster recovery, web hosting, documentation and other use cases. Trilio incorporates Linux’s Filesystem in Userspace (FUSE) with patent pending processing to optimize data handling using the object store. With that, Trilio maintains the same functionality in using S3 as with using the NFS backup target including:

• Incremental forever

• Snapshot retention policy with automatic retirement

• Synthetically full, mountable snapshots

• Efficient restores with minimum requirement of staging area

• Scalable solution that linearly scales with compute nodes without adding any performance or data bandwidth bottlenecks

Backup and restore of tenant’s private network topology

Another milestone achievement in release 3.0, is the ability to protect tenant’s network space. With this, Trilio helps tenants recover the entire network topology including:

• Networks

• Subnets

• Routers

• Static Routes

• Ports

• Floating IP’s

Taking advantage of this additional backup could not be any simpler, as tenants have nothing to do! The entire tenant’s network topology information is automatically included in every snapshot of every workload. This ensures the data is there when needed, eliminates the risk of human error in configuring another protection aspect and keeps it simple. For recovery, tenants may respectively use a point-in-time snapshot from any workload. A new option under Selective Restore is added to restore the network topology. Trilio will recreate the entire tenant network topology from scratch, to exactly the way it was at the time of backup. It will define the private networks with their subnets, recreate the routers, add the correct interfaces to each router and add static routes to the router if applicable. An important consideration in restoring tenants’ networks is that their public network interface may very well have changed. This is always the case in a disaster recovery scenario. For that reason, Trilio will stop short of connecting the new private networks to the public one, allowing tenants to take this last step manually. Note:

• To eliminate conflicts, the tenant’s space must have no networking components defined. The restore will fail if any conflict is found, and the network will be reinstated to what it was prior to the attempted restore.

• As always, Network Topology restore is fully enabled programmatically as well as through the GUI.

New High Availability Cluster Architecture with easier than ever Configurator

Starting with release 3.0, Trilio is deployed using a built-in high availability (HA) cluster architecture, supporting a single node or a three-node cluster. The three-node cluster is the recommended best practice for fault tolerance and load balancing. The deployment is HA ready even with a single node, allowing to expand to three nodes at a later time. For that reason, Trilio requires an additional IP for the cluster even in a single node deployment. The cluster IP (aka virtual IP, VIP) is used for managing the HA cluster and is used to register the Trilio service endpoint in the keystone service catalog. The Trilio installation and deployment process handles all the necessary software (e.g. HAProxy) so users don’t have to manage it on their own. The TVM nodes cannot be installed as VMs under the same OpenStack cloud being protected. They need to be outside of OpenStack on one or more independent KVM hosts. Ideally these KVM hosts would be managed as a virtualized infrastructure using oVirt/RHV, virt-manager or other management tools Configuration GUI The centralized deployment feature is accompanied by a new and improved GUI featuring a Grafana based dashboard, easy to view and modify configuration details, and easy to view Ansible outputs with collapsible level of information

Expanded lifecycle cloud management support

Red Hat director integration

Red Hat OpenStack Platform (RHOSP) director integration allows customers to deploy Trilio using the same lifecycle management tool they use for the cloud itself. The integration supports both cases where the overcloud is deployed for the first time or is already deployed. Release 3.0 supports RHOSP version long-lived version 10. Long-lived version 13 is expected to follow soon.

Mirantis distribution with Debian packaging

Mirantis field personnel and customers who are looking to deploy Trilio 3.0 can now do this through familiar Ubuntu package management tools.

Improved Ansible Automation

The Trilio configuration process has been completely rearchitected using ansible scripts. Ansible, in the last few years, has grown in popularity as a preferred configuration management tool and Trilio uses Ansible play books extensively to configure the Trilio cluster. Ansible modules are inherently idempotent and hence Trilio configuration can run any number of times to change or reconfigure the Trilio cluster.

Enhancement Requests

Release 3.0 includes the following requested enhancements:

Deprecated Functionality

Known Issues

This release contains the following known issues which are tracked for a future update.

Definition

A workload is a backup job that protects one or more Virtual Machines according to a configured policy. There can be as many workloads as needed. But each VM can only be part of one Workload.

List of Workloads

Using Horizon

To view all available workloads of a project inside Horizon do:

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

The overview in Horizon lists all workloads with the following additional information:

• Creation time

• Workload Name

• Workload description

• Total amount of Snapshots inside this workload

  • Total amount of succeeded Snapshots

  • Total amount of failed Snapshots

• Workload Type

• Status of the Workload

Using CLI

Create a Workload

Using Horizon

To create a workload inside Horizon do the following steps:

1. Login to Horizon

2. Navigate to the Backups

3. Navigate to Workloads

4. Click "Create Workload"

5. Provide Workload Name and Workload Description on the first tab "Details"

6. Choose between Serial or Parallel workload on the first tab "Details"

7. Choose the Policy if available to use on the first tab "Details"

8. Choose the VMs to protect on the second Tab "Workload Members"

9. Decide for the schedule of the workload on the Tab "Schedule"

10. Provide the Retention policy on the Tab "Policy"

11. Choose the Full Backup Interval on the Tab "Policy"

12. If required check "Pause VM" on the Tab "Options"

13. Click create

The created Workload will be available after a few seconds and starts to take backups according to the provided schedule and policy.

Using CLI

Workload Overview

A workload contains many information, which can be seen in the workload overview.

Using Horizon

To enter the workload overview inside Horizon do the following steps:

1. Login to Horizon

2. Navigate to the Backups

3. Navigate to Workloads

4. Identify the workload to show the details on

5. Click the workload name to enter the Workload overview

Details Tab

The Workload Details tab provides you with the general most important information about the workload:

• Name

• Description

• Availability Zone

• List of protected VMs including the information of qemu guest agent availability

Snapshots Tab

The Workload Snapshots Tab shows the list of all available Snapshots in the chosen Workload.

From here it is possible to work with the Snapshots, create Snapshots on demand and start Restores.

Policy Tab

The Workload Policy Tab gives an overview of the current configured scheduler and retention policy. The following elements are shown:

• Scheduler Enabled / Disabled

• Start Date / Time

• End Date / Time

• RPO

• Time till next Snapshot run

• Retention Policy and Value

• Full Backup Interval policy and value

Filesearch Tab

The Workload Filesearch Tab provides access to the powerful search engine, which allows to find files and folders on Snapshots without the need of a restore.

Misc. Tab

The Workload Miscellaneous Tab shows the remaining metadata of the Workload. The following information are provided:

• Creation time

• last update time

• Workload ID

• Workload Type

Using CLI

Edit a Workload

Workloads can be modified in all components to match changing needs.

Using Horizon

To edit a workload in Horizon do the following steps:

1. Login to the Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload to be modified

5. Click the small arrow next to "Create Snapshot" to open the sub-menu

6. Click "Edit Workload"

7. Modify the workload as desired - All parameters except workload type can be changed

8. Click "Update"

Using CLI

Delete a Workload

Once a workload is no longer needed it can be safely deleted.

Using Horizon

To delete a workload do the following steps:

1. Login to Horizon

2. Navigate to the Backups

3. Navigate to Workloads

4. Identify the workload to be deleted

5. Click the small arrow next to "Create Snapshot" to open the sub-menu

6. Click "Delete Workload"

7. Confirm by clicking "Delete Workload" yet again

Using CLI

Unlock a Workload

Workloads that are actively taking backups or restores are locked for further tasks. It is possible to unlock a workload by force if necessary.

Using Horizon

1. Login to the Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload to unlock

5. Click the small arrow next to "Create Snapshot" to open the sub-menu

6. Click "Unlock Workload"

7. Confirm by clicking "Unlock Workload" yet again

Using CLI

Reset a Workload

In rare cases it might be necessary to start a backup chain all over again, to ensure the quality of the created backups. To not recreate a Workload in such cases is it possible to reset a Workload.

The Workload reset will:

• Cancel all ongoing tasks

• Delete all existing Openstack Trilio Snapshots from the protected VMs

• recalculate the next Snapshot time

• take a full backup at the next Snapshot

Using Horizon

To reset a Workload do the following steps:

1. Login to the Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload to reset

5. Click the small arrow next to "Create Snapshot" to open the sub-menu

6. Click "Reset Workload"

7. Confirm by clicking "Reset Workload" yet again

Using CLI

Definition

A Snapshot is a single Trilio backup of a workload including all data and metadata. It contains the information of all VM's that are protected by the workload.

List of Snapshots

Using Horizon

1. Login to Horizon

2. Navigate to the Backups

3. Navigate to Workloads

4. Identify the workload to show the details on

5. Click the workload name to enter the Workload overview

6. Navigate to the Snapshots tab

The List of Snapshots for the chosen Workload contains the following additional information:

• Creation Time

• Name of the Snapshot

• Description of the Snapshot

• Total amount of Restores from this Snapshot

  • Total amount of succeeded Restores

  • Total amount of failed Restores

• Snapshot Type

• Snapshot Size

• Snapshot Status

Using CLI

Creating a Snapshot

Snapshots are automatically created by the Trilio scheduler. If necessary or in case of deactivated scheduler is it possible to create a Snapshot on demand.

Using Horizon

There are 2 possibilities to create a snapshot on demand.

Possibility 1: From the Workloads overview

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that shall create a Snapshot

5. Click "Create Snapshot"

6. Provide a name and description for the Snapshot

7. Decide between Full and Incremental Snapshot

8. Click "Create"

Possibility 2: From the Workload Snapshot list

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that shall create a Snapshot

5. Click the workload name to enter the Workload overview

6. Navigate to the Snapshots tab

7. Click "Create Snapshot"

8. Provide a name and description for the Snapshot

9. Decide between Full and Incremental Snapshot

10. Click "Create"

Using CLI

Snapshot overview

Each Snapshot contains a lot of information about the backup. These information can be seen in the Snapshot overview.

Using Horizon

To reach the Snapshot Overview follow these steps:

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that contains the Snapshot to show

5. Click the workload name to enter the Workload overview

6. Navigate to the Snapshots tab

7. Identify the searched Snapshot in the Snapshot list

8. Click the Snapshot Name

Details Tab

The Snapshot Details Tab shows the most important information about the Snapshot.

• Snapshot Name / Description

• Snapshot Type

• Time Taken

• Size

• Which VMs are part of the Snapshot

• for each VM in the Snapshot

  • Instance Info - Name & Status

  • Security Group(s) - Name & Type

  • Flavor - vCPUs, Disk & RAM

  • Networks - IP, Networkname & Mac Address

  • Attached Volumes - Name, Type, size (GB), Mount Point & Restore Size

  • Misc - Original ID of the VM

Restores Tab

The Snapshot Restores Tab shows the list of Restores that have been started from the chosen Snapshot. It is possible to start Restores from here.

Misc. Tab

The Snapshot Miscellaneous Tab provides the remaining metadata information about the Snapshot.

• Creation Time

• Last Update time

• Snapshot ID

• Workload ID of the Workload containing the Snapshot

Using CLI

Delete Snapshots

Once a Snapshot is no longer needed, it can be safely deleted from a Workload.

Using Horizon

There are 2 possibilities to delete a Snapshot.

Possibility 1: Single Snapshot deletion through the submenu

To delete a single Snapshot through the submenu follow these steps:

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that contains the Snapshot to delete

5. Click the workload name to enter the Workload overview

6. Navigate to the Snapshots tab

7. Identify the searched Snapshot in the Snapshot list

8. Click the small arrow in the line of the Snapshot next to "One Click Restore" to open the submenu

9. Click "Delete Snapshot"

10. Confirm by clicking "Delete"

Possibility 2: Multiple Snapshot deletion through checkbox in Snapshot overview

To delete one or more Snapshots through the Snapshot overview do the following:

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that contains the Snapshot to show

5. Click the workload name to enter the Workload overview

6. Navigate to the Snapshots tab

7. Identify the searched Snapshots in the Snapshot list

8. Check the checkbox for each Snapshot that shall be deleted

9. Click "Delete Snapshots"

10. Confirm by clicking "Delete"

Using CLI

Snapshot Cancel

Ongoing Snapshots can be canceled.

Using Horizon

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that contains the Snapshot to cancel

5. Click the workload name to enter the Workload overview

6. Navigate to the Snapshots tab

7. Identify the searched Snapshot in the Snapshot list

8. Click "Cancel" on the same line as the identified Snapshot

9. Confirm by clicking "Cancel"

Using CLI

Trilio, by TrilioData, is a native OpenStack service that provides policy-based comprehensive backup and recovery for OpenStack workloads. The solution captures point-in-time workloads (Application, OS, Compute, Network, Configurations, Data and Metadata of an environment) as full or incremental snapshots. These snapshots can be held in a variety of storage environments including NFS AWS S3 compatible storage. With Trilio and its single click recovery, organizations can improve Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO). With Trilio, IT departments are enabled to fully deploy OpenStack solutions and provide business assurance through enhanced data retention, protection and integrity.

With the use of Trilio’s VAST (Virtual Snapshot Technology), Enterprise IT and Cloud Service Providers can now deploy backup and disaster recovery as a service to prevent data loss or data corruption through point-in-time snapshots and seamless one-click recovery. Trilio takes point-in-time backup of the entire workload consisting of compute resources, network configurations and storage data as one unit. It also takes incremental backups that only captures the changes that were made since last backup. Incremental snapshots save time and storage space as the backup only includes changes since the last backup. The benefits of using VAST for backup and restore could be summarized as below:

1. Efficient capture and storage of snapshots. Since our full backups only include data that is committed to storage volume and the incremental backups only include changed blocks of data since last backup, our backup processes are efficient and storages backup images efficiently on the backup media

2. Faster and reliable recovery. When your applications become complex that snap multiple VMs and storage volumes, our efficient recovery process will bring your application from zero to operational with just click of button

3. Easy migration of workloads between clouds. Trilio captures all the details of your application and hence our migration includes your entire application stack without leaving any thing for guess work.

4. Through policy and automation lower the Total Cost of Ownership. Our tenant driven backup process and automation eliminates the need for dedicated backup administrators, there by improves your total cost of ownership.

The Trilio Appliance Dashboard gives an overview of the running services and their Status inside the Cluster.

It shows for each Trilio Appliance the Status of the following Trilio services:

• wlm-workloads

• wlm-scheduler

• wlm-api

To give administrators an overview of the HA status, does the dashboard also show the service status for:

• Pacemaker

• RabbitMQ

• MySQL Galeera Cluster

Every Trilio appliance provides the possibility to download the following elements:

• Shell-Scripts usable to install Trilio components

• Workloadmgr Python client

• Trilio Logs

Download Shell-Scripts

To download the shell scripts:

• Login into the Trilio web gui

• Go to "Downloads"

• Click on the script to be downloaded

  • tvault-contego-install.shused to install the nova-api extension and the datamover

  • tvault-contego-install.answerused for the automated installation method

  • tvault-horizon-plugin-install.shused to install the Trilio Horizon plugin

Download the workloadmgr python client package

To download the workloadmgr python client:

• Login into the Trilio web gui

• Go to "Downloads"

• Copy the link provided for the workloadmgr client package

• download the workloadmgr client using any webbrowser or cli tool (for example wget)

The workloadmgr client package is provided as deb or rpm.

Download Trilio logs

It is possible to download the Trilio logs directly through the Trilio web gui.

To download logs throught the Trilio web gui:

• Login into the Trilio web gui

• Go to "Logs"

• Choose the log to be downloaded

  • Each log for every Trilio Appliance can be downloaded seperatly

  • or a zip of all logfiles can be created and downloaded

The Trilio appliance can be reconfigured at any time to adjust the Trilio cluster to any changes in the Openstack environment or the general backup solution.

To reconfigure the Trilio Cluster go to the "Configure". The configure page shows the current configuration of the TriloVault cluster.

The configuration page also gives access to the ansible playbooks of the last successful configuration.

To start the reconfiguration of the Trilio Cluster click "Reconfigure" at the end of the table.

Follow the guide afterwards.

On the Trilio Nodes

The Trilio Cluster contains multiple log files.

The main log is workloadmgr-workloads.log, which contains all logs about ongoing and past Trilio backup and restore tasks. It can be found at:

/var/log/workloadmgr/workloadmgr-workloads.log

The next important log is the workloadmgr-api.log, which contains all logs about API calls received by the Trilio Cluster. It can be found at:

/var/log/workloadmgr/workloadmgr-api.log

The log for the remaining service is the workloadmgr-scheduler.log, which contains all logs about the internal job scheduling in the Trilio Cluster.

/var/log/workloadmgr/workloadmgr-scheulder.log

On the Controller node (nova-api extension)

The logs for the nova-api extension can be found here:

/var/log/nova/nova-api.log

On the Compute nodes (datamover)

The logs for the datamover can be found here:

/var/log/nova/tvault-contego.log

It is recommended to think about the following elements prior to the installation of Trilio for Openstack.

Tenant Quotas

Trilio uses Cinder snapshots for calculating full and incremental backups. For full backups, Trilio creates Cinder snapshots for all the volumes in the backup job. It then leaves these Cinder snapshots behind for calculating the incremental backup image during next backup. During an incremental backup operation it creates new Cinder snapshots, calculates the changed blocks between the new snapshots and the old snapshots that were left behind during full/previous backups. It then deletes the old snapshots but leaves the newly created snapshots behind. So, it is important that each tenant who is availing Trilio backup functionality has sufficient Cinder snapshot quotas to accommodate these additional snapshots. The guideline is to add 2 snapshots for every volume that is added to backups to volume snapshot quotas for that tenant. You may also increase the volume quotas for the tenant by the same amount because Trilio briefly creates a volume from snapshot to read data from the snapshot for backup purposes. During a restore process, Trilio creates additional instances and Cinder volumes. To accommodate restore operations, a tenant should have sufficient quota for Nova instances and Cinder volumes. Otherwise restore operations will result in failures.

AWS S3 eventual consistency¶

AWS S3 object consistency model includes:

1. Read-after-write

2. Read-after-update

3. Read-after-delete

Each of them describes how an object will reach its consistent state after an object is created/updated or deleted. None of them provides strong consistency and there is a lag time for an object to reach the consistent state. Though Trilio employed mechanisms to work around the limitations of eventual consistency of AWS S3, when an object reach its consistency state is not deterministic. There is no official statement from AWS on how long it takes for an object to reach consistent state. However read-after-write has a shorter time to reach consistency compared to other IO patterns. Our solution is designed to maximize read-after-write IO pattern. The time in which an object reaches eventual consistency also depends on the AWS region. For example, aws-standard region does not have strong consistency model compared to us-east or us-west. We suggest to use these regions when creating s3 buckets for Trilio. Though read-after-update IO pattern is hard to avoid completely, we employed ample delays in accessing objects to accommodate larger durations for objects to get into consistent state. However in rare occasions, backups may still fail and need to restarted.

Trilio Cluster¶

Trilio can be deployed as a single node or a three node cluster. It is highly recommended that Trilio is deployed as three node cluster for fault tolerance and load balancing. Starting with 3.0 release, Trilio requires additional IP for cluster and is required for both single node and three node deployments. Cluster ip a.k.a virtual ip is used for managing cluster and is used to register Trilio service endpoint in the keystone sevice catalog.

Trilio provides Backup-as-a-Service, which allows Openstack Users to manage and control their backups themselves. This doesn't eradicate the need for a Backup Administrator, who has an overview of the complete Backup Solution.

To provide Backup Administrators with the tools they need does Trilio for Openstack provide a Backup-Admin area in Horizon in addition to the API and CLI.

Access the Backups-Admin area

To access the Backups-Admin area follow these steps:

1. Login to Horizon using admin user.

2. Click on Admin Tab.

3. Navigate to Backups-Admin Tab.

4. Navigate to Trilio page.

The Backups-Admin area provides the following features.

It is possible to reduce the shown information down to a single tenant. That way seeing the exact impact the chosen Tenant has.

Status overview

The status overview is always visible in the Backups-Admin area. It provides the most needed information on a glance, including:

• Storage Usage (nfs only)

• Number of protected VMs compared to number of existing VMs

• Number of currently running Snapshots

• Status of TVault Nodes

• Status of Contego Nodes

Detailed Area

This area is available below the Status overview and consists of multiple tabs.

Workloads tab

This tab provides information about all currently existing Workloads. It is the most important overview tab for every Backup Administrator and therefor the default tab shown when opening the Backup-Admins area.

The following information are shown:

• User-ID that owns the Workload

• Project that contains the Workload

• Workload name

• Workload Type

• Availability Zone

• Amount of protected VMs

• Performance information about the last 30 backups

  • How much data was backed up (green bars)

  • How long did the Backup take (red line)

• Piechart showing amount of Full (Blue) Backups compared to Incremental (Red) Backups

• Number of successful Backups

• Number of failed Backups

• Storage used by that Workload

• Which Backup target is used

• When is the next Snapshot run

• What is the general intervall of the Workload

• Scheduler Status including a Switch to deactivate/activate the Workload

Usage tab

Administrators often need to figure out, where a lot of resources are used up, or they need to quickly provide usage information to a billing system. This tab helps in these tasks by providing the following information:

• Storage used by a Tenant

• VMs protected by a Tenant

It is possible to drill down to see the same information per workload and finally per protected VM.

Nodes tab

This tab displays information about Trilio cluster nodes. The following information are shown:

• Node name

• Node ID

• Trilio Version of the node

• IP Address

• Active Controller Node (True/False)

• Status of the Node

Data Movers tab (Trilio Data Mover Service)

This tab displays information about Trilio contego service. The following information are shown:

• Service-Name

• Compute Node the service is running on

• Zone

• Service Status from Openstack perspective (enabled/disabled)

• Version of the Service

• General Status

• last time the Status was updated

Storage tab

This tab displays information about the backup target storage. It contains the following information:

• Storage Name

• Capacity of the storage

• Total utilization of the storage

• Status of the storage

• Statistic information

  • Percentage all storages are used

  • Percentage how much storage is used for full backups

  • Amount of Full backups versus Incremental backups

Audit tab

Audit logs provide the sequence of workload related activities done by users, like workload creation, snapshot creation, etc. The following information are shown:

• Time of the entry

• What task has been done

• Project the task has performed in

• User that performed the task

The Audit log can be searched for strings to find for example only entries down by a specific user.

Additionally, can the shown timeframe be changed as necessary.

License tab

The license tab provides an overview over the current license and allows to upload new licenses, or validate the current license.

The following information about an active license are shown:

• Organization (License name)

• License ID

• Purchase date - when was the license created

• License Expiry Date

• Maintenance Expiry Date

• License value

• License Edition

• License Version

• License Type

• Description of the License

• Evaluation (True/False)

Policy tab

The policy tab gives Administrators the possibility to work with workload policies.

Settings tab

This tab manages all global settings for the whole cloud. Trilio has two types of settings:

1. Email settings

2. Job scheduler settings.

Email Settings¶

These settings will be used by Trilio to send email reports of snapshots and restores to users.

The following information are required to configure the email settings:

• SMTP Server

• SMTP username

• SMTP password

• SMTP port

• SMTP timeout

• Sender email address

Disable/Enable Job Scheduler

The Global Job Scheduler can be used to deactivate all scheduled workloads without modifying each one of them.

To activate/deactivate the Global Job Scheduler through the Backups-Admin area:

1. Login to Horizon using admin user.

2. Click on Admin Tab.

3. Navigate to Backups-Admin Tab.

4. Navigate to Trilio page.

5. Navigate to the Settings tab

6. Click "Disable/Enable Job Scheduler"

7. Check or Uncheck the box for "Job Scheduler Enabled"

8. Confirm by clicking on "Change"

The Global Job Scheduler can be controlled through CLI as well.

To get the status of the Global Job Scheduler use:

workloadmgr get-global-job-scheduler

To deactivate the Global Job Scheduler use:

workloadmgr disable-global-job-scheduler

To activate the Global Job Scheduler use:

workloadmgr enable-global-job-scheduler

Trilio Workloads are designed to allow a Disaster Recovery without the need to backup the Trilio database.

As long as the Trilio Workloads are existing on the Backup Target Storage and a Trilio installation has access to them, it is possible to restore the Workloads.

Disaster Recovery Process

1. Install and Configure Trilio for the target cloud

2. Verify required mount-paths and create if necessary

3. Reassign Workloads

4. Notify users to of Workloads being available

Mount-paths

Trilio incremental Snapshots involve a backing file to the prior backup taken, which makes every Trilio incremental backup a synthetic full backup.

Trilio is using qcow2 backing files for this feature:

qemu-img info 85b645c5-c1ea-4628-b5d8-1faea0e9d549
image: 85b645c5-c1ea-4628-b5d8-1faea0e9d549
file format: qcow2
virtual size: 1.0G (1073741824 bytes)
disk size: 21M
cluster_size: 65536
backing file: /var/triliovault-mounts/MTAuMTAuMi4yMDovdXBzdHJlYW0=/workload_3c2fbee5-ad90-4448-b009-5047bcffc2ea/snapshot_f4874ed7-fe85-4d7d-b22b-082a2e068010/vm_id_9894f013-77dd-4514-8e65-818f4ae91d1f/vm_res_id_9ae3a6e7-dffe-4424-badc-bc4de1a18b40_vda/a6289269-3e72-4085-adca-e228ba656984
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

As can be seen in the example is the backing file an absolute path, which makes it necessary, that this path exists so the backing files can be accessed.

Trilio is using the base64 hashing algorithm for the NFS mount-paths, to allow the configuration of multiple NFS Volumes at the same time. The hash value is calculated using the provided NFS path.

# echo -n 10.10.2.20:/upstream | base64
MTAuMTAuMi4yMDovdXBzdHJlYW0=

When the path of the backing file is not available on the Trilio appliance and Compute nodes, will the restores of incremental backups fail.

The tested and recommended method to make the backing files available is creating the required directory path and using mount --bind to make the path available for the backups.

#mount --bind <mount-path1> <mount-path2>

Running the mount --bind command will make the necessary path available until the next reboot. If it is required to have access to the path beyond a reboot is it necessary to edit the fstab.

#vi /etc/fstab
<mount-path1> <mount-path2>	none bind	0 0

Troubleshooting inside a complex environment like Openstack can be very time-consuming. The following tipps will help to speed up the troubleshooting process to identify root causes.

What is happening where

Openstack and Trilio are divided into multiple services. Each service has a very specific purpose that is called during a backup or recovery procedure. Knowing which service is doing what helps to understand where the error is happening, allowing more focused troubleshooting.

Trilio cluster

The Trilio Cluster is the Controller of Trilio. It receives all Workload related requests from the users.

Every task of a backup or restore process is triggered and managed from here. This includes the creation of the directory structure and initial metadata files on the Backup Target.

During a backup process

During a backup process is the Trilio cluster also responsible to gather the metadata about the backed up VMs and networks from the Openstack environment. It is sending API calls towards the Openstack endpoints on the configured endpoint type to fetch this information. Once the metadata has been received does the Trilio Cluster write it as json files on the Backup Target.

The Trilio cluster is also sending the Cinder Snapshot command.

During a restore process

During restore process is the Trilio cluster reading the VM metadata from its Database and is using the metadata to create the Shell for the restore. It is sending API calls to the Openstack environment to create the necessary resources.

dmapi

The dmapi service is the connector between the Trilio cluster and the datamover running on the compute nodes.

The purpose of the dmapi service is to identify which compute node is responsible for the current backup or restore task. To do so is the dmapi service connecting to the nova api database requesting the compute hose of a provided VM.

Once the compute host has been identified is the dmapi forwarding the command from the Trilio Cluster to the datamover running on the identified compute host.

datamover

The datamover is the Trilio service running on the compute nodes.

Each datamover is responsible for the VMs running on top of its compute node. A datamover can not work with VMs running on a different compute node.

The datamover is controlling the freeze and thaw of VMs as well as the actual movement of the data.

Everything on the Backup Target is happening as user nova

Trilio is reading and writing on the Backup Target as nova:nova.

The POSIX user-id and group-id of nova:nova need to be aligned between the Trilio Cluster and all compute nodes. Otherwise backup or restores may fail with permission or file not found issues.

Alternativ ways to achieve the goal are possible, as long as all required nodes can fully write and read as nova:nova on the Backup Target.

It is recommended to verify the required permissions on the Backup Target in case of any errors during the data transfer phase or in case of any file permission errors.

Trilio Trustee Role

Trilio is using RBAC to allow the usage of Trilio features to users.

This trustee role is absolutely required and can not be overwritten using the admin role.

It is recommended to verify the assignment of the Trilio Trustee Role in case of any permission errors from Trilio during creation of Workloads, backups or restores.

Openstack Quotas

Trilio is creating Cinder Snapshots and temporary Cinder Volumes. The Openstack Quotas need to allow that.

Every disk that is getting backed up requires one temporary Cinder Volumes.

Every Cinder Volume that is getting backup requires two Cinder Snapshots. The second Cinder Snapshot is temporary to calculate the incremental.

Definition

Trilio allows you to view or download a file from the snapshot. Any changes to the files or directories when snapshot is mounted are temporary and are discarded when the snapshot is unmounted. Mounting is a faster way to restore a single or multiple files. To mount a snapshot follow these steps.

Trilio ships with a ubuntu based file manager image. This image includes a web based file manager application that helps with browsing and download files when snapshot is mounted. Before you create a File Manager instance, File Manager (a.k.a Recovery Manager) image needs to be uploaded to the Glance.

Spin up a File Recovery Manager

To be able to mount a Snapshot a File Recovery Manager needs to be available in the tenant.

The File Recovery Manager is a normal Openstack instance, which requires at least 40GB boot disk.

It is recommended to provide the File Recovery Manager instance with a Floating IP and appropriate Security Groups. Appropriate would be to restrict access to ports 80, 443, and 22.

Mounting a Snapshot

Mounting a Snapshot to a File Recovery Manager provides read access to all data that is located on the in the mounted Snapshot.

Using Horizon

There are 2 possibilities to mount a Snapshot in Horizon.

Through the Snapshot list

To mount a Snapshot through the Snapshot list follow these steps:

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that contains the Snapshot to mount

5. Click the workload name to enter the Workload overview

6. Navigate to the Snapshots tab

7. Identify the searched Snapshot in the Snapshot list

8. Click the small arrow in the line of the Snapshot next to "One Click Restore" to open the submenu

9. Click "Mount Snapshot"

10. Choose the File Recovery Manager instance to mount to

11. Confirm by clicking "Mount"

Through the File Search results

To mount a Snapshot through the File Search results follow these steps:

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that contains the Snapshot to mount

5. Click the workload name to enter the Workload overview

6. Navigate to the File Search tab

7. Do a File Search

8. Identify the Snapshot to be mounted

9. Click "Mount Snapshot" for the chosen Snapshot

10. Choose the File Recovery Manager instance to mount to

11. Confirm by clicking "Mount"

Using CLI

workloadmgr snapshot-mount <snapshot_id> <mount_vm_id>

Accessing the File Recovery Manager

The File Recovery Manager is a normal Openstack instance running a webserver. It can be accessed through

• the Openstack VNC console (requires password access)

• SSH

• HTTP on port 80

The Webserver provides a File Commander application, which will show the mounted Snapshot and provides the possibility to navigate through it and download Files as necessary.

When using SSH or the Openstack VNC console can the mounted Snapshot be found under:

/home/ubuntu/tvault-mounts/mounts/

Identifying mounted Snapshots

Sometimes a Snapshot is mounted for a longer time and it needs to be identified, which Snapshots are mounted.

Using Horizon

There are 2 possibilities to identify mounted Snapshots inside Horizon.

From the File Recovery Manager instance Metadata

1. Login to Horizon

2. Navigate to Compute

3. Navigate to Instances

4. Identify the File Recovery Manager Instance

5. Click on the Name of the File Recovery Manager Instance to bring up its details

6. On the Overview tab look for Metadata

7. Identify the value for mounted_snapshot_url

The mounted_snapshot_url contains the Snapshot ID of the Snapshot that has been mounted last.

From the Snapshot list

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that contains the Snapshot to mount

5. Click the workload name to enter the Workload overview

6. Navigate to the Snapshots tab

7. Search for the Snapshot that has the option "Unmount Snapshot"

Using CLI

workloadmgr snapshot-mounted-list [--workloadid <workloadid>]

Unmounting a Snapshot

Once a mounted Snapshot is no longer needed it is possible and recommended to unmount the snapshot.

Using Horizon

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that contains the Snapshot to mount

5. Click the workload name to enter the Workload overview

6. Navigate to the Snapshots tab

7. Search for the Snapshot that has the option "Unmount Snapshot"

8. Click "Unmount Snapshot"

Using the CLI

workloadmgr snapshot-dismount <snapshot_id>

Definition