Post Installation Health-Check

After the installation and configuration of TrilioVault for RHV did succeed the following steps can be done to verify that the TrilioVault installation is healthy.

Verify the TrilioVault Appliance services are up

TrilioVault is using 3 main services on the TrilioVault Appliance:

wlm-api
wlm-scheduler
wlm-workloads

Those can be verified to be up and running using the systemctl status command.

systemctl status wlm-api
######
● wlm-api.service - Cluster Controlled wlm-api
   Loaded: loaded (/etc/systemd/system/wlm-api.service; disabled; vendor preset: disabled)
  Drop-In: /run/systemd/system/wlm-api.service.d
           └─50-pacemaker.conf
   Active: active (running) since Wed 2020-04-22 09:17:05 UTC; 1 day 2h ago
 Main PID: 21265 (python)
    Tasks: 1
   CGroup: /system.slice/wlm-api.service
           └─21265 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-api --config-file=/etc/workloadmgr/workloadmgr.conf

systemctl status wlm-scheduler
######
● wlm-scheduler.service - Cluster Controlled wlm-scheduler
   Loaded: loaded (/etc/systemd/system/wlm-scheduler.service; disabled; vendor preset: disabled)
  Drop-In: /run/systemd/system/wlm-scheduler.service.d
           └─50-pacemaker.conf
   Active: active (running) since Wed 2020-04-22 09:17:17 UTC; 1 day 2h ago
 Main PID: 21512 (python)
    Tasks: 1
   CGroup: /system.slice/wlm-scheduler.service
           └─21512 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-scheduler --config-file=/etc/workloadmgr/workloadmgr.conf

systemctl status wlm-workloads
######
● wlm-workloads.service - workloadmanager workloads service
   Loaded: loaded (/etc/systemd/system/wlm-workloads.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2020-04-22 09:15:43 UTC; 1 day 2h ago
 Main PID: 20079 (python)
    Tasks: 33
   CGroup: /system.slice/wlm-workloads.service
           ├─20079 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf
           ├─20180 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf
           [...]
           ├─20181 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf
           ├─20233 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf
           ├─20236 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf
           └─20237 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf

Check the TrilioVault pacemaker and nginx cluster

The second component to check the TrilioVault Appliance's health is the nginx and pacemaker cluster.

pcs status
######
Cluster name: triliovault

WARNINGS:
Corosync and pacemaker node names do not match (IPs used in setup?)
Stack: corosync
Current DC: om_tvm (version 1.1.19-8.el7_6.1-c3c624ea3d) -
partition with quorum
Last updated: Wed Dec 5 12:25:02 2018
Last change: Wed Dec 5 09:20:08 2018 by root via cibadmin on om_tvm
1 node configured
4 resources configured

Online: [ om_tvm ]
Full list of resources:
virtual_ip (ocf::'heartbeat:IPaddr2): Started om_tvm
wlm-api (systemd:wlm-api): Started om_tvm
wlm-scheduler (systemd:wlm-scheduler): Started om_tvm
Clone Set: lb_nginx-clone [lb_nginx]
Started: [ om_tvm ]
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled

Verify API connectivity from the RHV-Manager

The RHV-Manager is doing all API calls towards the TrilioVault Appliance. Therefore it is helpful to do a quick API connectivity check using curl.

The following curl command lists the available workload-types and verfifies that the connection is available and working:

curl -k -XGET https://30.30.1.11:8780/v1/admin/workload_types/detail -H "Content-Type: application/json" -H "X-OvirtAuth-User: admin@internal" -H "X-OvirtAuth-Password: password"
######
{"workload_types": [{"status": "available", "user_id": "admin@internal", "name": "Parallel", "links": [{"href": "https://myapp/v1/admin/workloadtypes/2ddd528d-c9b4-4d7e-8722-cc395140255a", "rel": "self"}, {"href": "https://myapp/admin/workloadtypes/2ddd528d-c9b4-4d7e-8722-cc395140255a", "rel": "bookmark"}], "created_at": "2020-04-02T15:38:51.000000", "updated_at": "2020-04-02T15:38:51.000000", "metadata": [], "is_public": true, "project_id": "admin", "id": "2ddd528d-c9b4-4d7e-8722-cc395140255a", "description": "Parallel workload that snapshots VM in the specified order"}, {"status": "available", "user_id": "admin@internal", "name": "Serial", "links": [{"href": "https://myapp/v1/admin/workloadtypes/f82ce76f-17fe-438b-aa37-7a023058e50d", "rel": "self"}, {"href": "https://myapp/admin/workloadtypes/f82ce76f-17fe-438b-aa37-7a023058e50d", "rel": "bookmark"}], "created_at": "2020-04-02T15:38:47.000000", "updated_at": "2020-04-02T15:38:47.000000", "metadata": [], "is_public": true, "project_id": "admin", "id": "f82ce76f-17fe-438b-aa37-7a023058e50d", "description": "Serial workload that snapshots VM in the specified order"}]}

Verify the ovirt-imageio services are up and running

TrilioVault is extending the already exiting ovirt-imageio services. The installation of these extensions does check if the ovirt-services come up. Still it is a good call to verify again afterwards:

On the RHV-Manager check the ovirt-imageio-proxy service:

systemctl status ovirt-imageio-proxy
######
● ovirt-imageio-proxy.service - oVirt ImageIO Proxy
   Loaded: loaded (/usr/lib/systemd/system/ovirt-imageio-proxy.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2020-04-08 05:05:25 UTC; 2 weeks 1 days ago
 Main PID: 1834 (python)
   CGroup: /system.slice/ovirt-imageio-proxy.service
           └─1834 bin/python proxy/ovirt-imageio-proxy

On the RHV-Host check the ovirt-imageio-daemon service:

systemctl status ovirt-imageio-daemon
######
● ovirt-imageio-daemon.service - oVirt ImageIO Daemon
   Loaded: loaded (/usr/lib/systemd/system/ovirt-imageio-daemon.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2020-04-08 04:40:50 UTC; 2 weeks 1 days ago
 Main PID: 1442 (python)
    Tasks: 4
   CGroup: /system.slice/ovirt-imageio-daemon.service
           └─1442 /opt/ovirt-imageio/bin/python daemon/ovirt-imageio-daemon

Verify the NFS Volume is correctly mounted

TrilioVault mounts the NFS Backup Target to the TrilioVault Appliance and RHV-Hosts.

To verify those are correctly mounted it is recommended to do the following checks.

First df -h looking for /var/triliovault-mounts/<hash-value>

df -h
######
Filesystem                                      Size  Used Avail Use% Mounted on
devtmpfs                                         63G     0   63G   0% /dev
tmpfs                                            63G   16K   63G   1% /dev/shm
tmpfs                                            63G   35M   63G   1% /run
tmpfs                                            63G     0   63G   0% /sys/fs/cgroup
/dev/mapper/rhvh-rhvh--4.3.8.1--0.20200126.0+1  7.1T  3.7G  6.8T   1% /
/dev/sda2                                       976M  198M  712M  22% /boot
/dev/mapper/rhvh-var                             15G  1.9G   12G  14% /var
/dev/mapper/rhvh-home                           976M  2.6M  907M   1% /home
/dev/mapper/rhvh-tmp                            976M  2.6M  907M   1% /tmp
/dev/mapper/rhvh-var_log                        7.8G  230M  7.2G   4% /var/log
/dev/mapper/rhvh-var_log_audit                  2.0G   17M  1.8G   1% /var/log/audit
/dev/mapper/rhvh-var_crash                      9.8G   37M  9.2G   1% /var/crash
30.30.1.4:/rhv_backup                           2.0T  5.3G  1.9T   1% /var/triliovault-mounts/MzAuMzAuMS40Oi9yaHZfYmFja3Vw
30.30.1.4:/rhv_data                             2.0T   37G  2.0T   2% /rhev/data-center/mnt/30.30.1.4:_rhv__data
tmpfs                                            13G     0   13G   0% /run/user/0
30.30.1.4:/rhv_iso                              2.0T   37G  2.0T   2% /rhev/data-center/mnt/30.30.1.4:_rhv__iso

Secondly do a read / write / delete test as the user vdsm:kvm (uid = 36 / gid = 36) from the TrilioVault Appliance and the RHV-Host.

su vdsm
######
[vdsm@rhv-tvm MzAuMzAuMS40Oi9yaHZfYmFja3Vw]$ touch foo
[vdsm@rhv-tvm MzAuMzAuMS40Oi9yaHZfYmFja3Vw]$ ll
total 24
drwxr-xr-x  3 vdsm kvm 4096 Apr  2 17:27 contego_tasks
-rw-r--r--  1 vdsm kvm    0 Apr 23 12:25 foo
drwxr-xr-x  2 vdsm kvm 4096 Apr  2 15:38 test-cloud-id
drwxr-xr-x 10 vdsm kvm 4096 Apr 22 11:00 workload_1540698c-8e22-4dd1-a898-8f49cd1a898c
drwxr-xr-x  9 vdsm kvm 4096 Apr  8 15:21 workload_51517816-6d5a-4fce-9ac7-46ee1e09052c
drwxr-xr-x  6 vdsm kvm 4096 Apr 22 11:30 workload_77fb42d2-8d34-4b8d-bfd5-4263397b636c
drwxr-xr-x  5 vdsm kvm 4096 Apr 23 06:15 workload_85bf16ed-d4fd-49a6-a753-98c5ca6e906b
[vdsm@rhv-tvm MzAuMzAuMS40Oi9yaHZfYmFja3Vw]$ rm foo
[vdsm@rhv-tvm MzAuMzAuMS40Oi9yaHZfYmFja3Vw]$ ll
total 24
drwxr-xr-x  3 vdsm kvm 4096 Apr  2 17:27 contego_tasks
drwxr-xr-x  2 vdsm kvm 4096 Apr  2 15:38 test-cloud-id
drwxr-xr-x 10 vdsm kvm 4096 Apr 22 11:00 workload_1540698c-8e22-4dd1-a898-8f49cd1a898c
drwxr-xr-x  9 vdsm kvm 4096 Apr  8 15:21 workload_51517816-6d5a-4fce-9ac7-46ee1e09052c
drwxr-xr-x  6 vdsm kvm 4096 Apr 22 11:30 workload_77fb42d2-8d34-4b8d-bfd5-4263397b636c
drwxr-xr-x  5 vdsm kvm 4096 Apr 23 06:15 workload_85bf16ed-d4fd-49a6-a753-98c5ca6e906b
[vdsm@rhv-tvm MzAuMzAuMS40Oi9yaHZfYmFja3Vw]$

PreviousInstallation of ovirt-imageio extensions NextUninstall Trilio

Last updated 5 years ago

hashtagVerify the TrilioVault Appliance services are up

hashtagCheck the TrilioVault pacemaker and nginx cluster

hashtagVerify API connectivity from the RHV-Manager

hashtagVerify the ovirt-imageio services are up and running

hashtagVerify the NFS Volume is correctly mounted

Verify the TrilioVault Appliance services are up

Check the TrilioVault pacemaker and nginx cluster

Verify API connectivity from the RHV-Manager

Verify the ovirt-imageio services are up and running

Verify the NFS Volume is correctly mounted