After the installation and configuration of TrilioVault for RHV did succeed the following steps can be done to verify that the TrilioVault installation is healthy.
TrilioVault is using 3 main services on the TrilioVault Appliance:
wlm-api
wlm-scheduler
wlm-workloads
Those can be verified to be up and running using the systemctl status
command.
systemctl status wlm-api######● wlm-api.service - Cluster Controlled wlm-apiLoaded: loaded (/etc/systemd/system/wlm-api.service; disabled; vendor preset: disabled)Drop-In: /run/systemd/system/wlm-api.service.d└─50-pacemaker.confActive: active (running) since Wed 2020-04-22 09:17:05 UTC; 1 day 2h agoMain PID: 21265 (python)Tasks: 1CGroup: /system.slice/wlm-api.service└─21265 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-api --config-file=/etc/workloadmgr/workloadmgr.conf
systemctl status wlm-scheduler######● wlm-scheduler.service - Cluster Controlled wlm-schedulerLoaded: loaded (/etc/systemd/system/wlm-scheduler.service; disabled; vendor preset: disabled)Drop-In: /run/systemd/system/wlm-scheduler.service.d└─50-pacemaker.confActive: active (running) since Wed 2020-04-22 09:17:17 UTC; 1 day 2h agoMain PID: 21512 (python)Tasks: 1CGroup: /system.slice/wlm-scheduler.service└─21512 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-scheduler --config-file=/etc/workloadmgr/workloadmgr.conf
systemctl status wlm-workloads######● wlm-workloads.service - workloadmanager workloads serviceLoaded: loaded (/etc/systemd/system/wlm-workloads.service; enabled; vendor preset: disabled)Active: active (running) since Wed 2020-04-22 09:15:43 UTC; 1 day 2h agoMain PID: 20079 (python)Tasks: 33CGroup: /system.slice/wlm-workloads.service├─20079 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf├─20180 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf[...]├─20181 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf├─20233 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf├─20236 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf└─20237 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf
The second component to check the TrilioVault Appliance's health is the nginx and pacemaker cluster.
pcs status######Cluster name: triliovaultWARNINGS:Corosync and pacemaker node names do not match (IPs used in setup?)Stack: corosyncCurrent DC: om_tvm (version 1.1.19-8.el7_6.1-c3c624ea3d) -partition with quorumLast updated: Wed Dec 5 12:25:02 2018Last change: Wed Dec 5 09:20:08 2018 by root via cibadmin on om_tvm1 node configured4 resources configuredOnline: [ om_tvm ]Full list of resources:virtual_ip (ocf::'heartbeat:IPaddr2): Started om_tvmwlm-api (systemd:wlm-api): Started om_tvmwlm-scheduler (systemd:wlm-scheduler): Started om_tvmClone Set: lb_nginx-clone [lb_nginx]Started: [ om_tvm ]Daemon Status:corosync: active/enabledpacemaker: active/enabledpcsd: active/enabled
The RHV-Manager is doing all API calls towards the TrilioVault Appliance. Therefore it is helpful to do a quick API connectivity check using curl.
The following curl command lists the available workload-types and verfifies that the connection is available and working:
curl -k -XGET https://30.30.1.11:8780/v1/admin/workload_types/detail -H "Content-Type: application/json" -H "X-OvirtAuth-User: admin@internal" -H "X-OvirtAuth-Password: password"######{"workload_types": [{"status": "available", "user_id": "admin@internal", "name": "Parallel", "links": [{"href": "https://myapp/v1/admin/workloadtypes/2ddd528d-c9b4-4d7e-8722-cc395140255a", "rel": "self"}, {"href": "https://myapp/admin/workloadtypes/2ddd528d-c9b4-4d7e-8722-cc395140255a", "rel": "bookmark"}], "created_at": "2020-04-02T15:38:51.000000", "updated_at": "2020-04-02T15:38:51.000000", "metadata": [], "is_public": true, "project_id": "admin", "id": "2ddd528d-c9b4-4d7e-8722-cc395140255a", "description": "Parallel workload that snapshots VM in the specified order"}, {"status": "available", "user_id": "admin@internal", "name": "Serial", "links": [{"href": "https://myapp/v1/admin/workloadtypes/f82ce76f-17fe-438b-aa37-7a023058e50d", "rel": "self"}, {"href": "https://myapp/admin/workloadtypes/f82ce76f-17fe-438b-aa37-7a023058e50d", "rel": "bookmark"}], "created_at": "2020-04-02T15:38:47.000000", "updated_at": "2020-04-02T15:38:47.000000", "metadata": [], "is_public": true, "project_id": "admin", "id": "f82ce76f-17fe-438b-aa37-7a023058e50d", "description": "Serial workload that snapshots VM in the specified order"}]}
TrilioVault is extending the already exiting ovirt-imageio services. The installation of these extensions does check if the ovirt-services come up. Still it is a good call to verify again afterwards:
On the RHV-Manager check the ovirt-imageio-proxy service:
systemctl status ovirt-imageio-proxy######● ovirt-imageio-proxy.service - oVirt ImageIO ProxyLoaded: loaded (/usr/lib/systemd/system/ovirt-imageio-proxy.service; enabled; vendor preset: disabled)Active: active (running) since Wed 2020-04-08 05:05:25 UTC; 2 weeks 1 days agoMain PID: 1834 (python)CGroup: /system.slice/ovirt-imageio-proxy.service└─1834 bin/python proxy/ovirt-imageio-proxy
On the RHV-Host check the ovirt-imageio-daemon service:
systemctl status ovirt-imageio-daemon######● ovirt-imageio-daemon.service - oVirt ImageIO DaemonLoaded: loaded (/usr/lib/systemd/system/ovirt-imageio-daemon.service; enabled; vendor preset: disabled)Active: active (running) since Wed 2020-04-08 04:40:50 UTC; 2 weeks 1 days agoMain PID: 1442 (python)Tasks: 4CGroup: /system.slice/ovirt-imageio-daemon.service└─1442 /opt/ovirt-imageio/bin/python daemon/ovirt-imageio-daemon
TrilioVault mounts the NFS Backup Target to the TrilioVault Appliance and RHV-Hosts.
To verify those are correctly mounted it is recommended to do the following checks.
First df -h looking for /var/triliovault-mounts/<hash-value>
df -h######Filesystem Size Used Avail Use% Mounted ondevtmpfs 63G 0 63G 0% /devtmpfs 63G 16K 63G 1% /dev/shmtmpfs 63G 35M 63G 1% /runtmpfs 63G 0 63G 0% /sys/fs/cgroup/dev/mapper/rhvh-rhvh--4.3.8.1--0.20200126.0+1 7.1T 3.7G 6.8T 1% //dev/sda2 976M 198M 712M 22% /boot/dev/mapper/rhvh-var 15G 1.9G 12G 14% /var/dev/mapper/rhvh-home 976M 2.6M 907M 1% /home/dev/mapper/rhvh-tmp 976M 2.6M 907M 1% /tmp/dev/mapper/rhvh-var_log 7.8G 230M 7.2G 4% /var/log/dev/mapper/rhvh-var_log_audit 2.0G 17M 1.8G 1% /var/log/audit/dev/mapper/rhvh-var_crash 9.8G 37M 9.2G 1% /var/crash30.30.1.4:/rhv_backup 2.0T 5.3G 1.9T 1% /var/triliovault-mounts/MzAuMzAuMS40Oi9yaHZfYmFja3Vw30.30.1.4:/rhv_data 2.0T 37G 2.0T 2% /rhev/data-center/mnt/30.30.1.4:_rhv__datatmpfs 13G 0 13G 0% /run/user/030.30.1.4:/rhv_iso 2.0T 37G 2.0T 2% /rhev/data-center/mnt/30.30.1.4:_rhv__iso
Secondly do a read / write / delete test as the user vdsm:kvm (uid = 36 / gid = 36) from the TrilioVault Appliance and the RHV-Host.
su vdsm######[vdsm@rhv-tvm MzAuMzAuMS40Oi9yaHZfYmFja3Vw]$ touch foo[vdsm@rhv-tvm MzAuMzAuMS40Oi9yaHZfYmFja3Vw]$ lltotal 24drwxr-xr-x 3 vdsm kvm 4096 Apr 2 17:27 contego_tasks-rw-r--r-- 1 vdsm kvm 0 Apr 23 12:25 foodrwxr-xr-x 2 vdsm kvm 4096 Apr 2 15:38 test-cloud-iddrwxr-xr-x 10 vdsm kvm 4096 Apr 22 11:00 workload_1540698c-8e22-4dd1-a898-8f49cd1a898cdrwxr-xr-x 9 vdsm kvm 4096 Apr 8 15:21 workload_51517816-6d5a-4fce-9ac7-46ee1e09052cdrwxr-xr-x 6 vdsm kvm 4096 Apr 22 11:30 workload_77fb42d2-8d34-4b8d-bfd5-4263397b636cdrwxr-xr-x 5 vdsm kvm 4096 Apr 23 06:15 workload_85bf16ed-d4fd-49a6-a753-98c5ca6e906b[vdsm@rhv-tvm MzAuMzAuMS40Oi9yaHZfYmFja3Vw]$ rm foo[vdsm@rhv-tvm MzAuMzAuMS40Oi9yaHZfYmFja3Vw]$ lltotal 24drwxr-xr-x 3 vdsm kvm 4096 Apr 2 17:27 contego_tasksdrwxr-xr-x 2 vdsm kvm 4096 Apr 2 15:38 test-cloud-iddrwxr-xr-x 10 vdsm kvm 4096 Apr 22 11:00 workload_1540698c-8e22-4dd1-a898-8f49cd1a898cdrwxr-xr-x 9 vdsm kvm 4096 Apr 8 15:21 workload_51517816-6d5a-4fce-9ac7-46ee1e09052cdrwxr-xr-x 6 vdsm kvm 4096 Apr 22 11:30 workload_77fb42d2-8d34-4b8d-bfd5-4263397b636cdrwxr-xr-x 5 vdsm kvm 4096 Apr 23 06:15 workload_85bf16ed-d4fd-49a6-a753-98c5ca6e906b[vdsm@rhv-tvm MzAuMzAuMS40Oi9yaHZfYmFja3Vw]$