# Post Installation Health-Check After the installation and configuration of TrilioVault for RHV did succeed the following steps can be done to verify that the TrilioVault installation is healthy. ## Verify the TrilioVault Appliance services are up TrilioVault is using 3 main services on the TrilioVault Appliance: * `wlm-api` * `wlm-scheduler` * `wlm-workloads` Those can be verified to be up and running using the `systemctl status` command. ```bash systemctl status wlm-api ###### ● wlm-api.service - Cluster Controlled wlm-api Loaded: loaded (/etc/systemd/system/wlm-api.service; disabled; vendor preset: disabled) Drop-In: /run/systemd/system/wlm-api.service.d └─50-pacemaker.conf Active: active (running) since Wed 2020-04-22 09:17:05 UTC; 1 day 2h ago Main PID: 21265 (python) Tasks: 1 CGroup: /system.slice/wlm-api.service └─21265 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-api --config-file=/etc/workloadmgr/workloadmgr.conf ``` ```bash systemctl status wlm-scheduler ###### ● wlm-scheduler.service - Cluster Controlled wlm-scheduler Loaded: loaded (/etc/systemd/system/wlm-scheduler.service; disabled; vendor preset: disabled) Drop-In: /run/systemd/system/wlm-scheduler.service.d └─50-pacemaker.conf Active: active (running) since Wed 2020-04-22 09:17:17 UTC; 1 day 2h ago Main PID: 21512 (python) Tasks: 1 CGroup: /system.slice/wlm-scheduler.service └─21512 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-scheduler --config-file=/etc/workloadmgr/workloadmgr.conf ``` ```bash systemctl status wlm-workloads ###### ● wlm-workloads.service - workloadmanager workloads service Loaded: loaded (/etc/systemd/system/wlm-workloads.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2020-04-22 09:15:43 UTC; 1 day 2h ago Main PID: 20079 (python) Tasks: 33 CGroup: /system.slice/wlm-workloads.service ├─20079 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf ├─20180 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf [...] ├─20181 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf ├─20233 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf ├─20236 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf └─20237 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf ``` ## Check the TrilioVault pacemaker and nginx cluster The second component to check the TrilioVault Appliance's health is the nginx and pacemaker cluster. ```bash pcs status ###### Cluster name: triliovault WARNINGS: Corosync and pacemaker node names do not match (IPs used in setup?) Stack: corosync Current DC: om_tvm (version 1.1.19-8.el7_6.1-c3c624ea3d) - partition with quorum Last updated: Wed Dec 5 12:25:02 2018 Last change: Wed Dec 5 09:20:08 2018 by root via cibadmin on om_tvm 1 node configured 4 resources configured Online: [ om_tvm ] Full list of resources: virtual_ip (ocf::'heartbeat:IPaddr2): Started om_tvm wlm-api (systemd:wlm-api): Started om_tvm wlm-scheduler (systemd:wlm-scheduler): Started om_tvm Clone Set: lb_nginx-clone [lb_nginx] Started: [ om_tvm ] Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled ``` ## Verify API connectivity from the RHV-Manager The RHV-Manager is doing all API calls towards the TrilioVault Appliance.\ Therefore it is helpful to do a quick API connectivity check using curl. The following curl command lists the available workload-types and verfifies that the connection is available and working: ```bash curl -k -XGET https://30.30.1.11:8780/v1/admin/workload_types/detail -H "Content-Type: application/json" -H "X-OvirtAuth-User: admin@internal" -H "X-OvirtAuth-Password: password" ###### {"workload_types": [{"status": "available", "user_id": "admin@internal", "name": "Parallel", "links": [{"href": "https://myapp/v1/admin/workloadtypes/2ddd528d-c9b4-4d7e-8722-cc395140255a", "rel": "self"}, {"href": "https://myapp/admin/workloadtypes/2ddd528d-c9b4-4d7e-8722-cc395140255a", "rel": "bookmark"}], "created_at": "2020-04-02T15:38:51.000000", "updated_at": "2020-04-02T15:38:51.000000", "metadata": [], "is_public": true, "project_id": "admin", "id": "2ddd528d-c9b4-4d7e-8722-cc395140255a", "description": "Parallel workload that snapshots VM in the specified order"}, {"status": "available", "user_id": "admin@internal", "name": "Serial", "links": [{"href": "https://myapp/v1/admin/workloadtypes/f82ce76f-17fe-438b-aa37-7a023058e50d", "rel": "self"}, {"href": "https://myapp/admin/workloadtypes/f82ce76f-17fe-438b-aa37-7a023058e50d", "rel": "bookmark"}], "created_at": "2020-04-02T15:38:47.000000", "updated_at": "2020-04-02T15:38:47.000000", "metadata": [], "is_public": true, "project_id": "admin", "id": "f82ce76f-17fe-438b-aa37-7a023058e50d", "description": "Serial workload that snapshots VM in the specified order"}]} ``` ## Verify the ovirt-imageio services are up and running TrilioVault is extending the already exiting ovirt-imageio services.\ The installation of these extensions does check if the ovirt-services come up.\ Still it is a good call to verify again afterwards: On the RHV-Manager check the ovirt-imageio-proxy service: ```bash systemctl status ovirt-imageio-proxy ###### ● ovirt-imageio-proxy.service - oVirt ImageIO Proxy Loaded: loaded (/usr/lib/systemd/system/ovirt-imageio-proxy.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2020-04-08 05:05:25 UTC; 2 weeks 1 days ago Main PID: 1834 (python) CGroup: /system.slice/ovirt-imageio-proxy.service └─1834 bin/python proxy/ovirt-imageio-proxy ``` On the RHV-Host check the ovirt-imageio-daemon service: ```bash systemctl status ovirt-imageio-daemon ###### ● ovirt-imageio-daemon.service - oVirt ImageIO Daemon Loaded: loaded (/usr/lib/systemd/system/ovirt-imageio-daemon.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2020-04-08 04:40:50 UTC; 2 weeks 1 days ago Main PID: 1442 (python) Tasks: 4 CGroup: /system.slice/ovirt-imageio-daemon.service └─1442 /opt/ovirt-imageio/bin/python daemon/ovirt-imageio-daemon ``` ## Verify the NFS Volume is correctly mounted TrilioVault mounts the NFS Backup Target to the TrilioVault Appliance and RHV-Hosts. To verify those are correctly mounted it is recommended to do the following checks. First df -h looking for /var/triliovault-mounts/\ ```bash df -h ###### Filesystem Size Used Avail Use% Mounted on devtmpfs 63G 0 63G 0% /dev tmpfs 63G 16K 63G 1% /dev/shm tmpfs 63G 35M 63G 1% /run tmpfs 63G 0 63G 0% /sys/fs/cgroup /dev/mapper/rhvh-rhvh--4.3.8.1--0.20200126.0+1 7.1T 3.7G 6.8T 1% / /dev/sda2 976M 198M 712M 22% /boot /dev/mapper/rhvh-var 15G 1.9G 12G 14% /var /dev/mapper/rhvh-home 976M 2.6M 907M 1% /home /dev/mapper/rhvh-tmp 976M 2.6M 907M 1% /tmp /dev/mapper/rhvh-var_log 7.8G 230M 7.2G 4% /var/log /dev/mapper/rhvh-var_log_audit 2.0G 17M 1.8G 1% /var/log/audit /dev/mapper/rhvh-var_crash 9.8G 37M 9.2G 1% /var/crash 30.30.1.4:/rhv_backup 2.0T 5.3G 1.9T 1% /var/triliovault-mounts/MzAuMzAuMS40Oi9yaHZfYmFja3Vw 30.30.1.4:/rhv_data 2.0T 37G 2.0T 2% /rhev/data-center/mnt/30.30.1.4:_rhv__data tmpfs 13G 0 13G 0% /run/user/0 30.30.1.4:/rhv_iso 2.0T 37G 2.0T 2% /rhev/data-center/mnt/30.30.1.4:_rhv__iso ``` Secondly do a read / write / delete test as the user vdsm:kvm (uid = 36 / gid = 36) from the TrilioVault Appliance and the RHV-Host. ```bash su vdsm ###### [vdsm@rhv-tvm MzAuMzAuMS40Oi9yaHZfYmFja3Vw]$ touch foo [vdsm@rhv-tvm MzAuMzAuMS40Oi9yaHZfYmFja3Vw]$ ll total 24 drwxr-xr-x 3 vdsm kvm 4096 Apr 2 17:27 contego_tasks -rw-r--r-- 1 vdsm kvm 0 Apr 23 12:25 foo drwxr-xr-x 2 vdsm kvm 4096 Apr 2 15:38 test-cloud-id drwxr-xr-x 10 vdsm kvm 4096 Apr 22 11:00 workload_1540698c-8e22-4dd1-a898-8f49cd1a898c drwxr-xr-x 9 vdsm kvm 4096 Apr 8 15:21 workload_51517816-6d5a-4fce-9ac7-46ee1e09052c drwxr-xr-x 6 vdsm kvm 4096 Apr 22 11:30 workload_77fb42d2-8d34-4b8d-bfd5-4263397b636c drwxr-xr-x 5 vdsm kvm 4096 Apr 23 06:15 workload_85bf16ed-d4fd-49a6-a753-98c5ca6e906b [vdsm@rhv-tvm MzAuMzAuMS40Oi9yaHZfYmFja3Vw]$ rm foo [vdsm@rhv-tvm MzAuMzAuMS40Oi9yaHZfYmFja3Vw]$ ll total 24 drwxr-xr-x 3 vdsm kvm 4096 Apr 2 17:27 contego_tasks drwxr-xr-x 2 vdsm kvm 4096 Apr 2 15:38 test-cloud-id drwxr-xr-x 10 vdsm kvm 4096 Apr 22 11:00 workload_1540698c-8e22-4dd1-a898-8f49cd1a898c drwxr-xr-x 9 vdsm kvm 4096 Apr 8 15:21 workload_51517816-6d5a-4fce-9ac7-46ee1e09052c drwxr-xr-x 6 vdsm kvm 4096 Apr 22 11:30 workload_77fb42d2-8d34-4b8d-bfd5-4263397b636c drwxr-xr-x 5 vdsm kvm 4096 Apr 23 06:15 workload_85bf16ed-d4fd-49a6-a753-98c5ca6e906b [vdsm@rhv-tvm MzAuMzAuMS40Oi9yaHZfYmFja3Vw]$ ```