After the installation and configuration of TrilioVault for Openstack did succeed the following steps can be done to verify that the TrilioVault installation is healthy.
TrilioVault is using 3 main services on the TrilioVault Appliance:
wlm-api
wlm-scheduler
wlm-workloads
Those can be verified to be up and running using the systemctl status
command.
systemctl status wlm-api######● wlm-api.service - Cluster Controlled wlm-apiLoaded: loaded (/etc/systemd/system/wlm-api.service; disabled; vendor preset: disabled)Drop-In: /run/systemd/system/wlm-api.service.d└─50-pacemaker.confActive: active (running) since Wed 2020-04-22 09:17:05 UTC; 1 day 2h agoMain PID: 21265 (python)Tasks: 1CGroup: /system.slice/wlm-api.service└─21265 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-api --config-file=/etc/workloadmgr/workloadmgr.conf
systemctl status wlm-scheduler######● wlm-scheduler.service - Cluster Controlled wlm-schedulerLoaded: loaded (/etc/systemd/system/wlm-scheduler.service; disabled; vendor preset: disabled)Drop-In: /run/systemd/system/wlm-scheduler.service.d└─50-pacemaker.confActive: active (running) since Wed 2020-04-22 09:17:17 UTC; 1 day 2h agoMain PID: 21512 (python)Tasks: 1CGroup: /system.slice/wlm-scheduler.service└─21512 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-scheduler --config-file=/etc/workloadmgr/workloadmgr.conf
systemctl status wlm-workloads######● wlm-workloads.service - workloadmanager workloads serviceLoaded: loaded (/etc/systemd/system/wlm-workloads.service; enabled; vendor preset: disabled)Active: active (running) since Wed 2020-04-22 09:15:43 UTC; 1 day 2h agoMain PID: 20079 (python)Tasks: 33CGroup: /system.slice/wlm-workloads.service├─20079 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf├─20180 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf[...]├─20181 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf├─20233 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf├─20236 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf└─20237 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf
The second component to check the TrilioVault Appliance's health is the nginx and pacemaker cluster.
pcs status######Cluster name: triliovaultWARNINGS:Corosync and pacemaker node names do not match (IPs used in setup?)Stack: corosyncCurrent DC: om_tvm (version 1.1.19-8.el7_6.1-c3c624ea3d) -partition with quorumLast updated: Wed Dec 5 12:25:02 2018Last change: Wed Dec 5 09:20:08 2018 by root via cibadmin on om_tvm1 node configured4 resources configuredOnline: [ om_tvm ]Full list of resources:virtual_ip (ocf::'heartbeat:IPaddr2): Started om_tvmwlm-api (systemd:wlm-api): Started om_tvmwlm-scheduler (systemd:wlm-scheduler): Started om_tvmClone Set: lb_nginx-clone [lb_nginx]Started: [ om_tvm ]Daemon Status:corosync: active/enabledpacemaker: active/enabledpcsd: active/enabled
Checking the availability of the TrilioVault API on the chosen endpoints is recommended.
The following example curl command lists the available workload-types and verifies that the connection is available and working:
curl http://10.10.2.34:8780/v1/8e16700ae3614da4ba80a4e57d60cdb9/workload_types/detail -X GET -H "X-Auth-Project-Id: admin" -H "User-Agent: python-workloadmgrclient" -H "Accept: application/json" -H "X-Auth-Token: gAAAAABe40NVFEtJeePpk1F9QGGh1LiGnHJVLlgZx9t0HRrK9rC5vqKZJRkpAcW1oPH6Q9K9peuHiQrBHEs1-g75Na4xOEESR0LmQJUZP6n37fLfDL_D-hlnjHJZ68iNisIP1fkm9FGSyoyt6IqjO9E7_YVRCTCqNLJ67ZkqHuJh1CXwShvjvjw
Please check the API guide for more commands and how to generate the X-Auth-Token.
The tvault-contego service is the Data Mover that got installed on all compute nodes nodes. It is recommended to check its status after the installation.
openstack compute service list+----+----------------------+--------------------+----------+----------+-------+----------------------------+| ID | Binary | Host | Zone | Status | State | Updated At |+----+----------------------+--------------------+----------+----------+-------+----------------------------+| 7 | nova-conductor | upstreamcontroller | internal | enabled | up | 2020-06-12T09:13:55.000000 || 8 | nova-scheduler | upstreamcontroller | internal | enabled | up | 2020-06-12T09:13:54.000000 || 9 | nova-consoleauth | upstreamcontroller | internal | enabled | up | 2020-06-12T09:13:52.000000 || 10 | nova-compute | upstreamcompute1 | US-East | enabled | up | 2020-06-12T09:13:50.000000 || 11 | nova-compute | upstreamcompute2 | US-West | enabled | up | 2020-06-12T09:13:51.000000 || 12 | nova-contego_3.0.174 | upstreamcompute2 | internal | enabled | up | 2020-06-12T09:13:51.000000 || 13 | nova-contego_3.0.174 | upstreamcompute1 | internal | enabled | up | 2020-06-12T09:13:47.000000 |+----+---------------------+--------------------+----------+----------+-------+----------------------------+
[root@upstreamcompute1 ~]# systemctl status tvault-contego.service● tvault-contego.service - Tvault contegoLoaded: loaded (/etc/systemd/system/tvault-contego.service; enabled; vendor preset: disabled)Active: active (running) since Wed 2020-06-10 10:07:28 EDT; 1 day 19h agoMain PID: 10384 (python)Tasks: 21CGroup: /system.slice/tvault-contego.service└─10384 /usr/bin/python /usr/bin/tvault-contego --config-file=/etc...Jun 12 03:15:33 upstreamcompute1 python[10384]: libvirt: QEMU Driver error :...dJun 12 03:15:33 upstreamcompute1 python[10384]: libvirt: QEMU Driver error :...dJun 12 03:16:11 upstreamcompute1 python[10384]: libvirt: QEMU Driver error :...dJun 12 03:16:31 upstreamcompute1 sudo[13977]: nova : TTY=unknown ; PWD=/...nJun 12 03:16:33 upstreamcompute1 sudo[14004]: nova : TTY=unknown ; PWD=/ ...Jun 12 05:15:33 upstreamcompute1 python[10384]: libvirt: QEMU Driver error :...dJun 12 05:15:33 upstreamcompute1 python[10384]: libvirt: QEMU Driver error :...dJun 12 05:16:11 upstreamcompute1 python[10384]: libvirt: QEMU Driver error :...dJun 12 05:16:29 upstreamcompute1 sudo[23356]: nova : TTY=unknown ; PWD=/...nJun 12 05:16:32 upstreamcompute1 sudo[23422]: nova : TTY=unknown ; PWD=/ ...Hint: Some lines were ellipsized, use -l to show in full.
TrilioVault mounts the NFS Backup Target to the TrilioVault Appliance and Compute nodes.
To verify those are correctly mounted it is recommended to do the following checks.
First df -h looking for /var/triliovault-mounts/<hash-value>
df -h######Filesystem Size Used Avail Use% Mounted ondevtmpfs 63G 0 63G 0% /devtmpfs 63G 16K 63G 1% /dev/shmtmpfs 63G 35M 63G 1% /runtmpfs 63G 0 63G 0% /sys/fs/cgroup/dev/mapper/rhvh-rhvh--4.3.8.1--0.20200126.0+1 7.1T 3.7G 6.8T 1% //dev/sda2 976M 198M 712M 22% /boot/dev/mapper/rhvh-var 15G 1.9G 12G 14% /var/dev/mapper/rhvh-home 976M 2.6M 907M 1% /home/dev/mapper/rhvh-tmp 976M 2.6M 907M 1% /tmp/dev/mapper/rhvh-var_log 7.8G 230M 7.2G 4% /var/log/dev/mapper/rhvh-var_log_audit 2.0G 17M 1.8G 1% /var/log/audit/dev/mapper/rhvh-var_crash 9.8G 37M 9.2G 1% /var/crash30.30.1.4:/rhv_backup 2.0T 5.3G 1.9T 1% /var/triliovault-mounts/MzAuMzAuMS40Oi9yaHZfYmFja3Vw30.30.1.4:/rhv_data 2.0T 37G 2.0T 2% /rhev/data-center/mnt/30.30.1.4:_rhv__datatmpfs 13G 0 13G 0% /run/user/030.30.1.4:/rhv_iso 2.0T 37G 2.0T 2% /rhev/data-center/mnt/30.30.1.4:_rhv__iso
Secondly do a read / write / delete test as the user nova:nova (uid = 36 / gid = 36) from the TrilioVault Appliance and the RHV-Host.
su nova######[nova@tvm MTAuMTAuMi4yMDovdXBzdHJlYW0=]$ touch foo[nova@tvm MTAuMTAuMi4yMDovdXBzdHJlYW0=]$ lltotal 24drwxr-xr-x 3 nova nova 4096 Apr 2 17:27 contego_tasks-rw-r--r-- 1 nova nova 0 Apr 23 12:25 foodrwxr-xr-x 2 nova nova 4096 Apr 2 15:38 test-cloud-iddrwxr-xr-x 10 nova nova 4096 Apr 22 11:00 workload_1540698c-8e22-4dd1-a898-8f49cd1a898cdrwxr-xr-x 9 nova nova 4096 Apr 8 15:21 workload_51517816-6d5a-4fce-9ac7-46ee1e09052cdrwxr-xr-x 6 nova nova 4096 Apr 22 11:30 workload_77fb42d2-8d34-4b8d-bfd5-4263397b636cdrwxr-xr-x 5 nova nova 4096 Apr 23 06:15 workload_85bf16ed-d4fd-49a6-a753-98c5ca6e906b[nova@tvm MTAuMTAuMi4yMDovdXBzdHJlYW0=]$ rm foo[nova@tvm MTAuMTAuMi4yMDovdXBzdHJlYW0=]$ lltotal 24drwxr-xr-x 3 nova nova 4096 Apr 2 17:27 contego_tasksdrwxr-xr-x 2 nova nova 4096 Apr 2 15:38 test-cloud-iddrwxr-xr-x 10 nova nova 4096 Apr 22 11:00 workload_1540698c-8e22-4dd1-a898-8f49cd1a898cdrwxr-xr-x 9 nova nova 4096 Apr 8 15:21 workload_51517816-6d5a-4fce-9ac7-46ee1e09052cdrwxr-xr-x 6 nova nova 4096 Apr 22 11:30 workload_77fb42d2-8d34-4b8d-bfd5-4263397b636cdrwxr-xr-x 5 nova nova 4096 Apr 23 06:15 workload_85bf16ed-d4fd-49a6-a753-98c5ca6e906b