Search…
Post Installation Health-Check
After the installation and configuration of TrilioVault for Openstack did succeed the following steps can be done to verify that the TrilioVault installation is healthy.

Verify the TrilioVault Appliance services are up

TrilioVault is using 3 main services on the TrilioVault Appliance:
    wlm-api
    wlm-scheduler
    wlm-workloads
Those can be verified to be up and running using the systemctl status command.
1
systemctl status wlm-api
2
######
3
● wlm-api.service - Cluster Controlled wlm-api
4
Loaded: loaded (/etc/systemd/system/wlm-api.service; disabled; vendor preset: disabled)
5
Drop-In: /run/systemd/system/wlm-api.service.d
6
└─50-pacemaker.conf
7
Active: active (running) since Wed 2020-04-22 09:17:05 UTC; 1 day 2h ago
8
Main PID: 21265 (python)
9
Tasks: 1
10
CGroup: /system.slice/wlm-api.service
11
└─21265 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-api --config-file=/etc/workloadmgr/workloadmgr.conf
Copied!
1
systemctl status wlm-scheduler
2
######
3
● wlm-scheduler.service - Cluster Controlled wlm-scheduler
4
Loaded: loaded (/etc/systemd/system/wlm-scheduler.service; disabled; vendor preset: disabled)
5
Drop-In: /run/systemd/system/wlm-scheduler.service.d
6
└─50-pacemaker.conf
7
Active: active (running) since Wed 2020-04-22 09:17:17 UTC; 1 day 2h ago
8
Main PID: 21512 (python)
9
Tasks: 1
10
CGroup: /system.slice/wlm-scheduler.service
11
└─21512 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-scheduler --config-file=/etc/workloadmgr/workloadmgr.conf
Copied!
1
systemctl status wlm-workloads
2
######
3
● wlm-workloads.service - workloadmanager workloads service
4
Loaded: loaded (/etc/systemd/system/wlm-workloads.service; enabled; vendor preset: disabled)
5
Active: active (running) since Wed 2020-04-22 09:15:43 UTC; 1 day 2h ago
6
Main PID: 20079 (python)
7
Tasks: 33
8
CGroup: /system.slice/wlm-workloads.service
9
├─20079 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf
10
├─20180 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf
11
[...]
12
├─20181 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf
13
├─20233 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf
14
├─20236 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf
15
└─20237 /home/rhv/myansible/bin/python /usr/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf
Copied!

Check the TrilioVault pacemaker and nginx cluster

The second component to check the TrilioVault Appliance's health is the nginx and pacemaker cluster.
1
pcs status
2
######
3
Cluster name: triliovault
4
5
WARNINGS:
6
Corosync and pacemaker node names do not match (IPs used in setup?)
7
Stack: corosync
8
Current DC: om_tvm (version 1.1.19-8.el7_6.1-c3c624ea3d) -
9
partition with quorum
10
Last updated: Wed Dec 5 12:25:02 2018
11
Last change: Wed Dec 5 09:20:08 2018 by root via cibadmin on om_tvm
12
1 node configured
13
4 resources configured
14
15
Online: [ om_tvm ]
16
Full list of resources:
17
virtual_ip (ocf::'heartbeat:IPaddr2): Started om_tvm
18
wlm-api (systemd:wlm-api): Started om_tvm
19
wlm-scheduler (systemd:wlm-scheduler): Started om_tvm
20
Clone Set: lb_nginx-clone [lb_nginx]
21
Started: [ om_tvm ]
22
Daemon Status:
23
corosync: active/enabled
24
pacemaker: active/enabled
25
pcsd: active/enabled
Copied!

Verify API connectivity of the TrilioVault Appliance

Checking the availability of the TrilioVault API on the chosen endpoints is recommended.
The following example curl command lists the available workload-types and verifies that the connection is available and working:
1
curl http://10.10.2.34:8780/v1/8e16700ae3614da4ba80a4e57d60cdb9/workload_types/detail -X GET -H "X-Auth-Project-Id: admin" -H "User-Agent: python-workloadmgrclient" -H "Accept: application/json" -H "X-Auth-Token: gAAAAABe40NVFEtJeePpk1F9QGGh1LiGnHJVLlgZx9t0HRrK9rC5vqKZJRkpAcW1oPH6Q9K9peuHiQrBHEs1-g75Na4xOEESR0LmQJUZP6n37fLfDL_D-hlnjHJZ68iNisIP1fkm9FGSyoyt6IqjO9E7_YVRCTCqNLJ67ZkqHuJh1CXwShvjvjw
Copied!
Please check the API guide for more commands and how to generate the X-Auth-Token.

Verify the tvault-contego services are up and running

The tvault-contego service is the Data Mover that got installed on all compute nodes nodes. It is recommended to check its status after the installation.
1
openstack compute service list
2
+----+----------------------+--------------------+----------+----------+-------+----------------------------+
3
| ID | Binary | Host | Zone | Status | State | Updated At |
4
+----+----------------------+--------------------+----------+----------+-------+----------------------------+
5
| 7 | nova-conductor | upstreamcontroller | internal | enabled | up | 2020-06-12T09:13:55.000000 |
6
| 8 | nova-scheduler | upstreamcontroller | internal | enabled | up | 2020-06-12T09:13:54.000000 |
7
| 9 | nova-consoleauth | upstreamcontroller | internal | enabled | up | 2020-06-12T09:13:52.000000 |
8
| 10 | nova-compute | upstreamcompute1 | US-East | enabled | up | 2020-06-12T09:13:50.000000 |
9
| 11 | nova-compute | upstreamcompute2 | US-West | enabled | up | 2020-06-12T09:13:51.000000 |
10
| 12 | nova-contego_3.0.174 | upstreamcompute2 | internal | enabled | up | 2020-06-12T09:13:51.000000 |
11
| 13 | nova-contego_3.0.174 | upstreamcompute1 | internal | enabled | up | 2020-06-12T09:13:47.000000 |
12
+----+---------------------+--------------------+----------+----------+-------+----------------------------+
Copied!
1
[[email protected] ~]# systemctl status tvault-contego.service
2
● tvault-contego.service - Tvault contego
3
Loaded: loaded (/etc/systemd/system/tvault-contego.service; enabled; vendor preset: disabled)
4
Active: active (running) since Wed 2020-06-10 10:07:28 EDT; 1 day 19h ago
5
Main PID: 10384 (python)
6
Tasks: 21
7
CGroup: /system.slice/tvault-contego.service
8
└─10384 /usr/bin/python /usr/bin/tvault-contego --config-file=/etc...
9
10
Jun 12 03:15:33 upstreamcompute1 python[10384]: libvirt: QEMU Driver error :...d
11
Jun 12 03:15:33 upstreamcompute1 python[10384]: libvirt: QEMU Driver error :...d
12
Jun 12 03:16:11 upstreamcompute1 python[10384]: libvirt: QEMU Driver error :...d
13
Jun 12 03:16:31 upstreamcompute1 sudo[13977]: nova : TTY=unknown ; PWD=/...n
14
Jun 12 03:16:33 upstreamcompute1 sudo[14004]: nova : TTY=unknown ; PWD=/ ...
15
Jun 12 05:15:33 upstreamcompute1 python[10384]: libvirt: QEMU Driver error :...d
16
Jun 12 05:15:33 upstreamcompute1 python[10384]: libvirt: QEMU Driver error :...d
17
Jun 12 05:16:11 upstreamcompute1 python[10384]: libvirt: QEMU Driver error :...d
18
Jun 12 05:16:29 upstreamcompute1 sudo[23356]: nova : TTY=unknown ; PWD=/...n
19
Jun 12 05:16:32 upstreamcompute1 sudo[23422]: nova : TTY=unknown ; PWD=/ ...
20
Hint: Some lines were ellipsized, use -l to show in full.
Copied!

Verify the NFS Volume is correctly mounted

TrilioVault mounts the NFS Backup Target to the TrilioVault Appliance and Compute nodes.
To verify those are correctly mounted it is recommended to do the following checks.
First df -h looking for /var/triliovault-mounts/<hash-value>
1
df -h
2
######
3
Filesystem Size Used Avail Use% Mounted on
4
devtmpfs 63G 0 63G 0% /dev
5
tmpfs 63G 16K 63G 1% /dev/shm
6
tmpfs 63G 35M 63G 1% /run
7
tmpfs 63G 0 63G 0% /sys/fs/cgroup
8
/dev/mapper/rhvh-rhvh--4.3.8.1--0.20200126.0+1 7.1T 3.7G 6.8T 1% /
9
/dev/sda2 976M 198M 712M 22% /boot
10
/dev/mapper/rhvh-var 15G 1.9G 12G 14% /var
11
/dev/mapper/rhvh-home 976M 2.6M 907M 1% /home
12
/dev/mapper/rhvh-tmp 976M 2.6M 907M 1% /tmp
13
/dev/mapper/rhvh-var_log 7.8G 230M 7.2G 4% /var/log
14
/dev/mapper/rhvh-var_log_audit 2.0G 17M 1.8G 1% /var/log/audit
15
/dev/mapper/rhvh-var_crash 9.8G 37M 9.2G 1% /var/crash
16
30.30.1.4:/rhv_backup 2.0T 5.3G 1.9T 1% /var/triliovault-mounts/MzAuMzAuMS40Oi9yaHZfYmFja3Vw
17
30.30.1.4:/rhv_data 2.0T 37G 2.0T 2% /rhev/data-center/mnt/30.30.1.4:_rhv__data
18
tmpfs 13G 0 13G 0% /run/user/0
19
30.30.1.4:/rhv_iso 2.0T 37G 2.0T 2% /rhev/data-center/mnt/30.30.1.4:_rhv__iso
Copied!
Secondly do a read / write / delete test as the user nova:nova (uid = 36 / gid = 36) from the TrilioVault Appliance and the RHV-Host.
1
su nova
2
######
3
[[email protected] MTAuMTAuMi4yMDovdXBzdHJlYW0=]$ touch foo
4
[[email protected] MTAuMTAuMi4yMDovdXBzdHJlYW0=]$ ll
5
total 24
6
drwxr-xr-x 3 nova nova 4096 Apr 2 17:27 contego_tasks
7
-rw-r--r-- 1 nova nova 0 Apr 23 12:25 foo
8
drwxr-xr-x 2 nova nova 4096 Apr 2 15:38 test-cloud-id
9
drwxr-xr-x 10 nova nova 4096 Apr 22 11:00 workload_1540698c-8e22-4dd1-a898-8f49cd1a898c
10
drwxr-xr-x 9 nova nova 4096 Apr 8 15:21 workload_51517816-6d5a-4fce-9ac7-46ee1e09052c
11
drwxr-xr-x 6 nova nova 4096 Apr 22 11:30 workload_77fb42d2-8d34-4b8d-bfd5-4263397b636c
12
drwxr-xr-x 5 nova nova 4096 Apr 23 06:15 workload_85bf16ed-d4fd-49a6-a753-98c5ca6e906b
13
[[email protected] MTAuMTAuMi4yMDovdXBzdHJlYW0=]$ rm foo
14
[[email protected] MTAuMTAuMi4yMDovdXBzdHJlYW0=]$ ll
15
total 24
16
drwxr-xr-x 3 nova nova 4096 Apr 2 17:27 contego_tasks
17
drwxr-xr-x 2 nova nova 4096 Apr 2 15:38 test-cloud-id
18
drwxr-xr-x 10 nova nova 4096 Apr 22 11:00 workload_1540698c-8e22-4dd1-a898-8f49cd1a898c
19
drwxr-xr-x 9 nova nova 4096 Apr 8 15:21 workload_51517816-6d5a-4fce-9ac7-46ee1e09052c
20
drwxr-xr-x 6 nova nova 4096 Apr 22 11:30 workload_77fb42d2-8d34-4b8d-bfd5-4263397b636c
21
drwxr-xr-x 5 nova nova 4096 Apr 23 06:15 workload_85bf16ed-d4fd-49a6-a753-98c5ca6e906b
Copied!
Last modified 1yr ago