# Healthcheck of Trilio Trilio is composed of multiple services, which can be checked in case of any errors. ## Verify the Trilio Appliance ### Verify the services are up Trilio is using 4 main services on the Trilio Appliance: * `wlm-api` * `wlm-scheduler` * `wlm-workloads` * `wlm-cron` ``` systemctl | grep wlm wlm-api.service loaded active running workloadmanager api service wlm-cron.service loaded active running Cluster Controlled wlm-cron wlm-scheduler.service loaded active running Cluster Controlled wlm-scheduler wlm-workloads.service loaded active running workloadmanager workloads service ``` Those can be verified to be up and running using the `systemctl status` command. ```bash systemctl status wlm-api ###### ● wlm-api.service - workloadmanager api service Loaded: loaded (/etc/systemd/system/wlm-api.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2021-11-02 19:41:19 UTC; 2 months 21 days ago Main PID: 4688 (workloadmgr-api) CGroup: /system.slice/wlm-api.service ├─4688 /home/stack/myansible/bin/python3 /home/stack/myansible/bin/workloadmgr-api --config-file=/etc/workloadmgr/workloadmgr.conf ``` ```bash systemctl status wlm-scheduler ###### ● wlm-scheduler.service - Cluster Controlled wlm-scheduler Loaded: loaded (/etc/systemd/system/wlm-scheduler.service; disabled; vendor preset: disabled) Drop-In: /run/systemd/system/wlm-scheduler.service.d └─50-pacemaker.conf Active: active (running) since Sat 2022-01-22 13:49:28 UTC; 1 day 23h ago Main PID: 9342 (workloadmgr-sch) CGroup: /system.slice/wlm-scheduler.service └─9342 /home/stack/myansible/bin/python3 /home/stack/myansible/bin/workloadmgr-scheduler --config-file=/etc/workloadmgr/workloadmgr.conf ``` ```bash systemctl status wlm-workloads ###### ● wlm-workloads.service - workloadmanager workloads service Loaded: loaded (/etc/systemd/system/wlm-workloads.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2021-11-02 19:51:05 UTC; 2 months 21 days ago Main PID: 606 (workloadmgr-wor) CGroup: /system.slice/wlm-workloads.service ├─ 606 /home/stack/myansible/bin/python3 /home/stack/myansible/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf ``` ``` systemctl status wlm-cron ###### ● wlm-cron.service - Cluster Controlled wlm-cron Loaded: loaded (/etc/systemd/system/wlm-cron.service; disabled; vendor preset: disabled) Drop-In: /run/systemd/system/wlm-cron.service.d └─50-pacemaker.conf Active: active (running) since Sat 2022-01-22 13:49:28 UTC; 1 day 23h ago Main PID: 9209 (workloadmgr-cro) CGroup: /system.slice/wlm-cron.service ├─9209 /home/stack/myansible/bin/python3 /home/stack/myansible/bin/workloadmgr-cron --config-file=/etc/workloadmgr/workloadmgr.conf ``` ### Check the Trilio pacemaker and nginx cluster The second component to check the Trilio Appliance's health is the nginx and pacemaker cluster. ```bash pcs status ###### Cluster name: triliovault WARNINGS: Corosync and pacemaker node names do not match (IPs used in setup?) Stack: corosync Current DC: TVM1 (version 1.1.21-4.el7-f14e36fd43) - partition with quorum Last updated: Mon Jan 24 13:42:01 2022 Last change: Tue Nov 2 19:07:04 2021 by root via crm_resource on TVM2 3 nodes configured 9 resources configured Online: [ TVM1 TVM2 TVM3 ] Full list of resources: virtual_ip (ocf::heartbeat:IPaddr2): Started TVM2 virtual_ip_public (ocf::heartbeat:IPaddr2): Started TVM2 virtual_ip_admin (ocf::heartbeat:IPaddr2): Started TVM2 virtual_ip_internal (ocf::heartbeat:IPaddr2): Started TVM2 wlm-cron (systemd:wlm-cron): Started TVM2 wlm-scheduler (systemd:wlm-scheduler): Started TVM2 Clone Set: lb_nginx-clone [lb_nginx] Started: [ TVM2 ] Stopped: [ TVM1 TVM3 ] Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled ``` ## Verify API connectivity of the Trilio Appliance Checking the availability of the Trilio API on the chosen endpoints is recommended. The following example curl command lists the available workload-types and verifies that the connection is available and working: ```bash curl http://10.10.2.34:8780/v1/8e16700ae3614da4ba80a4e57d60cdb9/workload_types/detail -X GET -H "X-Auth-Project-Id: admin" -H "User-Agent: python-workloadmgrclient" -H "Accept: application/json" -H "X-Auth-Token: gAAAAABe40NVFEtJeePpk1F9QGGh1LiGnHJVLlgZx9t0HRrK9rC5vqKZJRkpAcW1oPH6Q9K9peuHiQrBHEs1-g75Na4xOEESR0LmQJUZP6n37fLfDL_D-hlnjHJZ68iNisIP1fkm9FGSyoyt6IqjO9E7_YVRCTCqNLJ67ZkqHuJh1CXwShvjvjw ``` {% hint style="info" %} Please check the API guide for more commands and how to generate the X-Auth-Token. {% endhint %} ## Verify Trilio components on OpenStack ### On OpenStack Ansible #### Datamover-API service (dmapi) The dmapi service has its own Keystone endpoints, which should be checked in addition to the actual service status. ``` root@ansible:~# openstack endpoint list | grep dmapi | 190db2ce033e44f89de73abcbf12804e | US-WEST-2 | dmapi | datamover | True | public | https://osa-victoria-ubuntu20-2.triliodata.demo:8784/v2 | | dec1a323791b49f0ac7901a2dc806ee2 | US-WEST-2 | dmapi | datamover | True | admin | http://10.10.10.154:8784/v2 | | f8c4162c9c1246ffb0190d0d093c48af | US-WEST-2 | dmapi | datamover | True | internal | http://10.10.10.154:8784/v2 | root@ansible:~# curl http://10.10.10.154:8784 {"versions": [{"id": "v2.0", "status": "SUPPORTED", "version": "", "min_version": "", "updated": "2011-01-21T11:33:21Z", "links": [{"rel": "self", "href": "h ``` In order to check the dmapi service go to dmapi container which is residing on controller nodes and run below command ``` lxc-attach -n (go to dmapi conatiner) root@controller-dmapi-container-08df1e06:~# systemctl status tvault-datamover-api.service ● tvault-datamover-api.service - TrilioData DataMover API service Loaded: loaded (/lib/systemd/system/tvault-datamover-api.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2022-01-12 11:53:39 UTC; 1 day 17h ago Main PID: 23888 (dmapi-api) Tasks: 289 (limit: 57729) Memory: 607.7M CGroup: /system.slice/tvault-datamover-api.service ├─23888 /usr/bin/python3 /usr/bin/dmapi-api ├─23893 /usr/bin/python3 /usr/bin/dmapi-api ├─23894 /usr/bin/python3 /usr/bin/dmapi-api ├─23895 /usr/bin/python3 /usr/bin/dmapi-api ├─23896 /usr/bin/python3 /usr/bin/dmapi-api ├─23897 /usr/bin/python3 /usr/bin/dmapi-api ├─23898 /usr/bin/python3 /usr/bin/dmapi-api ├─23899 /usr/bin/python3 /usr/bin/dmapi-api ├─23900 /usr/bin/python3 /usr/bin/dmapi-api ├─23901 /usr/bin/python3 /usr/bin/dmapi-api ├─23902 /usr/bin/python3 /usr/bin/dmapi-api ├─23903 /usr/bin/python3 /usr/bin/dmapi-api ├─23904 /usr/bin/python3 /usr/bin/dmapi-api ├─23905 /usr/bin/python3 /usr/bin/dmapi-api ├─23906 /usr/bin/python3 /usr/bin/dmapi-api ├─23907 /usr/bin/python3 /usr/bin/dmapi-api └─23908 /usr/bin/python3 /usr/bin/dmapi-api Jan 12 11:53:39 controller-dmapi-container-08df1e06 systemd[1]: Started TrilioData DataMover API service. Jan 12 11:53:40 controller-dmapi-container-08df1e06 dmapi-api[23888]: Could not load Jan 12 11:53:40 controller-dmapi-container-08df1e06 dmapi-api[23888]: Could not load ``` #### Datamover service (tvault-contego) The datamover service is running on each compute node. Logging to compute node and run below command ``` root@compute:~# systemctl status tvault-contego ● tvault-contego.service - Tvault contego Loaded: loaded (/etc/systemd/system/tvault-contego.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2022-01-14 05:45:19 UTC; 2s ago Main PID: 1489651 (python3) Tasks: 19 (limit: 67404) Memory: 6.7G (max: 10.0G) CGroup: /system.slice/tvault-contego.service ├─ 998543 /bin/qemu-nbd -c /dev/nbd45 --object secret,id=sec0,data=payload-1234 --image-opts driver=qcow2,encrypt.format=luks,encrypt.key-secret=sec0,file> ├─ 998772 /bin/qemu-nbd -c /dev/nbd73 --object secret,id=sec0,data=payload-1234 --image-opts driver=qcow2,encrypt.format=luks,encrypt.key-secret=sec0,file> ├─ 998931 /bin/qemu-nbd -c /dev/nbd100 --object secret,id=sec0,data=payload-1234 --image-opts driver=qcow2,encrypt.format=luks,encrypt.key-secret=sec0,fil> ├─ 999147 /bin/qemu-nbd -c /dev/nbd35 --object secret,id=sec0,data=payload-1234 --image-opts driver=qcow2,encrypt.format=luks,encrypt.key-secret=sec0,file> ├─1371322 /bin/qemu-nbd -c /dev/nbd63 --object secret,id=sec0,data=payload-test1 --image-opts driver=qcow2,encrypt.format=luks,encrypt.key-secret=sec0,fil> ├─1371524 /bin/qemu-nbd -c /dev/nbd91 --object secret,id=sec0,data=payload-test1 --image-opts driver=qcow2,encrypt.format=luks,encrypt.key-secret=sec0,fil> └─1489651 /openstack/venvs/nova-22.3.1/bin/python3 /usr/bin/tvault-contego --config-file=/etc/nova/nova.conf --config-file=/etc/tvault-contego/tvault-cont> Jan 14 05:45:19 compute systemd[1]: Started Tvault contego. Jan 14 05:45:20 compute sudo[1489653]: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/openstack/venvs/nova-22.3.1/bin/nova-rootwrap /etc/nova/rootwrap.conf umou> Jan 14 05:45:20 compute sudo[1489653]: pam_unix(sudo:session): session opened for user root by (uid=0) Jan 14 05:45:21 compute python3[1489655]: umount: /var/triliovault-mounts/VHJpbGlvVmF1bHQ=: no mount point specified. Jan 14 05:45:21 compute sudo[1489653]: pam_unix(sudo:session): session closed for user root Jan 14 05:45:21 compute tvault-contego[1489651]: 2022-01-14 05:45:21.499 1489651 INFO __main__ [req-48c32a39-38d0-45b9-9852-931e989133c6 - - - - -] CPU Control group m> Jan 14 05:45:21 compute tvault-contego[1489651]: 2022-01-14 05:45:21.499 1489651 INFO __main__ [req-48c32a39-38d0-45b9-9852-931e989133c6 - - - - -] I/O Control Group m> lines 1-22/22 (END) ``` ### On Kolla Ansible OpenStack #### Datamover-API service (dmapi) The dmapi service has its own Keystone endpoints, which should be checked in addition to the actual service status. ``` root@ansible:~# openstack endpoint list | grep dmapi | 190db2ce033e44f89de73abcbf12804e | US-WEST-2 | dmapi | datamover | True | public | https://osa-victoria-ubuntu20-2.triliodata.demo:8784/v2 | | dec1a323791b49f0ac7901a2dc806ee2 | US-WEST-2 | dmapi | datamover | True | admin | http://10.10.10.154:8784/v2 | | f8c4162c9c1246ffb0190d0d093c48af | US-WEST-2 | dmapi | datamover | True | internal | http://10.10.10.154:8784/v2 | root@ansible:~# curl http://10.10.10.154:8784 {"versions": [{"id": "v2.0", "status": "SUPPORTED", "version": "", "min_version": "", "updated": "2011-01-21T11:33:21Z", "links": [{"rel": "self", "href": "h ``` Run the following command on “nova-api” nodes and make sure “triliovault\_datamover\_api” container is in started state. ``` [root@controller ~]# docker ps | grep triliovault_datamover_api 3f979c15cedc trilio/centos-binary-trilio-datamover-api:4.2.50-victoria "dumb-init --single-…" 3 days ago Up 3 days triliovault_datamover_api ``` #### Datamover service (tvault-contego) Run the following command on "nova-compute" nodes and make sure the container is in a started state. ``` [root@compute1 ~]# docker ps | grep triliovault_datamover 2f1ece820a59 trilio/centos-binary-trilio-datamover:4.2.50-victoria "dumb-init --single-…" 3 days ago Up 3 days triliovault_datamover ``` #### Trilio Horizon integration Run the following command on horizon nodes and make sure the container is in a started state. ``` [root@controller ~]# docker ps | grep horizon 4a004c786d47 trilio/centos-binary-trilio-horizon-plugin:4.2.50-victoria "dumb-init --single-…" 3 days ago Up 3 days (unhealthy) horizon ``` ### On Canonical OpenStack Run the following command on MAAS nodes and make sure all trilio units like `trilio-data-mover`, `trilio-dm-api`, `trilio-horizon-plugin`, `trilio-wlm`are in active state ``` root@jujumaas:~# juju status | grep trilio trilio-data-mover 4.2.51 active 3 trilio-data-mover jujucharms 9 ubuntu trilio-dm-api 4.2.51 active 1 trilio-dm-api jujucharms 7 ubuntu trilio-horizon-plugin 4.2.51 active 1 trilio-horizon-plugin jujucharms 6 ubuntu trilio-wlm 4.2.51 active 1 trilio-wlm jujucharms 9 ubuntu trilio-data-mover/8 active idle 172.17.1.5 Unit is ready trilio-data-mover/6 active idle 172.17.1.6 Unit is ready trilio-data-mover/7* active idle 172.17.1.7 Unit is ready trilio-horizon-plugin/2* active idle 172.17.1.16 Unit is ready trilio-dm-api/2* active idle 1/lxd/4 172.17.1.27 8784/tcp Unit is ready trilio-wlm/2* active idle 7 172.17.1.28 8780/tcp Unit is ready ``` ### On Red Hat OpenStack and TripleO #### On controller node Make sure the Trilio dmapi and horizon containers (shown below) are in a running state and no other Trilio container is deployed on controller nodes. If the containers are in restarting state or not listed by the following command then your deployment is not done correctly. Please note that the 'Horizon' container is replaced with the Trilio Horizon container. This container will have the latest OpenStack horizon + Trilio's horizon plugin. ``` On rhosp13 OS: docker ps | grep trilio- On other (rhosp16 onwards/tripleo) : podman ps | grep trilio- [root@overcloudtrain1-controller-0 heat-admin]# podman ps | grep trilio- e3530d6f7bec ucqa161.ctlplane.trilio.local:8787/trilio/trilio-datamover-api:4.2.47-rhosp16.1 kolla_start 2 weeks ago Up 2 weeks ago trilio_dmapi f93f7019f934 ucqa161.ctlplane.trilio.local:8787/trilio/trilio-horizon-plugin:4.2.47-rhosp16.1 kolla_start 2 weeks ago Up 2 weeks ago horizon ``` #### On compute node Make sure the Trilio datamover container (shown below) is in a running state and no other Trilio container is deployed on compute nodes. If the containers are in restarting state or not listed by the following command then your deployment is not done correctly. ``` On rhosp13 OS: docker ps | grep trilio- On other (rhosp/tripleo) : podman ps | grep trilio- [root@overcloudtrain3-novacompute-1 heat-admin]# podman ps | grep trilio- 4419b02e075c undercloud162.ctlplane.trilio.local:8787/trilio/trilio-datamover:dev-osp16.2-1-rhosp16.2 kolla_start 2 days ago Up 27 seconds ago trilio_datamover ``` #### On overcloud Please check dmapi endpoints on overcloud node. ``` (overcloudtrain1) [stack@ucqa161 ~]$ openstack endpoint list | grep datamover | 218b2f92569a4d259839fa3ea4d6103a | regionOne | dmapi | datamover | True | internal | https://overcloudtrain1internalapi.trilio.local:8784/v2 | | 4702c51aa5c24bed853e736499e194e2 | regionOne | dmapi | datamover | True | public | https://overcloudtrain1.trilio.local:13784/v2 | | c8169025eb1e4954ab98c7abdb0f53f6 | regionOne | dmapi | datamover | True | admin | https://overcloudtrain1internalapi.trilio.local:8784/v2 ```