After the installation and configuration of Trilio for Openstack did succeed the following steps can be done to verify that the Trilio installation is healthy.
Verify the Trilio Appliance
Verify the services are up
Trilio is using 4 main services on the Trilio Appliance:
wlm-api
wlm-scheduler
wlm-workloads
wlm-cron
systemctl | grep wlm
wlm-api.service loaded active running workloadmanager api service
wlm-cron.service loaded active running Cluster Controlled wlm-cron
wlm-scheduler.service loaded active running Cluster Controlled wlm-scheduler
wlm-workloads.service loaded active running workloadmanager workloads service
Those can be verified to be up and running using the systemctl status command.
Check the Trilio pacemaker and nginx cluster
The second component to check the Trilio Appliance's health is the nginx and pacemaker cluster.
Verify API connectivity of the Trilio Appliance
Checking the availability of the Trilio API on the chosen endpoints is recommended.
The following example curl command lists the available workload-types and verifies that the connection is available and working:
Please check the API guide for more commands and how to generate the X-Auth-Token.
Verify Trilio components on OpenStack
On OpenStack Ansible
Datamover-API service (dmapi)
The dmapi service has its own Keystone endpoints, which should be checked in addition to the actual service status.
In order to check the dmapi service go to dmapi container which is residing on controller nodes and run below command
Datamover service (tvault-contego)
The datamover service is running on each compute node. Logging to compute node and run below command
On Kolla Ansible OpenStack
Datamover-API service (dmapi)
The dmapi service has its own Keystone endpoints, which should be checked in addition to the actual service status.
Run the following command on “nova-api” nodes and make sure “triliovault_datamover_api” container is in started state.
Datamover service (tvault-contego)
Run the following command on "nova-compute" nodes and make sure the container is in a started state.
Trilio Horizon integration
Run the following command on horizon nodes and make sure the container is in a started state.
On Canonical OpenStack
Run the following command on MAAS nodes and make sure all trilio units like trilio-data-mover, trilio-dm-api, trilio-horizon-plugin, trilio-wlmare in active state
On Red Hat OpenStack and TripleO
On controller node
Make sure the Trilio dmapi and horizon containers (shown below) are in a running state and no other Trilio container is deployed on controller nodes. If the containers are in restarting state or not listed by the following command then your deployment is not done correctly. Please note that the 'Horizon' container is replaced with the Trilio Horizon container. This container will have the latest OpenStack horizon + Trilio's horizon plugin.
On compute node
Make sure the Trilio datamover container (shown below) is in a running state and no other Trilio container is deployed on compute nodes. If the containers are in restarting state or not listed by the following command then your deployment is not done correctly.
systemctl status wlm-api
######
● wlm-api.service - workloadmanager api service
Loaded: loaded (/etc/systemd/system/wlm-api.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2021-11-02 19:41:19 UTC; 2 months 21 days ago
Main PID: 4688 (workloadmgr-api)
CGroup: /system.slice/wlm-api.service
├─4688 /home/stack/myansible/bin/python3 /home/stack/myansible/bin/workloadmgr-api --config-file=/etc/workloadmgr/workloadmgr.conf
systemctl status wlm-scheduler
######
● wlm-scheduler.service - Cluster Controlled wlm-scheduler
Loaded: loaded (/etc/systemd/system/wlm-scheduler.service; disabled; vendor preset: disabled)
Drop-In: /run/systemd/system/wlm-scheduler.service.d
└─50-pacemaker.conf
Active: active (running) since Sat 2022-01-22 13:49:28 UTC; 1 day 23h ago
Main PID: 9342 (workloadmgr-sch)
CGroup: /system.slice/wlm-scheduler.service
└─9342 /home/stack/myansible/bin/python3 /home/stack/myansible/bin/workloadmgr-scheduler --config-file=/etc/workloadmgr/workloadmgr.conf
systemctl status wlm-workloads
######
● wlm-workloads.service - workloadmanager workloads service
Loaded: loaded (/etc/systemd/system/wlm-workloads.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2021-11-02 19:51:05 UTC; 2 months 21 days ago
Main PID: 606 (workloadmgr-wor)
CGroup: /system.slice/wlm-workloads.service
├─ 606 /home/stack/myansible/bin/python3 /home/stack/myansible/bin/workloadmgr-workloads --config-file=/etc/workloadmgr/workloadmgr.conf
systemctl status wlm-cron
######
● wlm-cron.service - Cluster Controlled wlm-cron
Loaded: loaded (/etc/systemd/system/wlm-cron.service; disabled; vendor preset: disabled)
Drop-In: /run/systemd/system/wlm-cron.service.d
└─50-pacemaker.conf
Active: active (running) since Sat 2022-01-22 13:49:28 UTC; 1 day 23h ago
Main PID: 9209 (workloadmgr-cro)
CGroup: /system.slice/wlm-cron.service
├─9209 /home/stack/myansible/bin/python3 /home/stack/myansible/bin/workloadmgr-cron --config-file=/etc/workloadmgr/workloadmgr.conf
pcs status
######
Cluster name: triliovault
WARNINGS:
Corosync and pacemaker node names do not match (IPs used in setup?)
Stack: corosync
Current DC: TVM1 (version 1.1.21-4.el7-f14e36fd43) - partition with quorum
Last updated: Mon Jan 24 13:42:01 2022
Last change: Tue Nov 2 19:07:04 2021 by root via crm_resource on TVM2
3 nodes configured
9 resources configured
Online: [ TVM1 TVM2 TVM3 ]
Full list of resources:
virtual_ip (ocf::heartbeat:IPaddr2): Started TVM2
virtual_ip_public (ocf::heartbeat:IPaddr2): Started TVM2
virtual_ip_admin (ocf::heartbeat:IPaddr2): Started TVM2
virtual_ip_internal (ocf::heartbeat:IPaddr2): Started TVM2
wlm-cron (systemd:wlm-cron): Started TVM2
wlm-scheduler (systemd:wlm-scheduler): Started TVM2
Clone Set: lb_nginx-clone [lb_nginx]
Started: [ TVM2 ]
Stopped: [ TVM1 TVM3 ]
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
[root@controller ~]# docker ps | grep triliovault_datamover_api
3f979c15cedc trilio/centos-binary-trilio-datamover-api:4.2.50-victoria "dumb-init --single-…" 3 days ago Up 3 days triliovault_datamover_api
[root@compute1 ~]# docker ps | grep triliovault_datamover
2f1ece820a59 trilio/centos-binary-trilio-datamover:4.2.50-victoria "dumb-init --single-…" 3 days ago Up 3 days triliovault_datamover
[root@controller ~]# docker ps | grep horizon
4a004c786d47 trilio/centos-binary-trilio-horizon-plugin:4.2.50-victoria "dumb-init --single-…" 3 days ago Up 3 days (unhealthy) horizon
root@jujumaas:~# juju status | grep trilio
trilio-data-mover 4.2.51 active 3 trilio-data-mover jujucharms 9 ubuntu
trilio-dm-api 4.2.51 active 1 trilio-dm-api jujucharms 7 ubuntu
trilio-horizon-plugin 4.2.51 active 1 trilio-horizon-plugin jujucharms 6 ubuntu
trilio-wlm 4.2.51 active 1 trilio-wlm jujucharms 9 ubuntu
trilio-data-mover/8 active idle 172.17.1.5 Unit is ready
trilio-data-mover/6 active idle 172.17.1.6 Unit is ready
trilio-data-mover/7* active idle 172.17.1.7 Unit is ready
trilio-horizon-plugin/2* active idle 172.17.1.16 Unit is ready
trilio-dm-api/2* active idle 1/lxd/4 172.17.1.27 8784/tcp Unit is ready
trilio-wlm/2* active idle 7 172.17.1.28 8780/tcp Unit is ready
On rhosp13 OS: docker ps | grep trilio-
On other (rhosp16 onwards/tripleo) : podman ps | grep trilio-
[root@overcloudtrain1-controller-0 heat-admin]# podman ps | grep trilio-
e3530d6f7bec ucqa161.ctlplane.trilio.local:8787/trilio/trilio-datamover-api:4.2.47-rhosp16.1 kolla_start 2 weeks ago Up 2 weeks ago trilio_dmapi
f93f7019f934 ucqa161.ctlplane.trilio.local:8787/trilio/trilio-horizon-plugin:4.2.47-rhosp16.1 kolla_start 2 weeks ago Up 2 weeks ago horizon
On rhosp13 OS: docker ps | grep trilio-
On other (rhosp/tripleo) : podman ps | grep trilio-
[root@overcloudtrain3-novacompute-1 heat-admin]# podman ps | grep trilio-
4419b02e075c undercloud162.ctlplane.trilio.local:8787/trilio/trilio-datamover:dev-osp16.2-1-rhosp16.2 kolla_start 2 days ago Up 27 seconds ago trilio_datamover
