Shutdown/Restart the TrilioVault cluster

To gracefully shutdown/restart the TrilioVault cluster the following steps are recommended.
Verify no snapshots or restores are running
It is recommended to verify that no snapshots or restores are running on the TrilioVault Cluster.
Stopping or restarting the TrilioVault cluster will cancel all running actively running backup or restore jobs. These jobs will be marked as errored after the system has come up again.
This can be verified using the following two commands:
1
workloadmgr snapshot-list --all=True
2
workloadmgr restore-list
Copied!
Identify the master node for the VIP(s) and wlm-cron service
The TrilioVault cluster is using the pacemaker service for setting the VIP(s) of the cluster and controlling the active node for the wlm-cron service. The identified node will be the last to shut down in case that the whole cluster gets shut down.
This can be checked using the following command:
1
pcs status
Copied!
In the following example is the master node the tvm1
1
pcs status
2
Cluster name: triliovault
3
​
4
WARNINGS:
5
Corosync and pacemaker node names do not match (IPs used in setup?)
6
​
7
Stack: corosync
8
Current DC: tvm3 (version 1.1.23-1.el7_9.1-9acf116022) - partition with quorum
9
Last updated: Thu Aug 26 12:10:32 2021
10
Last change: Thu Aug 26 08:02:51 2021 by root via crm_resource on tvm1
11
​
12
3 nodes configured
13
8 resource instances configured
14
​
15
Online: [ tvm1 tvm2 tvm3 ]
16
​
17
Full list of resources:
18
​
19
 virtual_ip     (ocf::heartbeat:IPaddr2):       Started tvm1
20
 virtual_ip_public      (ocf::heartbeat:IPaddr2):       Started tvm1
21
 virtual_ip_admin       (ocf::heartbeat:IPaddr2):       Started tvm1
22
 virtual_ip_internal    (ocf::heartbeat:IPaddr2):       Started tvm1
23
 wlm-cron       (systemd:wlm-cron):     Started tvm1
24
 Clone Set: lb_nginx-clone [lb_nginx]
25
     Started: [ tvm1 ]
26
     Stopped: [ tvm2 tvm3 ]
27
​
28
Daemon Status:
29
  corosync: active/enabled
30
  pacemaker: active/enabled
31
  pcsd: active/enabled
Copied!
Shutdown/Restart of a single node in the cluster
A single node in the cluster can be shut down or restarted without issues. All services will come up and the RabbitMQ and Galeera service will rejoin the remaining cluster.
When the master node gets shutdown or restarted the VIP(s) and the wlm-cron service will switch to one of the remaining cluster nodes.
Stop the services on the node
To speed up the shutdown/restart process it is recommended to stop the TrilioVault services, the RabbitMQ service, and the MariaDB service on the node.
The wlm-cron service and the VIP(s) are not getting stopped when only the master node gets rebooted or shut down. The pacemaker will automatically move the wlm-cron service and the VIP(s) to one of the remaining nodes.
1
systemctl stop wlm-api
2
systemctl stop wlm-scheduler
3
systemctl stop wlm-workloads
4
systemctl stop mysqld
5
rabbitmqctl stop
Copied!
Shutdown/Restart the node
After the services have been stopped the node can be restarted or shut down using standard Linux commands.
1
reboot
2
shutdown
Copied!
Restarting the complete cluster node by node
Restarting the whole cluster node by node follows the same procedure as restarting a single node, with the difference that each restarted node needs to be fully started again before the next node can be restarted.
Shutdown/Restart the complete cluster as a whole
When the complete cluster needs to get stopped and restarted at the same time the following procedure needs to be completed.
The procedure on a high level is:
Shutdown the two slave nodes
Shutdown the master node
Start the master node
Enable the Galeera cluster
Start the two slave nodes
Shutdown the two slave nodes
Before shutting down the two slave nodes it is recommended to stop running TrilioVault services, the RabbitMQ server, and the MariaDB on the nodes.
1
systemctl stop wlm-api
2
systemctl stop wlm-scheduler
3
systemctl stop wlm-workloads
4
systemctl stop mysqld
5
rabbitmqctl stop
Copied!
Afterward, the nodes can be shut down.
1
shutdown
Copied!
Shutdown the master node
Before shutting down the master node it is recommended to stop running TrilioVault services, the RabbitmQ server, the MariaDB, the wlm-cron and the VIP(s) resource in Pacemaker.
1
systemctl stop wlm-api
2
systemctl stop wlm-scheduler
3
systemctl stop wlm-workloads
4
systemctl stop mysqld
5
rabbitmqctl stop
6
pcs resource disable wlm-cron
7
pcs resource disable lb_nginx-clone
Copied!
Afterward, the node can be shut down.
1
shutdown
Copied!
Start the master node
The first server that is getting booted will be the master node.
It is highly recommended that the old master node will be booted first again.
Not booting the old mater node first again can lead to data loss when the Galeera Cluster is restarted.
Enable the Galeera cluster
Login into the freshly started master node and run the following command. This will restart the Galeera cluster with this node as master.
1
galera_new_cluster
Copied!
Start the slave nodes
After the master node has been booted and the Galeera cluster started the remaining nodes can be started and will automatically rejoin the TrilioVault cluster.