LogoLogo
T4O-4.1
T4O-4.1
  • About Trilio for OpenStack
  • Trilio for OpenStack Architecture
  • Trilio 4.1 Release Notes
    • Trilio 4.1 HF1 Release Notes
    • Trilio 4.1 HF2 Release Notes
    • Trilio 4.1 HF3 Release Notes
    • Trilio 4.1 HF4 Release
    • Trilio 4.1 HF5 Release
    • Trilio 4.1 HF6 Release
    • Trilio 4.1 HF7 Release
    • Trilio 4.1 HF8 Release
    • Trilio 4.1 HF9 Release
    • Trilio 4.1 HF10 Release
    • Trilio 4.1 HF11 Release
    • Trilio 4.1 HF12 Release
    • Trilio 4.1 HF13 Release
  • Deployment Guide
    • Compatibility Matrix
    • Requirements
    • Trilio network considerations
    • Preparing the installation
    • Spinning up the Trilio VM
    • Installing Trilio Components
      • Installing on RHOSP
      • Installing on Canonical Openstack
      • Installing on Kolla Ussuri
      • Installing on Kolla Victoria
      • Installing on Ansible Openstack Ussuri
      • Installing on Ansible Openstack Victoria
      • Installing on TripleO Train
    • Configuring Trilio
    • Apply the Trilio license
    • Advanced Ceph configuration
      • Additions for multiple CEPH configurations
      • Additions for multiple Ceph users
    • Post Installation Health-Check
    • Uninstall Trilio
      • Uninstalling from RHOSP
      • Uninstalling from Canonical OpenStack
      • Uninstalling from Kolla OpenStack
      • Uninstalling from Ansible OpenStack
    • Upgrade Trilio
      • Upgrading on RHOSP
      • Upgrading on Canonical Openstack
      • Upgrading on Kolla OpenStack
      • Upgrading on Ansible OpenStack
      • Online upgrade Trilio Appliance
      • Offline upgrade Trilio Appliance
    • Install workloadmgr CLI client
    • Switch Backup Target on Kolla-ansible
    • Switch NFS Backing file
  • Trilio Appliance Administration Guide
    • Set Trilio GUI login banner
    • Trilio Appliance Dashboard
    • Set network accessibility of Trilio GUI
    • Reconfigure the Trilio Cluster
    • Change the Trilio GUI password
    • Reset the Trilio GUI password
    • Reinitialize Trilio
    • Download Trilio logs
    • Change Certificates used by Trilio
    • Restart Trilio Services
    • Shutdown/Restart the Trilio cluster
  • User Guide
    • Workloads
    • Snapshots
    • Restores
    • File Search
    • Snapshot Mount
    • Schedulers
    • E-Mail Notifications
  • Admin Guide
    • Backups-Admin Area
    • Workload Policies
    • Workload Quotas
    • Managing Trusts
    • Workload Import & Migration
    • Disaster Recovery
      • Example runbook for Disaster Recovery using NFS
  • Troubleshooting
    • General Troubleshooting Tips
    • Using the workloadmgr CLI tool on the Trilio Appliance
    • Healthcheck of Trilio
    • Important log files
  • API GUIDE
    • Workloads
    • Snapshots
    • Restores
    • File Search
    • Snapshot Mount
    • Schedulers
    • E-Mail Notification Settings
    • Workload Policies
    • Workload Quotas
    • Managing Trusts
    • Workload Import and Migration
Powered by GitBook
On this page
  • What is happening where
  • Trilio cluster
  • dmapi
  • datamover
  • Everything on the Backup Target is happening as user nova
  • Trilio Trustee Role
  • Openstack Quotas
  • Trilio Configurator

Was this helpful?

Export as PDF
  1. Troubleshooting

General Troubleshooting Tips

Troubleshooting inside a complex environment like Openstack can be very time-consuming. The following tipps will help to speed up the troubleshooting process to identify root causes.

What is happening where

Openstack and Trilio are divided into multiple services. Each service has a very specific purpose that is called during a backup or recovery procedure. Knowing which service is doing what helps to understand where the error is happening, allowing more focused troubleshooting.

Trilio cluster

The Trilio Cluster is the Controller of Trilio. It receives all Workload related requests from the users.

Every task of a backup or restore process is triggered and managed from here. This includes the creation of the directory structure and initial metadata files on the Backup Target.

During a backup process

During a backup process is the Trilio cluster also responsible to gather the metadata about the backed up VMs and networks from the Openstack environment. It is sending API calls towards the Openstack endpoints on the configured endpoint type to fetch this information. Once the metadata has been received does the Trilio Cluster write it as json files on the Backup Target.

The Trilio cluster is also sending the Cinder Snapshot command.

During a restore process

During restore process is the Trilio cluster reading the VM metadata from its Database and is using the metadata to create the Shell for the restore. It is sending API calls to the Openstack environment to create the necessary resources.

dmapi

The dmapi service is the connector between the Trilio cluster and the datamover running on the compute nodes.

The purpose of the dmapi service is to identify which compute node is responsible for the current backup or restore task. To do so is the dmapi service connecting to the nova api database requesting the compute hose of a provided VM.

Once the compute host has been identified is the dmapi forwarding the command from the Trilio Cluster to the datamover running on the identified compute host.

datamover

The datamover is the Trilio service running on the compute nodes.

Each datamover is responsible for the VMs running on top of its compute node. A datamover can not work with VMs running on a different compute node.

The datamover is controlling the freeze and thaw of VMs as well as the actual movement of the data.

Everything on the Backup Target is happening as user nova

Trilio is reading and writing on the Backup Target as nova:nova.

The POSIX user-id and group-id of nova:nova need to be aligned between the Trilio Cluster and all compute nodes. Otherwise backup or restores may fail with permission or file not found issues.

Alternativ ways to achieve the goal are possible, as long as all required nodes can fully write and read as nova:nova on the Backup Target.

It is recommended to verify the required permissions on the Backup Target in case of any errors during the data transfer phase or in case of any file permission errors.

Trilio Trustee Role

Trilio is using RBAC to allow the usage of Trilio features to users.

This trustee role is absolutely required and can not be overwritten using the admin role.

It is recommended to verify the assignment of the Trilio Trustee Role in case of any permission errors from Trilio during creation of Workloads, backups or restores.

Openstack Quotas

Trilio is creating Cinder Snapshots and temporary Cinder Volumes. The Openstack Quotas need to allow that.

Every disk that is getting backed up requires one temporary Cinder Volumes.

Every Cinder Volume that is getting backup requires two Cinder Snapshots. The second Cinder Snapshot is temporary to calculate the incremental.

Trilio Configurator

Once Trilio is configured use virtual IP to access its dashboard. If service status panels on the dashboard page is not visible then access virtual IP on port 3001 (https://<TVO-VIP>:3001/) by accepting the exception and again refresh the dashboard page.

PreviousExample runbook for Disaster Recovery using NFSNextUsing the workloadmgr CLI tool on the Trilio Appliance

Last updated 1 year ago

Was this helpful?