6.0.0-beta

What's New?

1. Deployment of Trilio on RHOCP (with RHOSP17.1).

2. Support for multiple target backends.

3. Support for advanced scheduling of workloads' snapshots.

Known Issues

1. Workload import from non-default BTT gets stuck

   • If non-default BTT is provided as source btt in import command, then the same gets stuck and no workload gets imported. However same works well if source btt is the default one.

2. workload-create CLI expects jobschedule as mandatory parameter

   • Workaround:

     • While creating workload from CLI, parameter jobschedule to be provided with True/False values. Eg. --jobschedule enabled=False

3. get-importworkload-list and get-orphaned-workloads-list show incorrect workloads

   • The commands get-importworkload-list and get-orphaned-workloads-list show incorrect workloads in the output.

   • Workaround:

     • Use --project option with get-importworkload-list CLI command to get the list of workloads that can be imported on any particular project of the OpenStack. workloadmgr workload-get-importworkloads-list --project_id <project_id>

The Red Hat OpenStack Platform Director is the supported and recommended method to deploy and maintain any RHOSP installation.

Trilio is integrating natively into the RHOSP Director. Manual deployment methods are not supported for RHOSP.

1. Prepare for deployment

Refer to the below-mentioned acceptable values for the placeholders triliovault_tag, trilio_branch, RHOSP_version and CONTAINER-TAG-VERSION in this document as per the Openstack environment:

1.1] Select 'backup target' type

Backup target storage is used to store backup images taken by Trilio and details needed for configuration:

The following backup target types are supported by Trilio

a) NFS

Need NFS share path

b) Amazon S3

- S3 Access Key - Secret Key - Region - Bucket name

c) Other S3 compatible storage (Like Ceph based S3)

- S3 Access Key - Secret Key - Region - Endpoint URL (Valid for S3 other than Amazon S3) - Bucket name

1.2] Clone triliovault-cfg-scripts repository

The following steps are to be done on the undercloud node on an already installed RHOSP environment. The overcloud-deploy command has to be run successfully already and the overcloud should be available.

The following command clones the triliovault-cfg-scripts github repository.

cd /home/stack
source stackrc
git clone -b {{ trilio_branch }} https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>

1.3] Set executable permissions for all shell scripts

cd triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/scripts/
chmod +x *.sh

1.4] If the backup target type is 'Ceph based S3' with SSL:

If your backup target is CEPH S3 with SSL and SSL certificates are self-signed or authorized by a private CA, then the user needs to provide a CA chain certificate to validate the SSL requests. For all s3 backup target with self signed TLS certificates, user need to copy ca chain files in following location and in given file name format in trilio puppet module. Edit <S3_BACKUP_TARGET_NAME>, <S3_SELF_SIGNED_CERT_CA_CHAIN_FILE> parameters in following command.

 cp <S3_SELF_SIGNED_CERT_CA_CHAIN_FILE> /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp17/puppet/trilio/files/s3-cert-<S3_BACKUP_TARGET_NAME>.pem

For example if S3_BACKUP_TARGET_NAME = BT2_S3 and your S3_SELF_SIGNED_CERT_CA_CHAIN_FILE='s3-ca.pem' then command to copy this ca chain file to trilio puppet module would be

cp s3-ca.pem /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp17/puppet/trilio/files/s3-cert-BT2_S3.pem

1.5] If the VM migration feature needs to be enabled:

Please refer to this page to collect the necessary artifacts before continuing further.

1.5.1] Copy the VMware vSphere Virtual Disk Development Kit 7.0

Rename the file as vddk.tar.gz and place it at

/home/stack/triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/puppet/trilio/files/vddk.tar.gz

1.5.2] For secure connection with the vCenter

Copy the vCenter SSL cert file to

/home/stack/triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/puppet/trilio/files/

2] Update the overcloud roles data file to include Trilio services

Trilio contains multiple services. Add these services to your roles_data.yaml.

Add the following services to the roles_data.yaml

2.1] Add Trilio Datamover Api and Trilio Workload Manager services to role data file

This service needs to share the same role as the keystone and database service. In the case of the pre-defined roles, these services will run on the role Controller. In the case of custom-defined roles, it is necessary to use the same role where OS::TripleO::Services::Keystone service installed.

Add the following line to the identified role:

- OS::TripleO::Services::TrilioDatamoverApi
- OS::TripleO::Services::TrilioWlmApi
- OS::TripleO::Services::TrilioWlmWorkloads
- OS::TripleO::Services::TrilioWlmScheduler
- OS::TripleO::Services::TrilioWlmCron
- OS::TripleO::Services::TrilioObjectStore

2.2] Add Trilio Datamover Service to role data file

This service needs to share the same role as the nova-compute service. In the case of the pre-defined roles will the nova-compute service run on the role Compute. In the case of custom-defined roles, it is necessary to use the role that the nova-compute service uses.

Add the following line to the identified role:

- OS::TripleO::Services::TrilioDatamover
- OS::TripleO::Services::TrilioObjectStore

3] Prepare Trilio container images

Trilio containers are pushed to the RedHat Container Registry. Registry URL: 'registry.connect.redhat.com'. Container pull URLs are given below.

Trilio Datamover Container:       registry.connect.redhat.com/trilio/trilio-datamover:<CONTAINER-TAG-VERSION>-rhosp17.1
Trilio Datamover Api Container:   registry.connect.redhat.com/trilio/trilio-datamover-api:<CONTAINER-TAG-VERSION>-rhosp17.1
Trilio Horizon Plugin:            registry.connect.redhat.com/trilio/trilio-horizon-plugin:<CONTAINER-TAG-VERSION>-rhosp17.1
Trilio WLM Container:             registry.connect.redhat.com/trilio/trilio-wlm:<CONTAINER-TAG-VERSION>-rhosp17.1

There are three registry methods available in the RedHat OpenStack Platform.

1. Remote Registry

2. Local Registry

3. Satellite Server

3.1] Remote Registry

Follow this section when 'Remote Registry' is used.

In this method, container images get downloaded directly on overcloud nodes during overcloud deploy/update command execution. Users can set the remote registry to a redhat registry or any other private registry that they want to use. The user needs to provide credentials for the registry in containers-prepare-parameter.yaml file.

1. Make sure other OpenStack service images are also using the same method to pull container images. If it's not the case you can not use this method.

2. Populate containers-prepare-parameter.yaml with content like the following. Important parameters are push_destination: false, ContainerImageRegistryLogin: true and registry credentials. Trilio container images are published to registry registry.connect.redhat.com. Credentials of registry 'registry.redhat.io' will work for registry.connect.redhat.com registry too.

Note: This file containers-prepare-parameter.yaml

parameter_defaults:
  ContainerImagePrepare:
  - push_destination: false
    set:
      namespace: registry.redhat.io/...
      ...
  ...
  ContainerImageRegistryCredentials:
    registry.redhat.io:
      myuser: 'p@55w0rd!'
    registry.connect.redhat.com:
      myuser: 'p@55w0rd!'
  ContainerImageRegistryLogin: true

Redhat document for remote registry method: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/director_installation_and_usage/preparing-for-director-installation#container-image-preparation-parameters

Note: File 'containers-prepare-parameter.yaml' gets created as output of command 'openstack tripleo container image prepare'. Refer above document by RedHat

3. Make sure you have network connectivity to the above registries from all overcloud nodes. Otherwise image pull operation will fail.

4. The user needs to manually populate the trilio_env.yaml file with Trilio container image URLs as given below:

$ grep '<CONTAINER-TAG-VERSION>-rhosp17.1' trilio_env.yaml
   ContainerTriliovaultDatamoverImage: undercloudqa162.ctlplane.trilio.local:8787/trilio/trilio-datamover:<CONTAINER-TAG-VERSION>-rhosp17.1
   ContainerTriliovaultDatamoverApiImage: undercloudqa162.ctlplane.trilio.local:8787/trilio/trilio-datamover-api:<CONTAINER-TAG-VERSION>-rhosp17.1
   ContainerTriliovaultWlmImage: undercloudqa162.ctlplane.trilio.local:8787/trilio/trilio-wlm:<CONTAINER-TAG-VERSION>-rhosp17.1
   ContainerHorizonImage: undercloudqa162.ctlplane.trilio.local:8787/trilio/trilio-horizon-plugin:<CONTAINER-TAG-VERSION>-rhosp17.1

At this step, you have configured Trilio image URLs in the necessary environment file.

3.2] Local Registry

Follow this section when 'local registry' is used on the undercloud.

In this case, it is necessary to push the Trilio containers to the undercloud registry. Trilio provides shell scripts that will pull the containers from registry.connect.redhat.com and push them to the undercloud and update the trilio_env.yaml.

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp17/scripts/

sudo ./prepare_trilio_images.sh <UNDERCLOUD_REGISTRY_HOSTNAME> <CONTAINER-TAG-VERSION>-rhosp17.1

## Example of running the script with parameters
sudo ./prepare_trilio_images.sh undercloudqa17.ctlplane.trilio.local 6.0.0-rhosp17.1


## Verify changes
grep '<CONTAINER-TAG-VERSION>-rhosp17.1' ../environments/trilio_env.yaml
   ContainerTriliovaultDatamoverImage: undercloudqa17.ctlplane.trilio.local:8787/trilio/trilio-datamover:<CONTAINER-TAG-VERSION>-rhosp17.1
   ContainerTriliovaultDatamoverApiImage: undercloudqa17.ctlplane.trilio.local:8787/trilio/trilio-datamover-api:<CONTAINER-TAG-VERSION>-rhosp17.1
   ContainerTriliovaultWlmImage: undercloudqa17.ctlplane.trilio.local:8787/trilio/trilio-wlm:<CONTAINER-TAG-VERSION>-rhosp17.1
   ContainerHorizonImage: undercloudqa17.ctlplane.trilio.local:8787/trilio/trilio-horizon-plugin:<CONTAINER-TAG-VERSION>-rhosp17.1

$  openstack tripleo container image list | grep <CONTAINER-TAG-VERSION>-rhosp17.1
| docker://undercloudqa17.ctlplane.trilio.local:8787/trilio/trilio-datamover-api:<CONTAINER-TAG-VERSION>-rhosp17.1                |
| docker://undercloudqa17.ctlplane.trilio.local:8787/trilio/trilio-horizon-plugin:<CONTAINER-TAG-VERSION>-rhosp17.1               |
| docker://undercloudqa17.ctlplane.trilio.local:8787/trilio/trilio-datamover:<CONTAINER-TAG-VERSION>-rhosp17.1                    |
| docker://undercloudqa17.ctlplane.trilio.local:8787/trilio/trilio-wlm:<CONTAINER-TAG-VERSION>-rhosp17.1                          |

At this step, you have downloaded Trilio container images and configured Trilio image URLs in the necessary environment file.

3.3] Red Hat Satellite Server

Follow this section when a Satellite Server is used for the container registry.

Pull the Trilio containers on the Red Hat Satellite using the given Red Hat registry URLs.

Populate the trilio_env.yaml with container URLs.

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp17/environments

$ grep '<CONTAINER-TAG-VERSION>-rhosp17.1' trilio_env.yaml
   ContainerTriliovaultDatamoverImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover:<CONTAINER-TAG-VERSION>-rhosp17.1
   ContainerTriliovaultDatamoverApiImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover-api:<CONTAINER-TAG-VERSION>-rhosp17.1
   ContainerTriliovaultWlmImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-wlm:<CONTAINER-TAG-VERSION>-rhosp17.1
   ContainerHorizonImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-horizon-plugin:<CONTAINER-TAG-VERSION>-rhosp17.1

At this step, you have downloaded Trilio container images into the RedHat satellite server and configured Trilio image URLs in the necessary environment file.

4] Provide environment details in trilio-env.yaml

Edit /home/stack/triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_env.yaml file and provide backup target details and other necessary details in the provided environment file. This environment file will be used in the overcloud deployment to configure Trilio components. Container image names have already been populated in the preparation of the container images. Still, it is recommended to verify the container URLs.

You don't need to provide anything for resource_registry, keep it as it is.

4.1] TrilioBackupTargets Details

T4O supports setting up multiple target backend for storing snapshots. User can define any number of storage backends as required. At a high level, NFS and S3 are supported.

Following table provides the details of parameters to be set in trilio_env.yaml file against all the S3 target backends.

Following table provides the details of parameters to be set in trilio_env.yaml file against all the NFS target backends.

4.2] Update triliovault object store yaml

After you fill in details of backup targets in trilio_env.yaml, user needs to run following script from ‘scripts' directory on undercloud node. This script will update ‘services/triliovault-object-store.yaml' file. User don’t need to verify that.

cd redhat-director-scripts/rhosp17/scripts/
dnf install python3-ruamel-yaml
python3 update_object_store_yaml.py

4.3] For enabling the VM migration feature, populate the below mentioned parameters

5] Generate random passwords for TrilioVault OpenStack resources

The user needs to generate random passwords for Trilio resources using the following script.

This script will generate random passwords for all Trilio resources that are going to get created in OpenStack cloud.

5.1] Change the directory and run the script

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/scripts/
./generate_passwords.sh

5.2] Output will be written to the below file.

-e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/passwords.yaml

6] Fetch ids of required OpenStack resources

6.1] User needs to source 'overcloudrc' file of cloud admin user. This is needed to run OpenStack CLI.

source <OVERCLOUD_RC_FILE>

6.2] User must have filled in the cloud admin user details of overcloud in 'trilio_env.yaml' file in the 'Provide environment details in trilio-env.yaml' section. If not please do so.

vi /home/stack/triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_env.yaml

6.3] Cloud admin user should have admin role on cloud admin domain

openstack role add --user <cloud-Admin-UserName> --domain <Cloud-Admin-DomainName> admin

# Example
openstack role add --user admin --domain default admin

6.4] After this, user can run the following script.

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/scripts
./create_wlm_ids_conf.sh

The output will be written to

cat /home/stack/triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/puppet/trilio/files/triliovault_wlm_ids.conf

7] Load necessary Linux drivers on all Controller and Compute nodes

For TrilioVault functionality to work, we need the following Linux kernel modules to be loaded on all controllers and compute nodes(Where Trilio WLM and Datamover services are going to be installed).

7.1] Install nbd module

modprobe nbd nbds_max=128
lsmod | grep nbd

7.2] Install fuse module

modprobe fuse
lsmod | grep fuse

8] Upload Trilio puppet module

8.1] Source the stackrc

source stackrc

8.2] The following commands upload the Trilio puppet module to the overcloud registry. The actual upload happens upon the next deployment.

cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp17/scripts/
./upload_puppet_module.sh

## Output of above command looks like following
Creating tarball...
Tarball created.
renamed '/tmp/puppet-modules-MUIyvXI/puppet-modules.tar.gz' -> '/var/lib/tripleo/artifacts/overcloud-artifacts/puppet-modules.tar.gz'
Creating heat environment file: /home/stack/.tripleo/environments/puppet-modules-url.yaml
[stack@uc17-1 scripts]$ cat /home/stack/.tripleo/environments/puppet-modules-url.yaml
parameter_defaults:
  DeployArtifactFILEs:
  - /var/lib/tripleo/artifacts/overcloud-artifacts/puppet-modules.tar.gz

## Above command creates following file.
ls -ll /home/stack/.tripleo/environments/puppet-modules-url.yaml

9] Deploy overcloud with Trilio environment

9.1] Include the environment file defaults.yaml in overcloud deploy command with `-e` option as shown below.

This YAML file holds the default values, like default Trustee Role is creator and Keystone endpoint interface is Internal. There are some other parameters as well those User can update as per their requirements.

-e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/defaults.yaml 

9.2] Additionally include the following heat environment files and roles data file we mentioned in the above sections in an overcloud deploy command:

1. trilio_env.yaml

2. roles_data.yaml

3. passwords.yaml

4. defaults.yaml

5. Use the correct Trilio endpoint map file as per the available Keystone endpoint configuration. You have to remove your OpenStack's endpoint map file from overcloud deploy command and instead of that use Trilio endpoint map file.

   1. Instead of tls-endpoints-public-dns.yaml file, use triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_env_tls_endpoints_public_dns.yaml

   2. Instead of tls-endpoints-public-ip.yaml file, usetriliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_env_tls_endpoints_public_ip.yaml

   3. Instead of tls-everywhere-endpoints-dns.yaml file, usetriliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_env_tls_everywhere_dns.yaml

   4. Instead of no-tls-endpoints-public-ip.yaml file, usetriliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_env_non_tls_endpoints_ip.yaml

To include new environment files use -e option and for roles data files use -r option.\

Below is an example of an overcloud deploy command with Trilio environment:

openstack overcloud deploy --stack overcloudtrain5 --templates \
  --libvirt-type qemu \
  --ntp-server 192.168.1.34 \
  -e /home/stack/templates/node-info.yaml \
  -e /home/stack/containers-prepare-parameter.yaml \
  -e /home/stack/templates/ceph-config.yaml \
  -e /home/stack/templates/cinder_size.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/services/barbican.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/barbican-backend-simple-crypto.yaml \
  -e /home/stack/templates/configure-barbican.yaml \
  -e /home/stack/templates/multidomain_horizon.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-internal-tls.yaml \
  -e /home/stack/templates/tls-parameters.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/services/haproxy-public-tls-certmonger.yaml \
  -e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp17/environments/trilio_env.yaml \
  -e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp17/environments/trilio_env_tls_everywhere_dns.yaml \
  -e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp17/environments/defaults.yaml \
  -e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp17/environments/passwords.yaml \
  -r /usr/share/openstack-tripleo-heat-templates/roles_data.yaml

10] Verify deployment

Please follow this documentation to verify the deployment.

11] Troubleshooting for overcloud deployment failures

Trilio components will be deployed using puppet scripts.

In case of the overcloud deployment failing do the following command provide the list of errors. The following document also provides valuable insights: https://docs.openstack.org/tripleo-docs/latest/install/troubleshooting/troubleshooting-overcloud.html

openstack stack failures list overcloud
heat stack-list --show-nested -f "status=FAILED"
heat resource-list --nested-depth 5 overcloud | grep FAILED

=> If any Trilio containers do not start well or are in a restarting state on the Controller/Compute node, use the following logs to debug.

podman logs <trilio-container-name>

tailf /var/log/containers/<trilio-container-name>/<trilio-container-name>.log

12] Advanced Settings/Configuration

12.1] Configure Multi-IP NFS

i] On Undercloud node, change the directory

cd triliovault-cfg-scripts/common/

ii] Edit file triliovault_nfs_map_input.yml in the current directory and provide compute host and NFS share/IP map.

Get the overcloud Controller and Compute hostnames from the following command. Check Name column. Use exact host names in the triliovault_nfs_map_input.yml file.

(undercloud) [stack@ucqa161 ~]$ openstack server list
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+
| ID                                   | Name                          | Status | Networks             | Image          | Flavor  |
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+
| 8c3d04ae-fcdd-431c-afa6-9a50f3cb2c0d | overcloudtrain1-controller-2  | ACTIVE | ctlplane=172.30.5.18 | overcloud-full | control |
| 103dfd3e-d073-4123-9223-b8cf8c7398fe | overcloudtrain1-controller-0  | ACTIVE | ctlplane=172.30.5.11 | overcloud-full | control |
| a3541849-2e9b-4aa0-9fa9-91e7d24f0149 | overcloudtrain1-controller-1  | ACTIVE | ctlplane=172.30.5.25 | overcloud-full | control |
| 74a9f530-0c7b-49c4-9a1f-87e7eeda91c0 | overcloudtrain1-novacompute-0 | ACTIVE | ctlplane=172.30.5.30 | overcloud-full | compute |
| c1664ac3-7d9c-4a36-b375-0e4ee19e93e4 | overcloudtrain1-novacompute-1 | ACTIVE | ctlplane=172.30.5.15 | overcloud-full | compute |
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+

Edit the input map file triliovault_nfs_map_input.yml and fill in all the details. Refer to this page for details about the structure.

Below is an example of how you can set the multi-IP NFS details:

$ cat /home/stack/triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_nfs_map.yaml
# TriliovaultMultiIPNfsMap represents datamover, WLM nodes (compute and controller nodes) and it's NFS share mapping.
parameter_defaults:
  TriliovaultMultiIPNfsMap:
    overcloudtrain4-controller-0: 172.30.1.11:/rhospnfs
    overcloudtrain4-controller-1: 172.30.1.11:/rhospnfs
    overcloudtrain4-controller-2: 172.30.1.11:/rhospnfs
    overcloudtrain4-novacompute-0: 172.30.1.12:/rhospnfs
    overcloudtrain4-novacompute-1: 172.30.1.13:/rhospnfs

iii] Update pyyaml on the undercloud node only

sudo pip3 install PyYAML==5.1 3

Expand the map file to create a one-to-one mapping of the compute nodes and the NFS shares.

python3 ./generate_nfs_map.py

iv] Validate output map file

The result will be stored in the triliovault_nfs_map_output.yml file

Open file triliovault_nfs_map_output.yml available in the current directory and validate that all compute nodes are covered with all the necessary NFS shares.

v] Append this output map file to triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_nfs_map.yaml

grep ':.*:' triliovault_nfs_map_output.yml >> ../redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_nfs_map.yaml

Validate the changes in the file triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_nfs_map.yaml

vi] Include the environment file (trilio_nfs_map.yaml) in overcloud deploy command with '-e' option as shown below.

-e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_nfs_map.yaml

vii] Ensure that MultiIPNfsEnabled is set to true in the trilio_env.yaml file and that NFS is used as a backup target.

/home/stack/triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_env.yaml

viii] After this you need to run the overcloud deployment.

12.2] Haproxy customized configuration for Trilio dmapi service

Following is the HAproxy conf file location on HAproxy nodes of the overcloud. Trilio Datamover API service HAproxy configuration gets added to this file.

/var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg

Trilio Datamover HAproxy default configuration from the above file looks as follows:

listen triliovault_datamover_api
  bind 172.30.4.53:13784 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 172.30.4.53:8784 transparent ssl crt /etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem
  balance roundrobin
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  http-request set-header X-Forwarded-Port %[dst_port]
  maxconn 50000
  option httpchk
  option httplog
  option forwardfor
  retries 5
  timeout check 10m
  timeout client 10m
  timeout connect 10m
  timeout http-request 10m
  timeout queue 10m
  timeout server 10m
  server overcloudtraindev2-controller-0.internalapi.trilio.local 172.30.4.57:8784 check fall 5 inter 2000 rise 2 verifyhost overcloudtraindev2-controller-0.internalapi.trilio.local

The user can change the following configuration parameter values.

retries 5
timeout http-request 10m
timeout queue 10m
timeout connect 10m
timeout client 10m
timeout server 10m
timeout check 10m
balance roundrobin
maxconn 50000

To change these default values, you need to do the following steps. i) On the undercloud node, open the following file for editing.

/home/stack/triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/services/triliovault-datamover-api.yaml

ii) Search the following entries and edit as required

          tripleo::haproxy::trilio_datamover_api::options:
             'retries': '5'
             'maxconn': '50000'
             'balance': 'roundrobin'
             'timeout http-request': '10m'
             'timeout queue': '10m'
             'timeout connect': '10m'
             'timeout client': '10m'
             'timeout server': '10m'
             'timeout check': '10m'

iii) Save the changes and do the overcloud deployment again to reflect these changes for overcloud nodes.

12.3] Configure Custom Volume/Directory Mounts for the Trilio Datamover Service

i) If the user wants to add one or more extra volume/directory mounts to the Trilio Datamover Service container, they can use a variable named 'TrilioDatamoverOptVolumes' is available in the below file.

triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_env.yaml

ii) The variable 'TrilioDatamoverOptVolumes' accepts a list of volume/bind mounts. User needs to edit the file and add their volume mounts in the below format.

TrilioDatamoverOptVolumes:
   - <mount-dir-on-compute-host>:<mount-dir-inside-the-datamover-container>

## For example, below is the `/mnt/mount-on-host` mount directory mounted on Compute host that directory you want to mount on the `/mnt/mount-inside-container` directory inside the Datamover container

[root@overcloudtrain5-novacompute-0 heat-admin]# df -h | grep 172.
172.25.0.10:/mnt/tvault/42436   2.5T  2.3T  234G  91% /mnt/mount-on-host

## Then provide that mount in the below format

TrilioDatamoverOptVolumes:
   - /mnt/mount-on-host:/mnt/mount-inside-container

iii) Lastly you need to do overcloud deploy/update.

After successful deployment, you will see that volume/directory mount will be mounted inside the Trilio Datamover Service container.

[root@overcloudtrain5-novacompute-0 heat-admin]# podman exec -itu root triliovault_datamover bash
[root@overcloudtrain5-novacompute-0 heat-admin]# df -h | grep 172.
172.25.0.10:/mnt/tvault/42436   2.5T  2.3T  234G  91% /mnt/mount-inside-container

12.4] Advanced Ceph Configration (Optional)

We are using cinder's ceph user for interacting with Ceph cinder storage. This user name is defined using parameter - 'ceph_cinder_user'.

Trilio Data, Inc

Trilio Data, Inc. is the leader in providing backup and recovery solutions for cloud-native applications. Established in 2013, its flagship products, Trilio for OpenStack(T4O) and Trilio for Kubernetes(T4K) are used by a wide number of large corporations around the world.

About Trilio for OpenStack

T4O, by Trilio Data, is a native OpenStack service-based solution that provides policy-based comprehensive backup and recovery for OpenStack workloads. It captures point-in-time workloads (Application, OS, Compute, Network, Configurations, Data, and Metadata of an environment) as full or incremental snapshots. These snapshots can be held in a variety of storage environments including NFS, AWS S3, and other S3-compatible storages. With Trilio and its one-click recovery, organizations can improve Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO). With Trilio, IT departments are enabled to fully deploy OpenStack solutions and provide business assurance through enhanced data retention, protection, and integrity.

With the use of Trilio’s VAST (Virtual Snapshot Technology), Enterprise IT and Cloud Service Providers can now deploy backup and disaster recovery as a service to prevent data loss or data corruption through point-in-time snapshots and seamless one-click recovery. Trilio takes point-in-time backup of the entire workload consisting of compute resources, network configurations, and storage data as one unit. It also takes incremental backups that capture only the changes that were made since the last backup. Incremental snapshots save a considerable amount of storage space as the backup only includes changes since the last backup. The benefits of using VAST for backup and restore could be summarized below:

About this Documentation

This documentation serves as the end-user technical resource to accompany Trilio for OpenStack. You will learn about the architecture, installation, and vast number of operations of this product. The intended audience is anyone who wants to understand the value, operations, and nuances of protecting their cloud-native applications with Trilio for OpenStack.

The high-level process is the same for all Distributions.

1. Uninstall the Horizon Plugin or the Trilio Horizon container

2. Uninstall the datamover-api container

3. Uninstall the datamover container

4. Uninstall the workloadmgr container

Trilio seamlessly integrates with OpenStack, functioning exclusively through APIs utilizing the OpenStack Endpoints. Furthermore, Trilio establishes its own set of OpenStack endpoints. Additionally, both the Trilio appliance and compute nodes interact with the backup target, impacting the network strategy for a Trilio installation.

Existing endpoints in OpenStack

OpenStack comprises three endpoint groupings:

• Public Endpoints

Public endpoints are meant to be used by the OpenStack end-users to work with OpenStack.

• Internal Endpoints

Internal endpoints are intended to be used by the OpenStack services to communicate with each

• Admin Endpoints

Admin endpoints are meant to be used by OpenStack administrators.

Among these three endpoint categories, it's important to note that the admin endpoint occasionally hosts APIs not accessible through any other type of endpoint.

To learn more about OpenStack endpoints please visit the official OpenStack documentation.

OpenStack endpoints required by Trilio

Trilio communicates with all OpenStack services through a designated endpoint type, determined and configured during the deployment of Trilio's services.

The following network requirements can be identified this way:

• Trilio services need access to the Keystone admin endpoint on the admin endpoint network if it is available.

• Trilio services need access to all endpoints of the set endpoint type during deployment.

Recommendation: Provide access to all OpenStack Endpoint types

Trilio recommends granting comprehensive access to all OpenStack endpoints for all Trilio services, aligning with OpenStack's established standards and best practices.

Additionally, Trilio generates its own endpoints, which are integrated within the same network as other OpenStack API services.

To adhere to OpenStack's prescribed standards and best practices, it's advisable that Trilio containers operate on the same network as other OpenStack containers.

• The public endpoint to be used by OpenStack users when using Trilio CLI or API

• The internal endpoint to communicate with the OpenStack services

• The admin endpoint to use the required admin-only APIs of Keystone

Backup target access required by Trilio

The Trilio solution uses backup target storage to place the backup data securely. Trilio divides its backup data into two parts:

1. Metadata

2. Volume Disk Data

The first type of data is generated by the Trilio Workloadmgr services through communication with the OpenStack Endpoints. All metadata that is stored together with a backup is written by the Trilio Workloadmgr services to the backup target in the JSON format.

The second type of data is generated by the Trilio Datamover service running on the compute nodes. The Datamover service reads the Volume Data from the Cinder or Nova storage and transfers this data as a qcow2 image to the backup target. Each Datamover service is hereby responsible for the VMs running on its compute node.

The network requirements are therefor:

• Every Trilio Workloadmgr service containers need access to the backup target

• Every Trilio Datamover service containers need access to the backup target

Before embarking on the installation process for Trilio in your OpenStack environment, it is highly advisable to carefully consider several key elements. These considerations will not only streamline the installation procedure but also ensure the optimal setup and functionality of Trilio's solutions within your OpenStack infrastructure.

Tenant Quotas

Trilio leverages Cinder snapshots to facilitate the computation of both full and incremental backups.

When executing full backups, Trilio orchestrates the generation of Cinder snapshots for all volumes included in the backup job. These Cinder snapshots remain intact for subsequent incremental backup image calculations.

During incremental backup operations, Trilio generates fresh Cinder snapshots and computes the altered blocks between these new snapshots and the earlier retained snapshots from full or previous backups. The old snapshots are subsequently deleted, while the newly generated snapshots are preserved.

Consequently, it becomes imperative for each tenant benefiting from Trilio's backup functionality to possess adequate Cinder snapshot quotas capable of accommodating these supplementary snapshots. As a guiding principle, it is recommended to append 2 snapshots for each volume incorporated into the backup quotas for the respective tenant. Additionally, a commensurate increase in volume quotas for the tenant is advisable, as Trilio briefly materializes a volume from the snapshot to access data for backup purposes.

During the restoration process, Trilio generates supplementary instances and Cinder volumes. To facilitate seamless restore operations, tenants should maintain adequate quota levels for Nova instances and Cinder volumes. Failure to meet these quota requirements may lead to disruptions in restoration procedures.

AWS S3 Eventual Consistency

The AWS S3 object consistency model includes:

1. Read-after-write

2. Read-after-update

3. Read-after-delete

Each of these models explains how an object becomes consistent after being created, updated, or deleted. None of these methods ensures strong consistency, leading to a delay before an object becomes fully consistent.

Although Trilio has introduced measures to address AWS S3's eventual consistency limitations, the exact time an object achieves consistency cannot be predicted through deterministic means.

There is no official statement from AWS on how long it takes for an object to reach a consistent state. However, read-after-write has a shorter time to reach consistency compared to other IO patterns. Therefore, our solution is designed to maximize the read-after-write IO pattern.

AWS Region Considerations

The time in which an object reaches eventual consistency also depends on the AWS region.

For instance, the AWS-standard region doesn't offer the same level of strong consistency as regions like us-east or us-west. Opting for these regions when setting up S3 buckets for Trilio is advisable. While fully avoiding the read-after-update IO pattern is complex, we've introduced significant access delays for objects to achieve consistency over longer periods. On rare occasions when this does happen, it will cause a backup failure and require a retry.

Trilio Cluster

Trilio can be deployed as a single node or a three-node cluster. It is highly recommended that Trilio is deployed as a three-node cluster for fault tolerance and load balancing. Starting with the 3.0 release, Trilio requires additional IP or FQDN for the cluster and is required for both single-node and three-node deployments. Cluster IP a.k.a virtual IP is used for managing clusters and is used to register the Trilio service endpoint in the keystone service catalog.

After all Trilio components are installed, the license can be applied.

The license can be applied either through the Admin tab in Horizon or the CLI

Apply license through Horizon

To apply for the license through Horizon follow these steps:

1. Login to Horizon using admin user.

2. Click on the Admin Tab.

3. Navigate to Backups-Admin

4. Navigate to Trilio

5. Navigate to License

6. Click "Update License"

7. Read the license agreement

8. Click on "I accept the terms in the License Agreement"

9. Click on "Next"

10. Click "Choose File"

11. Choose license file on the client system

12. Click "Apply"

Apply license through CLI

Read and accept the End User License Agreement to complete the license application.

Trilio for OpenStack Compatibility Matrix

NFS & S3 Support:

All versions of T4O-6.x releases support NFSv3 and S3 as backup targets on all the compatible distributions.

Encryption Support:

All versions of T4O-6.x releases support encryption using Barbican service on all the compatible distributions.

QEMU Guest Agent

QEMU guest agent is highly recommended to be running inside the VM that being backed up to avoid data corruption during backup process. QEMU Guest Agent is a daemon that runs inside a virtual machine (VM) and communicates with the host system (the hypervisor) to provide enhanced management and control of the VM. It is an essential component in virtualized environments specially OpenStack.

Backup-as-a-Service

Trilio is an add-on service to OpenStack cloud infrastructure and provides backup and disaster recovery solutions for tenant workloads. Trilio is very similar to other OpenStack services including Nova, Cinder, Glance, etc., and adheres to all tenets of OpenStack. It is a stateless service that scales with your cloud.

Main Components

Trilio has three main software components which are again segregated into multiple services.

WorkloadManager

This component is registered as a keystone service of type workloads which manages all the workloads being created for utilizing the snapshot and restore functionalities. It has 4 services responsible for managing these workloads, their snapshots, and restores.

1. workloadmgr-api

2. workloadmgr-scheduler

3. workloadmgr-workloads

4. workloadmgr-cron

DataMover

Similar to WorkloadManager, this component registers a keystone service of the type datamover which manages the transfer of extensive data to/from the backup targets. It has 2 services that are responsible for taking care of the data transfer and communication with WorkloadManager

1. datamover-api

2. datamover

Horizon Dashboard Plugin

For ease of access and better user experience, T4O provides an integrated UI with the OpenStack dashboard service Horizon,

1. Trilio API is a Python module that is installed on all OpenStack controller nodes where the nova-api service is running.

2. Trilio Datamover is a Python module that is installed on every OpenStack compute nodes

3. Trilio Horizon plugin is installed as an add-on to Horizon servers. This module is installed on every server that runs Horizon service.

Service Endpoints

Trilio is both a provider and consumer in the OpenStack ecosystem. It uses other OpenStack services such as Nova, Cinder, Glance, Neutron, and Keystone and provides its own services to OpenStack tenants. To accommodate all possible OpenStack deployments, Trilio can be configured to use either public or internal URLs of services. Likewise, Trilio provides its own public, internal, and admin URLs for two of its services WorkloadManager API and Datamover API.

Network Topology

Unlike the previous versions of Trilio for OpenStack, now it utilizes the existing network of the OpenStack deployed environment. The networks for the Trilio services can be configured as per the user's desire in the same way the user configures any other OpenStack service. Additionally, a dedicated network can be provided to Trilio services on both control and compute planes for storing and retrieving backup data from the backup target store.

Cloud-Native and Containerized

T4O represents a comprehensively containerized deployment model, eliminating the necessity for any KVM-based appliances for service deployment. This marks a departure from earlier T4O releases, where such appliances were required.

Summary of Requirements

Trilio requires its containers to be deployed on the same plane as OpenStack, utilizing existing cluster resources.

As described in the architecture overview, Trilio requires sufficient cluster resources to deploy its components on both the Controller Plane and Compute Planes.

1. Valid Trilio License & Acceptance of the EULA

2. Sufficient resources available on the target OpenShift cluster nodes

3. Sufficient storage capacity and connectivity on Cinder for snapshotting operations

4. Sufficient network capabilities for efficient data transfer of workloads

5. User and Role permissions for access to required cluster objects

6. Optional features may have specific requirements such as encryption, file search, snapshot mount, FRM instance, etc

7. Set hw_qemu_guest_agent=True property on the image and install qemu-guest-agent on the VM, in order to avoid any file system inconsistencies post restore.

8. For the VMware to OpenStack migration feature, please refer to the prerequisite and limitations pages.

Deployment Scripts

git clone -b 6.0.0-beta-2 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/redhat-director-scripts/rhosp17/

Container Information

Package Repositories & Versions

All packages are Python 3 (py3.6 - py3.10) compatible only.

https://pypi.fury.io/trilio-beta-6-0/

deb [trusted=yes] https://apt.fury.io/trilio-beta-6-0/ /

https://yum.fury.io/trilio-beta-6-0/

[trilio-fury]
name=Trilio Gemfury Private Repo
baseurl=https://yum.fury.io/trilio-beta-6-0/
enabled=1
gpgcheck=0

Each T4O release includes a set of artifacts such as version-tagged containers, package repositories, and distribution packages.

To help users quickly identify the resources associated with each release, we have added dedicated sub-pages corresponding to a specific release version.

After the installation and configuration of Trilio for Openstack did succeed the following steps can be done to verify that the Trilio installation is healthy.

On the Controller node

Make sure below containers are in a running state, triliovault-wlm-cron would be running on only one of the controllers in case of multi-controller setup.

• triliovault_datamover_api

• triliovault_wlm_api

• triliovault_wlm_scheduler

• triliovault_wlm_workloads

• triliovault-wlm-cron

If the containers are in a restarting state or not listed by the following command then your deployment is not done correctly. Please recheck if you followed the complete documentation.

After successful deployment, triliovault-wlm-cron service would get added in pcs cluster as a cluster resource, you can verify through pcs status command

Verify the HAproxy configuration under:

On Compute node

Make sure the Trilio Datamover container is in running state and no other Trilio container is deployed on compute nodes.

Check if provided backup target is mounted well on Compute host.

On the node with Horizon service

Make sure the horizon container is in a running state. Please note that the Horizon container is replaced with Trilio's Horizon container. This container will have the latest OpenStack horizon + Trilio horizon plugin.

The is the supported and recommended method to deploy and maintain any RHOSP installation.

Trilio is integrating natively into the RHOSP Director. Manual deployment methods are not supported for RHOSP.

1. Prepare for deployment

Refer to the below-mentioned acceptable values for the placeholders triliovault_tag, trilio_branch, RHOSP_version and CONTAINER-TAG-VERSION in this document as per the Openstack environment:

1.1] Select 'backup target' type

Backup target storage is used to store backup images taken by Trilio and details needed for configuration:

The following backup target types are supported by Trilio

a) NFS

Need NFS share path

b) Amazon S3

- S3 Access Key - Secret Key - Region - Bucket name

c) Other S3 compatible storage (Like Ceph based S3)

- S3 Access Key - Secret Key - Region - Endpoint URL (Valid for S3 other than Amazon S3) - Bucket name

1.2] Clone triliovault-cfg-scripts repository

The following steps are to be done on the workstation node on an already installed RHOSP environment. The overcloud-deploy command has to be run successfully already and the overcloud should be available.

The following command clones the triliovault-cfg-scripts github repository.

1.3] Set executable permissions for all shell scripts

1.4] If the backup target type is 'Ceph based S3' with SSL:

If your backup target is CEPH S3 with SSL and SSL certificates are self-signed or authorized by a private CA, then the user needs to provide a CA chain certificate to validate the SSL requests. For all s3 backup target with self signed TLS certificates, user need to copy ca chain files in following location and in given file name format in trilio puppet module. Edit <S3_BACKUP_TARGET_NAME>, <S3_SELF_SIGNED_CERT_CA_CHAIN_FILE> parameters in following command.

For example if S3_BACKUP_TARGET_NAME = BT2_S3 and your S3_SELF_SIGNED_CERT_CA_CHAIN_FILE='s3-ca.pem' then command to copy this ca chain file to trilio puppet module would be

1.5] If the VM migration feature needs to be enabled:

1.5.1] Copy the VMware vSphere Virtual Disk Development Kit 7.0

Rename the file as vddk.tar.gz and place it at

1.5.2] For secure connection with the vCenter

Copy the vCenter SSL cert file to

2] Update the overcloud roles data file to include Trilio services

Trilio contains multiple services. Add these services to your roles_data.yaml.

You need to find roles_data.yaml file that is getting used for openstack deployment. You will find it in 'custom-templates' directory on workstation node, where cloud administrator has kept all custom heat templates. This directory name can be anything.

Please add all Trilio services to this roles_data.yaml file.

2.1] Add Trilio Datamover Api and Trilio Workload Manager services to role data file

This service needs to share the same role as the keystone and database service. In the case of the pre-defined roles, these services will run on the role Controller. In the case of custom-defined roles, it is necessary to use the same role where OS::TripleO::Services::Keystone service installed.

Add the following line to the identified role:

2.2] Add Trilio Datamover Service to role data file

This service needs to share the same role as the nova-compute service. In the case of the pre-defined roles will the nova-compute service run on the role Compute. In the case of custom-defined roles, it is necessary to use the role that the nova-compute service uses.

Add the following line to the identified role:

This change will trigger step X: "Recreate config map tripleo-tarball-config due to change in custom templates"

3] Prepare Trilio container images

Trilio containers are pushed to the RedHat Container Registry. Registry URL: 'registry.connect.redhat.com'. Container pull URLs are given below.

There are two registry methods available in the RedHat OpenStack Platform 17.1 on RHOCP.

1. Remote Registry

2. Image Registry on Red Hat Satellite Server

3.1] Remote Registry

1] Follow this section when 'Remote Registry' is used.

In this method, container images get downloaded directly on overcloud nodes during overcloud deploy/update command execution.

Add 'registry.connect.redhat.com' redhat connect registry credentials to containers-prepare-parameter.yaml env file. Please refer below example.

If you want to use TrilioVault images from dockerhub, please use following approach. Add 'docker.io' registry to containers-prepare-parameter.yaml env file. Please refer below example.

Note: File 'containers-prepare-parameter.yaml' gets created as output of command 'openstack tripleo container image prepare'. Refer above document by RedHat

2. Make sure you have network connectivity to the above registries from all overcloud nodes. Otherwise image pull operation will fail.

3. The user needs to manually populate the trilio_env.yaml file with Trilio container image URLs as given below:

RedHat Connect Registry URL: registry.connect.redhat.com Dockerhub Registry URL: docker.io

You can use any one registry from above. But make sure to use same registry as you configured in step 1 of this section

At this step, you have configured Trilio image URLs in the necessary environment file. You can skip step 3.2

3.2] Image Regisry on Red Hat Satellite Server

Follow this section when a Satellite Server is used for the container registry.

Populate the trilio_env.yaml with container URLs.

At this step, you have downloaded Trilio container images into the RedHat satellite server and configured Trilio image URLs in the necessary environment file.

4] Provide environment details in trilio-env.yaml

Edit /home/stack/triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_env.yaml file and provide backup target details and other necessary details in the provided environment file. This environment file will be used in the overcloud deployment to configure Trilio components. Container image names have already been populated in the preparation of the container images. Still, it is recommended to verify the container URLs.

You don't need to provide anything for resource_registry, keep it as it is.

4.1] Update triliovault object store yaml

After you fill in details of backup targets in trilio_env.yaml, user needs to run following script from ‘scripts' directory on undercloud node. This script will update ‘services/triliovault-object-store.yaml' file. User don’t need to verify that.

4.2] For enabling the VM migration feature, populate the below mentioned parameters in trilio_env.yaml

5] Generate random passwords for TrilioVault OpenStack resources []

The user needs to generate random passwords for Trilio resources using the following script.

This script will generate random passwords for all Trilio resources that are going to get created in OpenStack cloud.

5.1] Change the directory and run the script

5.2] Output will be written to the below file.

6] Fetch ids of required OpenStack resources [Run from OpenStackClient Pod]

Run all steps of this section from 'openstackclient' pod.

Login to openstackclient pod.

6.1] User needs to source 'overcloudrc' file of cloud admin user. This is needed to run OpenStack CLI and Clone repo

6.2] User must have filled in the cloud admin user details of overcloud in 'trilio_env.yaml' file in the 'Provide environment details in trilio-env.yaml' section. If not please do so.

6.3] Cloud admin user should have admin role on cloud admin domain

6.4] After this, user can run the following script.

The output will be written to

7] Load necessary Linux drivers on all Controller and Compute nodes

For TrilioVault functionality to work, we need the following Linux kernel modules to be loaded on all controllers and compute nodes(Where Trilio WLM and Datamover services are going to be installed).

7.1] Install nbd module

7.2] Install fuse module

8] Upload Trilio puppet module [From inside openstackclient pod]

Use ansible ad hoc commands or playbooks to copy the Trilio puppet module from openstack client pod to all overcloud nodes.

8.1] Login to openstackclient pod

Login to openstackclient pod.

8.2] Create a ansible playbook file at path '/home/cloud-admin/copy_trilio_puppet_module.yaml' and copy following content into it.

8.3] Run ansible playbook to copy Trilio puppet module to all controller and compute nodes

This step copies Trilio puppet module from path '/home/cloud-admin/triliovault-cfg-scripts/redhat-director-scripts/rhosp17/puppet/trilio' to all controller and compute nodes at path '/etc/puppet/modules/trilio'

9] Recreate config map tripleo-tarball-config due to change in custom templates [On workstation node]

In this step we will copy all TrilioVault heat service templates to custom templates folder and re-create custom-config.tar.gz This section is valid for following TrilioVault heat templates

All heat templates paths should be relative to following directory :

9.1] Identify custom_templates folder location(From where tar file custom-config.tar.gz created) and navigate to it. If you are not using any custom heat templates, you can skip this step.

9.2] If you are is not using this custom_templates tar (Using all default heat templates) then we need to create one for TrilioVault. [Skip this step if you performed previous step]

9.3] Copy endpoint_map.yaml heat template having TrilioVault endpoint data appended to openstack endpoint map data.

9.4] Create tar ball custom-config.tar.gz

9.5] Create config map named tripleo-tarball-config using the above tar ball. If the config map is already there on RHOCP, you may need to delete existing config map or use other oc command to update existing config map. Check config map name in “openstack-config-generator.yaml“, use the same name.

First we need to delete existing config map

10] Recreate config map 'heat-env-config' due to change in environment files [On workstation node]

RHOSP Reference document: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/17.1/html-single/deploying_an_overcloud_in_a_red_hat_openshift_container_platform_cluster_with_director_operator/index#proc_adding-custom-environment-files-to-the-overcloud-configuration_OSPdO-with-HCI

10.1] On Workstation node, identify custom environment files directory that overcloud deployment is using to create heat-env-config config map.

In this step user need to copy Trilio environment files to custome environment files directory. This diretcory name can be anything.

Let's say this directory path is "PATH_TO/dir_custom_environment_files"

10.2] Create the heat-env-config ConfigMap object again. You can delete recreate the existing config map or update it. Check config map name in "openstack-config-generator.yaml", use the same name

In following commands, replace <dir_custom_environment_files> with the directory that contains the environment files you want to use in your overcloud deployment. The ConfigMap object stores these as individual data entries.

11] Creating Ansible playbooks for overcloud configuration with the OpenStackConfigGenerator CRD [On Workstation node]

RHOSP Reference Document: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/17.1/html-single/deploying_an_overcloud_in_a_red_hat_openshift_container_platform_cluster_with_director_operator/index#proc_creating-ansible-playbooks-for-overcloud-configuration-with-the-openstackplaybookgenerator-CRD_OSPdO-overcloud-deploy

11.1] Delete existing OpenStackConfigGenerator CRD resource 'default' . This step does not delete overcloud.

Before running this step, please take permission from Redhat deployment team.

11.2] Create OpenStackConfigGenerator resource again. Make sure you are in directory where 'openstack-config-generator.yaml' file exists. This yaml file must have been used to deploy RHOSP cloud.

Check status of the resource, wait till it’s status chnges to “Finished“

12] Deploy overcloud with Trilio environment [From Old doc step number 9] [On Workstation node]

RHOSP Reference Document: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/17.1/html-single/deploying_an_overcloud_in_a_red_hat_openshift_container_platform_cluster_with_director_operator/index#proc_applying-overcloud-configuration-with-director-operator_OSPdO-overcloud-deploy

In this step we will apply the new ansible playbooks generated in previous step to overcloud. Use config version of the playbooks generated in previous step. Update config version in openstack-deployment.yaml file

Apply the updated definition.

Check deployment progress and logs

13] Verify deployment

• Make sure you see a successful deployment message at the bottom of the following logs You may need to adjust your deployment name deploy-openstack-default' as per your environment.

Trilio employs a base64 hash to establish the mount point for NFS Backup targets, ensuring compatibility across multiple NFS Shares within a single Trilio installation. This hash is an integral component of Trilio's incremental backups, functioning as an absolute path for backing files.

Consequently, during a disaster recovery or rapid migration situation, the utilization of a mount bind becomes necessary.

In scenarios that allow for a comprehensive migration period, an alternative approach comes into play. This involves modifying the backing file, thereby enabling the accessibility of Trilio backups from a different NFS Share. The backing file is updated to correspond with the mount point of the new NFS Share.

Backing file change script

Trilio provides a shell script for the purpose of changing the backing file. This script is used after the Trilio appliance has been reconfigured to use the new NFS share.

Downloading the shell script

Prerequisites

The following requirements need to be met before the change of the backing file can be attempted.

• The Trilio Appliance has been reconfigured with the new NFS Share

• The Openstack environment has been reconfigured with the new NFS Share

  • Please check here for Red Hat Openstack Platform

• The workloads are available on the new NFS Share

• The workloads are owned by nova:nova user

Usage

The shell script is changing one workload at a time.

Run the following command:

./backing_file_update.sh /var/triliovault-mounts/<base64>/workload_<workload_id>

with

• /var/triliovault-mounts/<base64>/ being the new NFS mount path

• workload_<workload_id> being the workload to rebase

Logging of the procedure

The shell script is generating the following log file at the following location:

/tmp/backing_file_update.log

The log file will not get overwritten when the script is run multiple times. Each run of the script will append the available log file.

The following versions can be upgraded to each other:

Introduction

Filename and location:

triliovault-cfg-scripts/common/triliovault_nfs_map_input.yml

This file exclusively comes into play when users aim to configure Trilio with a NFS backup target that employs multiple network endpoints. For all other scenarios, such as single IP NFS or S3, this file remains inactive, and in such instances, please consult the standard installation documentation.

When using an NFS backup target with multiple network endpoints, T4O will mount a single IP/endpoint on a designated compute node for a specific NFS share. This approach enables users to distribute NFS share IPs/endpoints across various compute nodes.

The 'triliovault_nfs_map_input.yml' file allows users to distribute/load balance NFS share endpoints across compute nodes in a given cloud.

Examples

Using hostnames

Here, the User has ‘one’ NFS share exposed with three IP addresses. 192.168.1.34, 192.168.1.35, 192.168.1.33 Share directory path is: /var/share1

So, this NFS share supports the following full paths that clients can mount:

192.168.1.33:/var/share1 
192.168.1.34:/var/share1 
192.168.1.35:/var/share1

There are 32 compute nodes in the OpenStack cloud. 30 node hostnames have the following naming pattern

prod-compute-1.trilio.demo 
prod-compute-2.trilio.demo 
prod-compute-3.trilio.demo 
. 
. 
. 
prod-compute-30.trilio.demo

The remaining 2 node hostnames do not follow any format/pattern.

compute_bare.trilio.demo 
compute_virtual

Now the mapping file will look like this

multi_ip_nfs_shares: 
 - "192.168.1.34:/var/share1": ['prod-compute-[1:10].trilio.demo', 'compute_bare.trilio.demo'] 
   "192.168.1.35:/var/share1": ['prod-compute-[11:20].trilio.demo', 'compute_virtual'] 
   "192.168.1.33:/var/share1": ['prod-compute-[21:30].trilio.demo'] 

single_ip_nfs_shares: []

Using IPs

Compute node IP range used here: 172.30.3.11-40 and 172.30.4.40, 172.30.4.50 Total of 32 compute nodes

multi_ip_nfs_shares: 
 - "192.168.1.34:/var/share1": ['172.30.3.[11:20]', '172.30.4.40'] 
   "192.168.1.35:/var/share1": ['172.30.3.[21:30]', '172.30.4.50'] 
   "192.168.1.33:/var/share1": ['172.30.3.[31:40]'] 

single_ip_nfs_shares: []

Getting the correct compute hostnames/IPs

RHOSP

Use the following command to get compute hostnames. Check the ‘Name' column. Use these exact hostnames in 'triliovault_nfs_map_input.yml' file.

In the following command output, ‘overcloudtrain1-novacompute-0' and ‘overcloudtrain1-novacompute-1' are correct hostnames.

(undercloud) [stack@ucqa161 ~]$ openstack server list
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+
| ID                                   | Name                          | Status | Networks             | Image          | Flavor  |
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+
| 8c3d04ae-fcdd-431c-afa6-9a50f3cb2c0d | overcloudtrain1-controller-2  | ACTIVE | ctlplane=172.30.5.18 | overcloud-full | control |
| 103dfd3e-d073-4123-9223-b8cf8c7398fe | overcloudtrain1-controller-0  | ACTIVE | ctlplane=172.30.5.11 | overcloud-full | control |
| a3541849-2e9b-4aa0-9fa9-91e7d24f0149 | overcloudtrain1-controller-1  | ACTIVE | ctlplane=172.30.5.25 | overcloud-full | control |
| 74a9f530-0c7b-49c4-9a1f-87e7eeda91c0 | overcloudtrain1-novacompute-0 | ACTIVE | ctlplane=172.30.5.30 | overcloud-full | compute |
| c1664ac3-7d9c-4a36-b375-0e4ee19e93e4 | overcloudtrain1-novacompute-1 | ACTIVE | ctlplane=172.30.5.15 | overcloud-full | compute |
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+

1] Prerequisites

Please ensure the following requirements are met before starting the upgrade process:

• No Snapshot or Restore is running

• Global job scheduler is disabled

• wlm-cron is disabled on the Trilio Appliance

Deactivating the wlm-cron service

The following sets of commands will disable the wlm-cron service and verify that it is has been completely shutdown.

[root@TVM2 ~]# pcs resource disable wlm-cron
[root@TVM2 ~]# systemctl status wlm-cron
● wlm-cron.service - workload's scheduler cron service
   Loaded: loaded (/etc/systemd/system/wlm-cron.service; disabled; vendor preset                                                                                                                                                                                                                                                                                                                                                           : disabled)
   Active: inactive (dead)

Jun 11 08:27:06 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:06 - INFO - 1...t
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 140686268624368 Child 11389 ki...5
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - 1...5
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: Shutting down thread pool
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - S...l
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: Stopping the threads
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - S...s
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: All threads are stopped succes...y
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - A...y
Jun 11 08:27:09 TVM2 systemd[1]: Stopped workload's scheduler cron service.
Hint: Some lines were ellipsized, use -l to show in full.
[root@TVM2 ~]# pcs resource show wlm-cron
 Resource: wlm-cron (class=systemd type=wlm-cron)
  Meta Attrs: target-role=Stopped
  Operations: monitor interval=30s on-fail=restart timeout=300s (wlm-cron-monito                                                                                                                                                                                                                                                                                                                                                           r-interval-30s)
              start interval=0s on-fail=restart timeout=300s (wlm-cron-start-int                                                                                                                                                                                                                                                                                                                                                           erval-0s)
              stop interval=0s timeout=300s (wlm-cron-stop-interval-0s)
[root@TVM2 ~]# ps -ef | grep -i workloadmgr-cron
root     15379 14383  0 08:27 pts/0    00:00:00 grep --color=auto -i workloadmgr                                                                                                                                                                                                                                                                                                                                                           -cron

2] Install latest Trilio release

Please follow this documentation to install the latest Trilio release.

3] Enable mount-bind for NFS

T4O has changed the calculation of the mount point from 4.2 release onwards. It is necessary to set up the mount-bind to make T4O 4.1 or older backups available for T4O 4.2

Please follow this documentation to set up the mount bind for RHOSP.

4] Enable Global Job Scheduler

After the upgrade, global job scheduler would be disabled. You can enable that either through UI or CLI.

4.1] Through UI

Login to the Dashboard and go to Admin -> Backups-Admin -> Settings tab and check the 'Job Scheduler Enabled' checkbox. By clicking the 'Change' button you can enable the Global Job Scheduler.

4.2] Through CLI

Login to the any WLM container, create and source the admin rc file. Execute the enable the global job scheduler cli.

$ podman exec -itu root triliovault_wlm_api /bin/bash
$ source admin.rc
$ workloadmgr enable-global-job-scheduler
Global job scheduler is successfully enabled

5] Import old workloads existing on the Backup Target

Workload import allows importing Workloads existing on the Backup Target into the Trilio database.

Please follow this documentation to import the Workloads.

Clean Trilio Datamover API and Workloadmanager services

The following steps need to be run on all nodes, which have the Trilio Datamover API & Workloadmanager services running. Those nodes can be identified by checking the roles_data.yaml for the role that contains the below entries

OS::TripleO::Services::TrilioDatamoverApi
OS::TripleO::Services::TrilioWlmApi
OS::TripleO::Services::TrilioWlmWorkloads
OS::TripleO::Services::TrilioWlmScheduler
OS::TripleO::Services::TrilioWlmCron

Once the role that runs the Trilio Datamover API & Workloadmanager services has been identified will the following commands clean the nodes from the service.

Remove triliovault_datamover_api container.

podman rm -f triliovault_datamover_api
podman rm -f triliovault_datamover_api_db_sync
podman rm -f triliovault_datamover_api_init_log

Clean Trilio Datamover API service conf directory.

rm -rf /var/lib/config-data/puppet-generated/triliovaultdmapi
rm /var/lib/config-data/puppet-generated/triliovaultdmapi.md5sum
rm -rf /var/lib/config-data/triliovaultdmapi*
rm -f /var/lib/config-data/triliovault_datamover_api*

Clean Trilio Datamover API service log directory.

rm -rf /var/log/containers/triliovault-datamover-api/

Remove triliovault_wlm_api container.

podman rm -f triliovault_wlm_api
podman rm -f triliovault_wlm_api_cloud_trust_init
podman rm -f triliovault_wlm_api_db_sync
podman rm -f triliovault_wlm_api_config_dynamic
podman rm -f triliovault_wlm_api_init_log

Clean Trilio Workloadmanager API service conf directory.

rm -rf /var/lib/config-data/puppet-generated/triliovaultwlmapi
rm /var/lib/config-data/puppet-generated/triliovaultwlmapi.md5sum
rm -rf /var/lib/config-data/triliovaultwlmapi*
rm -f /var/lib/config-data/triliovault_wlm_api*

Clean Trilio Workloadmanager API service log directory.

rm -rf /var/log/containers/triliovault-wlm-api/

Remove triliovault_wlm_workloads container.

podman rm -f triliovault_wlm_workloads
podman rm -f triliovault_wlm_workloads_config_dynamic
podman rm -f triliovault_wlm_workloads_init_log

Clean Trilio Workloadmanager Workloads service conf directory.

rm -rf /var/lib/config-data/puppet-generated/triliovaultwlmworkloads
rm /var/lib/config-data/puppet-generated/triliovaultwlmworkloads.md5sum
rm -rf /var/lib/config-data/triliovaultwlmworkloads*

Clean Trilio Workloadmanager Workloads service log directory.

rm -rf /var/log/containers/triliovault-wlm-api/

Remove triliovault_wlm_scheduler container.

podman rm -f triliovault_wlm_scheduler
podman rm -f triliovault_wlm_scheduler_config_dynamic
podman rm -f triliovault_wlm_scheduler_init_log

Clean Trilio Workloadmanager Scheduler service conf directory.

rm -rf /var/lib/config-data/puppet-generated/triliovaultwlmscheduler
rm /var/lib/config-data/puppet-generated/triliovaultwlmscheduler.md5sum
rm -rf /var/lib/config-data/triliovaultwlmscheduler*

Clean Trilio Workloadmanager Scheduler service log directory.

rm -rf /var/log/containers/triliovault-wlm-scheduler/

Remove triliovault-wlm-cron-podman-0 container from controller.

podman rm -f triliovault-wlm-cron-podman-0
podman rm -f triliovault_wlm_cron_config_dynamic
podman rm -f triliovault_wlm_cron_init_log

Clean Trilio Workloadmanager Cron service conf directory.

rm -rf /var/lib/config-data/puppet-generated/triliovaultwlmcron
rm /var/lib/config-data/puppet-generated/triliovaultwlmcron.md5sum
rm -rf /var/lib/config-data//triliovaultwlmcron*

Clean Trilio Workloadmanager Cron service log directory.

rm -rf /var/log/containers/triliovault-wlm-cron/

Clean Trilio Datamover Service

The following steps need to be run on all nodes, which have the Trilio Datamover service running. Those nodes can be identified by checking the roles_data.yaml for the role that contains the entry OS::TripleO::Services::TrilioDatamover.

Once the role that runs the Trilio Datamover service has been identified will the following commands clean the nodes from the service.

Remove triliovault_datamover container.

podman rm -f triliovault_datamover

Unmount the Trilio Backup Target on the compute host.

## Following steps are applicable for all supported RHOSP releases.

# Check triliovault backup target mount point
mount | grep trilio

# Unmount it
-- If it's NFS	(COPY UUID_DIR from your compute host using above command)
umount /var/lib/nova/triliovault-mounts/<UUID_DIR>

-- If it's S3
umount /var/lib/nova/triliovault-mounts

# Verify that it's unmounted		
mount | grep trilio
	
df -h  | grep trilio

# Remove mount point directory after verifying that backup target unmounted successfully.
# Otherwise actual data from backup target may get cleaned.	

rm -rf /var/lib/nova/triliovault-mounts

Clean Trilio Datamover service conf directory.

rm -rf /var/lib/config-data/puppet-generated/triliovaultdm/
rm /var/lib/config-data/puppet-generated/triliovaultdm.md5sum
rm -rf /var/lib/config-data/triliovaultdm*

Clean log directory of Trilio Datamover service.

rm -rf /var/log/containers/triliovault-datamover/

Clean wlm-cron resource from pcs cluster

Remove wlm cron resource from pcs cluster on the controller node.

pcs resource delete triliovault-wlm-cron

Clean Trilio HAproxy resources

The following steps need to be run on all nodes, which have the HAproxy service running. Those nodes can be identified by checking the roles_data.yaml for the role that contains the entry OS::TripleO::Services::HAproxy.

Once the role that runs the HAproxy service has been identified will the following commands clean the nodes from all the Trilio resources.

Edit the following file inside the HAproxy container and remove all Trilio entries.

/var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg

An example of these entries is given below.

listen triliovault_datamover_api
  bind 172.30.5.23:13784 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 172.30.5.23:8784 transparent ssl crt /etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem
  balance roundrobin
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  http-request set-header X-Forwarded-Port %[dst_port]
  maxconn 50000
  option httpchk
  option httplog
  retries 5
  timeout check 10m
  timeout client 10m
  timeout connect 10m
  timeout http-request 10m
  timeout queue 10m
  timeout server 10m
  server overcloudtrain1-controller-0.internalapi.trilio.local 172.30.5.28:8784 check fall 5 inter 2000 rise 2 verifyhost overcloudtrain1-controller-0.internalapi.trilio.local

listen triliovault_wlm_api
  bind 172.30.5.23:13781 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 172.30.5.23:8781 transparent ssl crt /etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem
  balance roundrobin
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  http-request set-header X-Forwarded-Port %[dst_port]
  maxconn 50000
  option httpchk
  option httplog
  retries 5
  timeout check 10m
  timeout client 10m
  timeout connect 10m
  timeout http-request 10m
  timeout queue 10m
  timeout server 10m
  server overcloudtrain1-controller-0.internalapi.trilio.local 172.30.5.28:8780 check fall 5 inter 2000 rise 2 verifyhost overcloudtrain1-controller-0.internalapi.trilio.local

Restart the HAproxy container once all edits have been done.

podman restart haproxy-bundle-podman-0

Clean Trilio Keystone resources

Trilio registers services and users in Keystone. Those need to be unregistered and deleted.

openstack service delete dmapi
openstack user delete dmapi
openstack service delete TrilioVaultWLM
openstack user delete triliovault

Clean Trilio database resources

Trilio creates databases for dmapi and workloadmgr services. These databases need to be cleaned.

Login into the database cluster

podman exec -it galera-bundle-podman-0 mysql -u root

Run the following SQL statements to clean the database.

## Clean database
DROP DATABASE dmapi;

## Clean dmapi user
=> List 'dmapi' user accounts
MariaDB [mysql]> select user, host from mysql.user where user='dmapi';
+-------+-----------------------------------------+
| user  | host                                    |
+-------+-----------------------------------------+
| dmapi | %                                       |
| dmapi | 172.30.5.28                             |
| dmapi | overcloudtrain1internalapi.trilio.local |
+-------+-----------------------------------------+
3 rows in set (0.000 sec)

=> Delete those user accounts
MariaDB [(none)]> DROP USER dmapi@'%';
Query OK, 0 rows affected (0.005 sec)

MariaDB [(none)]> DROP USER dmapi@172.30.5.28;
Query OK, 0 rows affected (0.006 sec)

MariaDB [(none)]> DROP USER dmapi@overcloudtrain1internalapi.trilio.local;
Query OK, 0 rows affected (0.005 sec)

=> Verify that dmapi user got cleaned
MariaDB [mysql]> select user, host from mysql.user where user='dmapi';
Empty set (0.00 sec)

## Clean database
DROP DATABASE workloadmgr;

## Clean workloadmgr user
=> List 'workloadmgr' user accounts
MariaDB [(none)]> select user, host from mysql.user where user='workloadmgr';
+-------------+-----------------------------------------+
| user        | host                                    |
+-------------+-----------------------------------------+
| workloadmgr | %                                       |
| workloadmgr | 172.30.5.28                             |
| workloadmgr | overcloudtrain1internalapi.trilio.local |
+-------------+-----------------------------------------+
3 rows in set (0.000 sec)

=> Delete those user accounts
MariaDB [(none)]> DROP USER workloadmgr@'%';
Query OK, 0 rows affected (0.012 sec)

MariaDB [(none)]> DROP USER workloadmgr@172.30.5.28;
Query OK, 0 rows affected (0.006 sec)

MariaDB [(none)]> DROP USER workloadmgr@overcloudtrain1internalapi.trilio.local;
Query OK, 0 rows affected (0.005 sec)

=> Verify that workloadmgr user got cleaned
MariaDB [(none)]> select user, host from mysql.user where user='workloadmgr';
Empty set (0.000 sec)

Revert overcloud deploy command

Remove the following entries from roles_data.yaml used in the overcloud deploy command.

• OS::TripleO::Services::TrilioDatamoverApi

• OS::TripleO::Services::TrilioWlmApi

• OS::TripleO::Services::TrilioWlmWorkloads

• OS::TripleO::Services::TrilioWlmScheduler

• OS::TripleO::Services::TrilioWlmCron

• OS::TripleO::Services::TrilioDatamover

Follow these steps to clean the overcloud deploy command from all Trilio entries.

1. Remove trilio_env.yaml entry

2. Remove trilio endpoint map file Replace with original map file if existing

Revert back to the original RHOSP Horizon container

Run the cleaned overcloud deploy command.

Ceph is the most common OpenSource solution to provide block storage through OpenStack Cinder.

Ceph is a very flexible solution. The possibilities of Ceph require additional steps to the Trilio solution.

It is possible to configure Cinder and Ceph to use different Ceph users for different Ceph pools and Cinder volume types. Or to have the nova boot volumes and cinder block volumes controlled by different users.

In the case of multiple Ceph users, it is required to delete the keyring extension from the triliovault-datamover.conf inside the Ceph block by following below mentioned steps:

1. Deploy Trilio as per the documented steps.

2. Post successful deployment, please modify the triliovault-datamover.conf file present at following locations on all compute nodes.

   For RHOSP : /var/lib/config-data/puppet-generated/triliovaultdm/etc/triliovault-datamover/

   For Kolla : /etc/kolla/triliovault-datamover/

3. Modify keyring_ext value with valid keyring extension (eg. .keyring). This extension is expected to be same for all the keyring files. It will be present under [ceph] block in triliovault-datamover.conf file.

Sample conf entry below. This will try all files with the extension keyring that are located inside /etc/ceph to access the Ceph cluster for a Trilio related task.

1. Restart triliovault_datamover container on all compute nodes.

Definition

A Snapshot is a single Trilio backup of a workload including all data and metadata. It contains the information of all VM's that are protected by the workload.

List of Snapshots

Using Horizon

1. Login to Horizon

2. Navigate to the Backups

3. Navigate to Workloads

4. Identify the workload to show the details on

5. Click the workload name to enter the Workload overview

6. Navigate to the Snapshots tab

The List of Snapshots for the chosen Workload contains the following additional information:

• Creation Time

• Name of the Snapshot

• Description of the Snapshot

• Total amount of Restores from this Snapshot

  • Total amount of succeeded Restores

  • Total amount of failed Restores

• Snapshot Type

• Snapshot Size

• Snapshot Status

Using CLI

 workloadmgr snapshot-list [--workload_id <workload_id>]
                           [--tvault_node <host>]
                           [--date_from <date_from>]
                           [--date_to <date_to>]
                           [--all {True,False}]

Creating a Snapshot

Snapshots are automatically created by the Trilio scheduler. If necessary or in case of deactivated scheduler is it possible to create a Snapshot on demand.

Using Horizon

There are 2 possibilities to create a snapshot on demand.

Possibility 1: From the Workloads overview

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that shall create a Snapshot

5. Click "Create Snapshot"

6. Provide a name and description for the Snapshot

7. Decide between Full and Incremental Snapshot

8. Click "Create"

Possibility 2: From the Workload Snapshot list

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that shall create a Snapshot

5. Click the workload name to enter the Workload overview

6. Navigate to the Snapshots tab

7. Click "Create Snapshot"

8. Provide a name and description for the Snapshot

9. Decide between Full and Incremental Snapshot

10. Click "Create"

Using CLI

workloadmgr workload-snapshot [--full] [--display-name <display-name>]
                              [--display-description <display-description>]
                              <workload_id>

Snapshot overview

Each Snapshot contains a lot of information about the backup. These information can be seen in the Snapshot overview.

Using Horizon

To reach the Snapshot Overview follow these steps:

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that contains the Snapshot to show

5. Click the workload name to enter the Workload overview

6. Navigate to the Snapshots tab

7. Identify the searched Snapshot in the Snapshot list

8. Click the Snapshot Name

Details Tab

The Snapshot Details Tab shows the most important information about the Snapshot.

• Snapshot Name / Description

• Snapshot Type

• Time Taken

• Size

• Which VMs are part of the Snapshot

• for each VM in the Snapshot

  • Instance Info - Name & Status

  • Security Group(s) - Name & Type

  • Flavor - vCPUs, Disk & RAM

  • Networks - IP, Networkname & Mac Address

  • Attached Volumes - Name, Type, size (GB), Mount Point & Restore Size

  • Misc - Original ID of the VM

Restores Tab

The Snapshot Restores Tab shows the list of Restores that have been started from the chosen Snapshot. It is possible to start Restores from here.

Misc. Tab

The Snapshot Miscellaneous Tab provides the remaining metadata information about the Snapshot.

• Creation Time

• Last Update time

• Snapshot ID

• Workload ID of the Workload containing the Snapshot

Using CLI

workloadmgr snapshot-show [--output <output>] <snapshot_id>

Delete Snapshots

Once a Snapshot is no longer needed, it can be safely deleted from a Workload.

Using Horizon

There are 2 possibilities to delete a Snapshot.

Possibility 1: Single Snapshot deletion through the submenu

To delete a single Snapshot through the submenu follow these steps:

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that contains the Snapshot to delete

5. Click the workload name to enter the Workload overview

6. Navigate to the Snapshots tab

7. Identify the searched Snapshot in the Snapshot list

8. Click the small arrow in the line of the Snapshot next to "One Click Restore" to open the submenu

9. Click "Delete Snapshot"

10. Confirm by clicking "Delete"

Possibility 2: Multiple Snapshot deletion through checkbox in Snapshot overview

To delete one or more Snapshots through the Snapshot overview do the following:

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that contains the Snapshot to show

5. Click the workload name to enter the Workload overview

6. Navigate to the Snapshots tab

7. Identify the searched Snapshots in the Snapshot list

8. Check the checkbox for each Snapshot that shall be deleted

9. Click "Delete Snapshots"

10. Confirm by clicking "Delete"

Using CLI

workloadmgr snapshot-delete <snapshot_id>

Snapshot Cancel

Ongoing Snapshots can be canceled.

Using Horizon

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that contains the Snapshot to cancel

5. Click the workload name to enter the Workload overview

6. Navigate to the Snapshots tab

7. Identify the searched Snapshot in the Snapshot list

8. Click "Cancel" on the same line as the identified Snapshot

9. Confirm by clicking "Cancel"

Using CLI

workloadmgr snapshot-cancel <snapshot_id>

Definition

A workload is a backup job that protects one or more Virtual Machines according to a configured policy. There can be as many workloads as needed. But each VM can only be part of one Workload.

List of Workloads

Using Horizon

To view all available workloads of a project inside Horizon do:

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

The overview in Horizon lists all workloads with the following additional information:

• Creation time

• Workload Name

• Workload description

• Total amount of Snapshots inside this workload

  • Total amount of succeeded Snapshots

  • Total amount of failed Snapshots

• Status of the Workload

Using CLI

workloadmgr workload-list [--all {True,False}] [--nfsshare <nfsshare>]

Workload Create

Using Horizon

To create a workload inside Horizon do the following steps:

1. Login to Horizon

2. Navigate to the Backups

3. Navigate to Workloads

4. Click "Create Workload"

5. Provide Workload Name and Workload Description on the first tab "Details"

6. Choose the Policy if available to use on the first tab "Details"

7. Choose if the Workload is encrypted on the first tab "Details"

8. Provide the secret UUID if Workload is encrypted on the first tab "Details"

9. Choose the VMs to protect on the second Tab "Workload Members"

10. Decide for the schedule of the workload on the Tab "Schedule"

11. Provide the Retention policy on the Tab "Policy"

12. Choose the Full Backup Interval on the Tab "Policy"

13. If required check "Pause VM" on the Tab "Options"

14. Click create

The created Workload will be available after a few seconds and starts to take backups according to the provided schedule and policy.

Using CLI

workloadmgr workload-create [--display-name <display-name>]
                            [--display-description <display-description>]
                            [--source-platform <source-platform>]
                            [--jobschedule <key=key-name>]
                            [--metadata <key=key-name>]
                            [--policy-id <policy_id>]
                            [--encryption <True/False>]
                            [--secret-uuid <secret_uuid>]
                            <instance-id=instance-uuid> [<instance-id=instance-uuid> ...]

Workload Overview

A workload contains many information, which can be seen in the workload overview.

Using Horizon

To enter the workload overview inside Horizon do the following steps:

1. Login to Horizon

2. Navigate to the Backups

3. Navigate to Workloads

4. Identify the workload to show the details on

5. Click the workload name to enter the Workload overview

Details Tab

The Workload Details tab provides you with the general most important information about the workload:

• Name

• Description

• Availability Zone

• List of protected VMs including the information of qemu guest agent availability

Snapshots Tab

The Workload Snapshots Tab shows the list of all available Snapshots in the chosen Workload.

From here it is possible to work with the Snapshots, create Snapshots on demand and start Restores.

Policy Tab

The Workload Policy Tab gives an overview of the current configured scheduler and retention policy. The following elements are shown:

• Scheduler Enabled / Disabled

• Start Date / Time

• End Date / Time

• RPO

• Time till next Snapshot run

• Retention Policy and Value

• Full Backup Interval policy and value

Filesearch Tab

The Workload Filesearch Tab provides access to the powerful search engine, which allows to find files and folders on Snapshots without the need of a restore.

Misc. Tab

The Workload Miscellaneous Tab shows the remaining metadata of the Workload. The following information are provided:

• Creation time

• last update time

• Workload ID

Using CLI

workloadmgr workload-show <workload_id> [--verbose <verbose>]

Edit a Workload

Workloads can be modified in all components to match changing needs.

Using Horizon

To edit a workload in Horizon do the following steps:

1. Login to the Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload to be modified

5. Click the small arrow next to "Create Snapshot" to open the sub-menu

6. Click "Edit Workload"

7. Modify the workload as desired - All parameters except workload type can be changed

8. Click "Update"

Using CLI

usage: workloadmgr workload-modify [--display-name <display-name>]
                                   [--display-description <display-description>]
                                   [--instance <instance-id=instance-uuid>]
                                   [--jobschedule <key=key-name>]
                                   [--metadata <key=key-name>]
                                   [--policy-id <policy_id>]
                                   <workload_id>

Delete a Workload

Once a workload is no longer needed it can be safely deleted.

Using Horizon

To delete a workload do the following steps:

1. Login to Horizon

2. Navigate to the Backups

3. Navigate to Workloads

4. Identify the workload to be deleted

5. Click the small arrow next to "Create Snapshot" to open the sub-menu

6. Click "Delete Workload"

7. Confirm by clicking "Delete Workload" yet again

Using CLI

workloadmgr workload-delete [--database_only <True/False>] <workload_id>

Unlock a Workload

Workloads that are actively taking backups or restores are locked for further tasks. It is possible to unlock a workload by force if necessary.

Using Horizon

1. Login to the Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload to unlock

5. Click the small arrow next to "Create Snapshot" to open the sub-menu

6. Click "Unlock Workload"

7. Confirm by clicking "Unlock Workload" yet again

Using CLI

workloadmgr workload-unlock <workload_id>

Reset a Workload

In rare cases it might be necessary to start a backup chain all over again, to ensure the quality of the created backups. To not recreate a Workload in such cases is it possible to reset a Workload.

The Workload reset will:

• Cancel all ongoing tasks

• Delete all existing Openstack Trilio Snapshots from the protected VMs

• recalculate the next Snapshot time

• take a full backup at the next Snapshot

Using Horizon

To reset a Workload do the following steps:

1. Login to the Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload to reset

5. Click the small arrow next to "Create Snapshot" to open the sub-menu

6. Click "Reset Workload"

7. Confirm by clicking "Reset Workload" yet again

Using CLI

workloadmgr workload-reset <workload_id>

Definition

Trilio can notify users via E-Mail upon the completion of backup and restore jobs.

The E-Mail will be sent to the owner of the Workload.

Requirements to activate E-Mail Notifications

To use the E-mail notifications, two requirements need to be met.

User E-Mail assigned

As the E-Mail will be sent to the owner of the Workload does the Openstack User, who created the workload, require to have an E-Mail address associated.

Trilio E-Mail Server configured

Trilio needs to know which E-Mail server to use, to send the E-mail notifications. Backup Administrators can do this in the "Backup Admin" area.

Activate/Deactivate the E-Mail Notifications

E-Mail notifications are activated tenant wide. To activate the E-Mail notification feature for a tenant follow these steps:

1. Login to Horizon

2. Navigate to the Backups

3. Navigate to Settings

4. Check/Uncheck the box for "Enable Email Alerts"

Example E-Mails

The following screenshots show example E-mails send by Trilio.

Definition

The file search functionality allows the user to search for files and folders located on a chosen VM in a workload in one or more Backups.

Navigating to the file search tab in Horizon

The file search tab is part of every workload overview. To reach it follow these steps:

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload a file search shall be done in

5. Click the workload name to enter the Workload overview

6. Click File Search to enter the file search tab

Configuring and starting a file search Horizon

A file search runs against a single virtual machine for a chosen subset of backups using a provided search string.

To run a file search the following elements need to be decided and configured

Choose the VM the file search shall run against

Under VM Name/ID choose the VM that the search is done upon. The drop down menu provides a list of all VMs that are part of any Snapshot in the Workload.

Set the File Path

The File Path defines the search string that is run against the chosen VM and Snapshots. This search string does support basic RegEx.

Example File Path for all files inside /etc : /etc/*

Define the Snapshots to search in

"Filter Snapshots by" is the third and last component that needs to be set. This defines which Snapshots are going to be searched.

There are 3 possibilities for a pre-filtering:

1. All Snapshots - Lists all Snapshots that contain the chosen VM from all available Snapshots

2. Last Snapshots - Choose between the last 10, 25, 50, or custom Snapshots and click Apply to get the list of the available Snapshots for the chosen VM that match the criteria.

3. Date Range - Set a start and end date and click apply to get the list of all available Snapshots for the chosen VM within the set dates.

After the pre-filtering is done all matching Snapshots are automatic prechosen. Uncheck any Snapshot that shall not be searched.

Start the File Search and retrieve the results in Horizon

To start a File Search the following elements need to be set:

• A VM to search in has to be chosen

• A valid File Path provided

• At least one Snapshot to search in selected

Once those have been set click "Search" to start the file search.

After a short time the results will be presented. The results are presented in a tabular format grouped by Snapshots and Volumes inside the Snapshot.

For each found file or folder the following information are provided:

• POSIX permissions

• Amount of links pointing to the file or folder

• User ID who owns the file or folder

• Group ID assigned to the file or folder

• Actual size in Bytes of the file or folder

• Time of creation

• Time of last modification

• Time of last access

• Full path to the found file or folder

Once the Snapshot of interest has been identified it is possible to go directly to the Snapshot using the "View Snapshot" option at the top of the table. It is also possible to directly mount the Snapshot using the "Mound Snapshot" Button at the end of the table.

Doing a CLI File Search

Definition

Every Workload has its own schedule. Those schedules can be activated, deactivated and modified.

A schedule is defined by:

• Status (Enabled/Disabled)

• Start Day/Time

• End Day

• Hrs between 2 snapshots

Disable a schedule

Using Horizon

To disable the scheduler of a single Workload in Horizon do the following steps:

1. Login to the Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload to be modified

5. Click the small arrow next to "Create Snapshot" to open the sub-menu

6. Click "Edit Workload"

7. Navigate to the tab "Schedule"

8. Uncheck "Enabled"

9. Click "Update"

Using CLI

Enable a schedule

Using Horizon

To disable the scheduler of a single Workload in Horizon do the following steps:

1. Login to the Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload to be modified

5. Click the small arrow next to "Create Snapshot" to open the sub-menu

6. Click "Edit Workload"

7. Navigate to the tab "Schedule"

8. check "Enabled"

9. Click "Update"

Using CLI