Trilio is integrating natively into the RHOSP Director.
Manual deployment methods are not supported for RHOSP.
1. Prepare for deployment
Refer to the below-mentioned acceptable values for the placeholders triliovault_tag, trilio_branch, RHOSP_version and CONTAINER-TAG-VERSION in this document as per the Openstack environment:
Trilio Release
triliovault_tag
trilio_branch
RHOSP_version
CONTAINER-TAG-VERSION
1.1] Select 'backup target' type
Backup target storage is used to store backup images taken by Trilio and details needed for configuration:
The following backup target types are supported by Trilio
a) NFS
Need NFS share path
b) Amazon S3
- S3 Access Key
- Secret Key
- Region
- Bucket name
c) Other S3 compatible storage (Like Ceph based S3)
- S3 Access Key
- Secret Key
- Region
- Endpoint URL (Valid for S3 other than Amazon S3)
- Bucket name
1.2] Clone triliovault-cfg-scripts repository
The following steps are to be done on the undercloud node on an already installed RHOSP environment.
The overcloud-deploy command has to be run successfully already and the overcloud should be available.
All commands need to be run as a stack user on the undercloud node
Refer to the word <RHOSP_RELEASE_DIRECTORY> as either rhosp16 or rhosp17 depending on your based RHOSP version in the below sections
The following command clones the triliovault-cfg-scripts github repository.
cd /home/stack
source stackrc
git clone -b {{ trilio_branch }} https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>
1.3] Set executable permissions for all shell scripts
cd triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/scripts/
chmod +x *.sh
1.4] If the backup target type is 'Ceph based S3' with SSL:
If your backup target is CEPH S3 with SSL and SSL certificates are self-signed or authorized by a private CA, then the user needs to provide a CA chain certificate to validate the SSL requests. For that, the user needs to rename his ca chain cert file to s3-cert.pem and copy it into the directory - triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/puppet/trilio/files
2.1] Add Trilio Datamover Api and Trilio Workload Manager services to role data file
This service needs to share the same role as the keystone and database service.
In the case of the pre-defined roles, these services will run on the role Controller.
In the case of custom-defined roles, it is necessary to use the same role where OS::TripleO::Services::Keystone service installed.
2.2] Add Trilio Datamover Service to role data file
This service needs to share the same role as the nova-compute service.
In the case of the pre-defined roles will the nova-compute service run on the role Compute.
In the case of custom-defined roles, it is necessary to use the role that the nova-compute service uses.
Add the following line to the identified role:
'OS::TripleO::Services::TrilioDatamover'
3] Prepare Trilio container images
All commands need to be run as a 'stack' user
Trilio containers are pushed to the RedHat Container Registry.
Registry URL: 'registry.connect.redhat.com'. Container pull URLs are given below.
There are three registry methods available in the RedHat OpenStack Platform.
Remote Registry
Local Registry
Satellite Server
3.1] Remote Registry
Follow this section when 'Remote Registry' is used.
In this method, container images get downloaded directly on overcloud nodes during overcloud deploy/update command execution. Users can set the remote registry to a redhat registry or any other private registry that they want to use.
The user needs to provide credentials for the registry in containers-prepare-parameter.yaml file.
Make sure other OpenStack service images are also using the same method to pull container images.
If it's not the case you can not use this method.
Populate containers-prepare-parameter.yaml with content like the following. Important parameters are push_destination: false,
ContainerImageRegistryLogin: true and registry credentials.
Trilio container images are published to registry registry.connect.redhat.com.
Credentials of registry 'registry.redhat.io' will work for registry.connect.redhat.com registry too.
Note: File 'containers-prepare-parameter.yaml' gets created as output of command 'openstack tripleo container image prepare'. Refer above document by RedHat
3. Make sure you have network connectivity to the above registries from all overcloud nodes. Otherwise image pull operation will fail.
4. The user needs to manually populate the trilio_env.yaml file with Trilio container image URLs as given below:
trilio_env.yaml file path:
cd triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/
At this step, you have configured Trilio image URLs in the necessary environment file.
3.2] Local Registry
Follow this section when 'local registry' is used on the undercloud.
In this case, it is necessary to push the Trilio containers to the undercloud registry.
Trilio provides shell scripts that will pull the containers from registry.connect.redhat.com and push them to the undercloud and update the trilio_env.yaml.
cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16/scripts/
sudo ./prepare_trilio_images.sh <UNDERCLOUD_REGISTRY_HOSTNAME> <CONTAINER-TAG-VERSION>-rhosp16.1
## Run following command to find 'UNDERCLOUD_REGISTRY_HOSTNAME', 'undercloud2-161.ctlplane.trilio.local' is the undercloud registry hostname in below example
$ openstack tripleo container image list | grep keystone
| docker://undercloud2-161.ctlplane.trilio.local:8787/rhosp-rhel8/openstack-keystone:16.1 |
| docker://undercloud2-161.ctlplane.trilio.local:8787/rhosp-rhel8/openstack-barbican-keystone-listener:16.1 |
## Example of running the script with parameters
sudo ./prepare_trilio_images.sh undercloud2-161.ctlplane.trilio.local 5.0.14-rhosp16.1
## Verify changes
$ grep '<CONTAINER-TAG-VERSION>-rhosp16.1' ../environments/trilio_env.yaml
ContainerTriliovaultDatamoverImage: undercloudqa161.ctlplane.trilio.local:8787/trilio/trilio-datamover:<CONTAINER-TAG-VERSION>-rhosp16.1
ContainerTriliovaultDatamoverApiImage: undercloudqa161.ctlplane.trilio.local:8787/trilio/trilio-datamover-api:<CONTAINER-TAG-VERSION>-rhosp16.1
ContainerTriliovaultWlmImage: undercloudqa161.ctlplane.trilio.local:8787/trilio/trilio-wlm:<CONTAINER-TAG-VERSION>-rhosp16.1
ContainerHorizonImage: undercloudqa161.ctlplane.trilio.local:8787/trilio/trilio-horizon-plugin:<CONTAINER-TAG-VERSION>-rhosp16.1
$ openstack tripleo container image list | grep <CONTAINER-TAG-VERSION>-rhosp16.1
| docker://undercloudqa161.ctlplane.trilio.local:8787/trilio/trilio-datamover-api:<CONTAINER-TAG-VERSION>-rhosp16.1 |
| docker://undercloudqa161.ctlplane.trilio.local:8787/trilio/trilio-horizon-plugin:<CONTAINER-TAG-VERSION>-rhosp16.1 |
| docker://undercloudqa161.ctlplane.trilio.local:8787/trilio/trilio-datamover:<CONTAINER-TAG-VERSION>-rhosp16.1 |
| docker://undercloudqa161.ctlplane.trilio.local:8787/trilio/trilio-wlm:<CONTAINER-TAG-VERSION>-rhosp16.1 |
cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16/scripts/
sudo ./prepare_trilio_images.sh <UNDERCLOUD_REGISTRY_HOSTNAME> <CONTAINER-TAG-VERSION>-rhosp16.2
## Run following command to find 'UNDERCLOUD_REGISTRY_HOSTNAME', 'undercloudqa162.ctlplane.trilio.local' is the undercloud registry hostname in below example
$ openstack tripleo container image list | grep keystone
| docker://undercloudqa162.ctlplane.trilio.local:8787/rhosp-rhel8/openstack-barbican-keystone-listener:16.2 |
| docker://undercloudqa162.ctlplane.trilio.local:8787/rhosp-rhel8/openstack-keystone:16.2 |
## Example of running the script with parameters
sudo ./prepare_trilio_images.sh undercloudqa162.ctlplane.trilio.local 5.0.14-rhosp16.2
## Verify changes
grep '<CONTAINER-TAG-VERSION>-rhosp16.2' ../environments/trilio_env.yaml
ContainerTriliovaultDatamoverImage: undercloudqa162.ctlplane.trilio.local:8787/trilio/trilio-datamover:<CONTAINER-TAG-VERSION>-rhosp16.2
ContainerTriliovaultDatamoverApiImage: undercloudqa162.ctlplane.trilio.local:8787/trilio/trilio-datamover-api:<CONTAINER-TAG-VERSION>-rhosp16.2
ContainerTriliovaultWlmImage: undercloudqa162.ctlplane.trilio.local:8787/trilio/trilio-wlm:<CONTAINER-TAG-VERSION>-rhosp16.2
ContainerHorizonImage: undercloudqa162.ctlplane.trilio.local:8787/trilio/trilio-horizon-plugin:<CONTAINER-TAG-VERSION>-rhosp16.2
$ openstack tripleo container image list | grep <CONTAINER-TAG-VERSION>-rhosp16.2
| docker://undercloudqa162.ctlplane.trilio.local:8787/trilio/trilio-datamover-api:<CONTAINER-TAG-VERSION>-rhosp16.2 |
| docker://undercloudqa162.ctlplane.trilio.local:8787/trilio/trilio-horizon-plugin:<CONTAINER-TAG-VERSION>-rhosp16.2 |
| docker://undercloudqa162.ctlplane.trilio.local:8787/trilio/trilio-datamover:<CONTAINER-TAG-VERSION>-rhosp16.2 |
| docker://undercloudqa162.ctlplane.trilio.local:8787/trilio/trilio-wlm:<CONTAINER-TAG-VERSION>-rhosp16.2 |
cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp17/scripts/
sudo ./prepare_trilio_images.sh <UNDERCLOUD_REGISTRY_HOSTNAME> <CONTAINER-TAG-VERSION>-rhosp17.1
## Example of running the script with parameters
sudo ./prepare_trilio_images.sh undercloudqa17.ctlplane.trilio.local 5.2.2-rhosp17.1
## Verify changes
grep '<CONTAINER-TAG-VERSION>-rhosp17.1' ../environments/trilio_env.yaml
ContainerTriliovaultDatamoverImage: undercloudqa17.ctlplane.trilio.local:8787/trilio/trilio-datamover:<CONTAINER-TAG-VERSION>-rhosp17.1
ContainerTriliovaultDatamoverApiImage: undercloudqa17.ctlplane.trilio.local:8787/trilio/trilio-datamover-api:<CONTAINER-TAG-VERSION>-rhosp17.1
ContainerTriliovaultWlmImage: undercloudqa17.ctlplane.trilio.local:8787/trilio/trilio-wlm:<CONTAINER-TAG-VERSION>-rhosp17.1
ContainerHorizonImage: undercloudqa17.ctlplane.trilio.local:8787/trilio/trilio-horizon-plugin:<CONTAINER-TAG-VERSION>-rhosp17.1
$ openstack tripleo container image list | grep <CONTAINER-TAG-VERSION>-rhosp17.1
| docker://undercloudqa17.ctlplane.trilio.local:8787/trilio/trilio-datamover-api:<CONTAINER-TAG-VERSION>-rhosp17.1 |
| docker://undercloudqa17.ctlplane.trilio.local:8787/trilio/trilio-horizon-plugin:<CONTAINER-TAG-VERSION>-rhosp17.1 |
| docker://undercloudqa17.ctlplane.trilio.local:8787/trilio/trilio-datamover:<CONTAINER-TAG-VERSION>-rhosp17.1 |
| docker://undercloudqa17.ctlplane.trilio.local:8787/trilio/trilio-wlm:<CONTAINER-TAG-VERSION>-rhosp17.1 |
At this step, you have downloaded Trilio container images and configured Trilio image URLs in the necessary environment file.
3.3] Red Hat Satellite Server
Follow this section when a Satellite Server is used for the container registry.
Pull the Trilio containers on the Red Hat Satellite using the given Red Hat registry URLs.
At this step, you have downloaded Trilio container images into the RedHat satellite server and configured Trilio image URLs in the necessary environment file.
4] Provide environment details in trilio-env.yaml
Edit /home/stack/triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_env.yaml file and provide backup target details and other necessary details in the provided environment file. This environment file will be used in the overcloud deployment to configure Trilio components. Container image names have already been populated in the preparation of the container images. Still, it is recommended to verify the container URLs.
You don't need to provide anything for resource_registry, keep it as it is.
4.1] For enabling the VM migration feature, populate the below mentioned parameters
6.1] User needs to source 'overcloudrc' file of cloud admin user. This is needed to run OpenStack CLI.
For only this section user needs to source the cloudrc file of overcloud node
source <OVERCLOUD_RC_FILE>
6.2] User must have filled in the cloud admin user details of overcloud in 'trilio_env.yaml' file in the 'Provide environment details in trilio-env.yaml' section. If not please do so.
vi /home/stack/triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_env.yaml
6.3] Cloud admin user should have admin role on cloud admin domain
openstack role add --user <cloud-Admin-UserName> --domain <Cloud-Admin-DomainName> admin
# Example
openstack role add --user admin --domain default admin
6.4] After this, user can run the following script.
cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/scripts
./create_wlm_ids_conf.sh
7] Load necessary Linux drivers on all Controller and Compute nodes
For TrilioVault functionality to work, we need the following Linux kernel modules to be loaded on all controllers and compute nodes(Where Trilio WLM and Datamover services are going to be installed).
7.1] Install nbd module
modprobe nbd nbds_max=128
lsmod | grep nbd
7.2] Install fuse module
modprobe fuse
lsmod | grep fuse
8] Upload Trilio puppet module
All commands need to be run as a 'stack' user on undercloud node
8.1] Source the stackrc
source stackrc
8.2] The following commands upload the Trilio puppet module to the overcloud registry. The actual upload happens upon the next deployment.
cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16/scripts/
./upload_puppet_module.sh
## Output of the above command looks like following
Creating tarball...
Tarball created.
Creating heat environment file: /home/stack/.tripleo/environments/puppet-modules-url.yaml
Uploading file to swift: /tmp/puppet-modules-B1bp1Bk/puppet-modules.tar.gz
+-----------------------+---------------------+----------------------------------+
| object | container | etag |
+-----------------------+---------------------+----------------------------------+
| puppet-modules.tar.gz | overcloud-artifacts | 17ed9cb7a08f67e1853c610860b8ea99 |
+-----------------------+---------------------+----------------------------------+
Upload complete
## Above command creates the following file
ls -ll /home/stack/.tripleo/environments/puppet-modules-url.yaml
cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp17/scripts/
./upload_puppet_module.sh
## Output of above command looks like following
Creating tarball...
Tarball created.
renamed '/tmp/puppet-modules-MUIyvXI/puppet-modules.tar.gz' -> '/var/lib/tripleo/artifacts/overcloud-artifacts/puppet-modules.tar.gz'
Creating heat environment file: /home/stack/.tripleo/environments/puppet-modules-url.yaml
[stack@uc17-1 scripts]$ cat /home/stack/.tripleo/environments/puppet-modules-url.yaml
parameter_defaults:
DeployArtifactFILEs:
- /var/lib/tripleo/artifacts/overcloud-artifacts/puppet-modules.tar.gz
## Above command creates following file.
ls -ll /home/stack/.tripleo/environments/puppet-modules-url.yaml
9] Deploy overcloud with Trilio environment
9.1] Include the environment file defaults.yaml in overcloud deploy command with `-e` option as shown below.
This YAML file holds the default values, like default Trustee Role is creator and Keystone endpoint interface is Internal. There are some other parameters as well those User can update as per their requirements.
9.2] Additionally include the following heat environment files and roles data file we mentioned in the above sections in an overcloud deploy command:
trilio_env.yaml
roles_data.yaml
passwords.yaml
defaults.yaml
Use the correct Trilio endpoint map file as per the available Keystone endpoint configuration. You have to remove your OpenStack's endpoint map file from overcloud deploy command and instead of that use Trilio endpoint map file.
Instead of tls-endpoints-public-dns.yaml file, use triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_env_tls_endpoints_public_dns.yaml
Instead of tls-endpoints-public-ip.yaml file, usetriliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_env_tls_endpoints_public_ip.yaml
Instead of tls-everywhere-endpoints-dns.yaml file, usetriliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_env_tls_everywhere_dns.yaml
Instead of no-tls-endpoints-public-ip.yaml file, usetriliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_env_non_tls_endpoints_ip.yaml
To include new environment files use -e option and for roles data files use -r option.\
Below is an example of an overcloud deploy command with Trilio environment:
Post-deployment for the multipath enabled environment, log into respective Datamover container and add uxsock_timeout with value as 60000 (i.e. 60 sec) in /etc/multipath.conf. Restart datamover container
This section is only required when the Multi-IP feature for NFS is required.
This feature allows us to set the IP to access the NFS Volume per datamover instead of globally.
i] On Undercloud node, change the directory
cd triliovault-cfg-scripts/common/
ii] Edit file triliovault_nfs_map_input.yml in the current directory and provide compute host and NFS share/IP map.
Get the overcloud Controller and Compute hostnames from the following command. Check Name column. Use exact host names in the triliovault_nfs_map_input.yml file.
Run this command on undercloud by sourcing stackrc.
(undercloud) [stack@ucqa161 ~]$ openstack server list
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+
| 8c3d04ae-fcdd-431c-afa6-9a50f3cb2c0d | overcloudtrain1-controller-2 | ACTIVE | ctlplane=172.30.5.18 | overcloud-full | control |
| 103dfd3e-d073-4123-9223-b8cf8c7398fe | overcloudtrain1-controller-0 | ACTIVE | ctlplane=172.30.5.11 | overcloud-full | control |
| a3541849-2e9b-4aa0-9fa9-91e7d24f0149 | overcloudtrain1-controller-1 | ACTIVE | ctlplane=172.30.5.25 | overcloud-full | control |
| 74a9f530-0c7b-49c4-9a1f-87e7eeda91c0 | overcloudtrain1-novacompute-0 | ACTIVE | ctlplane=172.30.5.30 | overcloud-full | compute |
| c1664ac3-7d9c-4a36-b375-0e4ee19e93e4 | overcloudtrain1-novacompute-1 | ACTIVE | ctlplane=172.30.5.15 | overcloud-full | compute |
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+
Edit the input map file triliovault_nfs_map_input.yml and fill in all the details. Refer to this page for details about the structure.
Below is an example of how you can set the multi-IP NFS details:
You can not configure the different IPs for the Controllers/WLM nodes, you need to use the same share on all the controller nodes.
You can configure the different IPs for Compute/Datamover nodes
If pip isn't available please install pip on the undercloud.
sudo pip3 install PyYAML==5.1 3
Expand the map file to create a one-to-one mapping of the compute nodes and the NFS shares.
python3 ./generate_nfs_map.py
iv] Validate output map file
The result will be stored in the triliovault_nfs_map_output.yml file
Open file triliovault_nfs_map_output.yml available in the current directory and validate that all compute nodes are covered with all the necessary NFS shares.
v] Append this output map file to triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_nfs_map.yaml
viii] After this you need to run the overcloud deployment.
12.2] Haproxy customized configuration for Trilio dmapi service
The existing default HAproxy configuration works fine with most of the environments. Only when timeout issues with the Trilio Datamover Api are observed or other reasons are known, change the configuration as described here.
Following is the HAproxy conf file location on HAproxy nodes of the overcloud.
Trilio Datamover API service HAproxy configuration gets added to this file.
iii) Save the changes and do the overcloud deployment again to reflect these changes for overcloud nodes.
12.3] Configure Custom Volume/Directory Mounts for the Trilio Datamover Service
i) If the user wants to add one or more extra volume/directory mounts to the Trilio Datamover Service container, they can use a variable named 'TrilioDatamoverOptVolumes' is available in the below file.
To add one more extra volume/directoy mount to the Trilio Datamover Service container it is necessary that volumes/directories should already be mounted on the Compute host
ii) The variable 'TrilioDatamoverOptVolumes' accepts a list of volume/bind mounts. User needs to edit the file and add their volume mounts in the below format.
TrilioDatamoverOptVolumes:
- <mount-dir-on-compute-host>:<mount-dir-inside-the-datamover-container>
## For example, below is the `/mnt/mount-on-host` mount directory mounted on Compute host that directory you want to mount on the `/mnt/mount-inside-container` directory inside the Datamover container
[root@overcloudtrain5-novacompute-0 heat-admin]# df -h | grep 172.
172.25.0.10:/mnt/tvault/42436 2.5T 2.3T 234G 91% /mnt/mount-on-host
## Then provide that mount in the below format
TrilioDatamoverOptVolumes:
- /mnt/mount-on-host:/mnt/mount-inside-container
iii) Lastly you need to do overcloud deploy/update.
After successful deployment, you will see that volume/directory mount will be mounted inside the Trilio Datamover Service container.
