The Red Hat Openstack Platform Director is the supported and recommended method to deploy and maintain any RHOSP installation. Trilio integrates natively into the RHOSP Director and manual deployment methods are not supported for RHOSP.
1. Prepare for deployment
1.1] Select 'backup target' type
Backup target storage is used to store backup images taken by Trilio and also associated configuration needs. The following backup target types are supported by Trilio:
Backup Target Types
Required Configuration
1.2] Clone triliovault-cfg-scripts repository
The overcloud-deploy command must already have been run successfully prior to this point and overcloud should be available. Perform the following steps for 'undercloud' node on an existing RHOSP environment:
All commands need to be run as user 'stack' on undercloud node
RHOSP 16.0 is not supported anymore as RedHat has officially stopped supporting it. However, Trilio maintained it for some time and stopped the support from 4.1HF11 onwards. The latest hotfix available for RHOSP16.0 is 41.HF10. Reach out to the Support team for any help.
Ensure that the Trilio appliance connected to this installation is on the latest Hotfix version. Failure to ensure this may lead to your installation not working as expected.
Refer to this doc :
Run the following command to clone the triliovault-cfg-scripts github repository:
cd /home/stack
git clone -b hotfix-13-TVO/4.1 https://github.com/trilioData/triliovault-cfg-scripts.git
``
If your backup target type is 'Ceph-based S3' with SSL, skip this step. Otherwise, access the Red Hat Director scripts according to the RHOSP version being used:
RHOSP 13 - cd triliovault-cfg-scripts/redhat-director-scripts/rhosp13/
RHOSP 16.1 - cd triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/
RHOSP 16.2 - cd triliovault-cfg-scripts/redhat-director-scripts/rhosp16.2/
If your backup target is Ceph S3 with SSL and your SSL certificates are self-signed or authorized by private CA, you must provide the CA chain certificate to validate the SSL requests. Otherwise, skip this step. To do this:
Rename your CA chain cert file to s3-cert.pem.
Copy the renamed file into the following directory:
triliovault-cfg-scripts/redhat-director-scripts/redhat-director-scripts/<RHOSP_RELEASE_Directory>/puppet/trilio/filesIf your overcloud deploy command uses any other deploy artifact through an environment file, then you must merge Trilio deploy artifact url and your url in a single file.
Then access the Red Hat Director scripts according to the version being used:
From this point onwards in the documentation, only the following path will be used for examples: cd triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/
2] Upload Trilio-Puppet module
RHOSP 16.1
cd triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/
The following commands upload the Trilio puppet module to the overcloud registry. The upload only happens upon the next deployment.
Step 1: -
cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/scripts/
./upload_puppet_module.sh
The output of the above command looks like the following.
## Above command creates the following file.
ls -ll /home/stack/.tripleo/environments/puppet-modules-url.yaml
Trilio puppet module is uploaded to overcloud as a swift deploy artifact with heat resource name 'DeployArtifactURLs'.
Step 2: - Check Trilio's Puppet module artifact file and ensure that it looks like the following:
(undercloud) [stack@ucloud161 ~]$ cat /home/stack/.tripleo/environments/puppet-modules-url.yaml
# Heat environment to deploy artifacts via Swift Temp URL(s)
parameter_defaults:
DeployArtifactURLs:
- 'http://172.25.0.103:8080/v1/AUTH_46ba596219d143c8b076e9fcc4139fed/overcloud-artifacts/puppet-modules.tar.gz?temp_url_sig=c3972b7ce75226c278ab3fa8237d31cc1f2115bd&temp_url_expires=1646738377'
Step 3: -
Firstly, check to make sure that your overcloud deploy environment files uses deploy artifacts. To do this check string DeployArtifactURLs in your environment files (only those mentioned in the overcloud deploy command with -e option). If you find any environment file with the -e option, then your deploy command is using deploy artifacts.
If your deploy command is using deploy artifact, you must merge all deploy artifacts in a single file. For example, if your artifact file path is /home/stack/templates/user-artifacts.yaml, then perform the following steps to merge both urls in single file, which is passed to the overcloud deploy command with the -e option.
3.1] Add Trilio Datamover Api Service to role data file
This service needs to share the same role as the keystone and database service.
In case of the pre-defined roles will these services run on the role Controller.
In case of custom-defined roles, it is necessary to use the same role where OS::TripleO::Services::Keystone service installed.
Add the following line to the identified role:
'OS::TripleO::Services::TrilioDatamoverApi'
3.2] Add Trilio Datamover Service to role data file
This service needs to share the same role as the nova-compute service.
In case of the pre-defined roles will the nova-compute service run on the role Compute.
In case of custom-defined roles, it is necessary to use the role that the nova-compute service is using.
Add the following line to the identified role:
'OS::TripleO::Services::TrilioDatamover'
4] Prepare Trilio container images
All commands need to be run as user 'stack'
Trilio containers are pushed to 'RedHat Container Registry'.
Registry URL: registry.connect.redhat.com Container pull URLs are given below.
Please note that using the hotfix containers requires that the Trilio Appliance is getting upgraded to the desired hotfix level as well.
Refer to the word <HOTFIX-TAG-VERSION> as 4.1.94-hotfix-16 in the below sections
There are three registry methods available in the RedHat OpenStack Platform.
Remote Registry
Local Registry
Satellite Server
4.1] Remote Registry
Follow this section when 'Remote Registry' is used.
In this method, container images get downloaded directly on overcloud nodes during overcloud deploy/update command execution. User can set the remote registry to the RedHat registry or any other private registry that he wants to use.
The user needs to provide credentials for the registry in containers-prepare-parameter.yaml file.
Make sure other OpenStack service images are also using the same method to pull container images.
If it's not the case you can not use this method.
Populate containers-prepare-parameter.yaml with content like following. Important parameters are 'push_destination: false',
ContainerImageRegistryLogin: true and registry credentials.
Trilio container images are published to the registry registry.connect.redhat.com
Credentials of registry 'registry.redhat.io' will work for registry.connect.redhat.com registry too.
Note: This file - containers-prepare-parameter.yaml
Note: File 'containers-prepare-parameter.yaml' gets created as output of command 'openstack tripleo container image prepare'. Refer above document by RedHat
3. Make sure you have network connectivity to the above registries from all overcloud nodes. Otherwise, image pull operation will fail.
4. Populate the trilio_env.yaml with Trilio container image URLs:
Trilio Datamover container
Trilio Datamover API container
Trilio Horizon Plugin
trilio_env.yaml will be available in
cd triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_Directory/
vi trilio_env.yaml
Follow this section when 'local registry' is used on the undercloud.
In this case, it is necessary to push the Trilio containers to the undercloud registry.
Trilio provides shell scripts that will pull the containers from 'registry.connect.redhat.com' and push them to the undercloud and update the trilio_env.yaml.
RHOSP13
cd /home/stack/triliovault-cfg-scripts/redhat-director-scripts/rhosp13/scripts/
./prepare_trilio_images.sh <undercloud_ip/hostname> <HOTFIX-TAG-VERSION>-rhosp13
Provide backup target details and other necessary details in the provided environment file. This environment file will be used in the overcloud deployment to configure Trilio components. Container image names have already been populated in the preparation of the container images. Still it is recommended to verify the container URLs.
The following information are required additionally:
Network for the datamover api
datamover password
Backup target type {nfs/s3}
In case of NFS
list of NFS Shares
NFS options
In case of S3
S3 type {amazon_s3/ceph_s3}
S3 Access key
S3 Secret key
S3 Region name
S3 Bucket
S3 Endpoint URL
S3 Signature Version
S3 Auth Version
S3 SSL Enabled {true/false}
S3 SSL Cert
Use ceph_s3 for any non-aws S3 backup targets.
resource_registry:
OS::TripleO::Services::TrilioDatamover: ../services/trilio-datamover.yaml
OS::TripleO::Services::TrilioDatamoverApi: ../services/trilio-datamover-api.yaml
# NOTE: If there are addition customizations to the endpoint map (e.g. for
# other integration), this will need to be regenerated.
OS::TripleO::EndpointMap: endpoint_map.yaml
parameter_defaults:
## Enable Trilio's quota functionality on horizon
ExtraConfig:
horizon::customization_module: 'dashboards.overrides'
## Define network map for trilio datamover api service
ServiceNetMap:
TrilioDatamoverApiNetwork: internal_api
## Trilio Datamover Password for keystone and database
TrilioDatamoverPassword: "test1234"
## Trilio container pull urls
DockerTrilioDatamoverImage: devundercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-TAG-VERSION>-rhosp16.1
DockerTrilioDmApiImage: devundercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-TAG-VERSION>-rhosp16.1
## If you do not want Trilio's horizon plugin to replace your horizon container, just comment following line.
ContainerHorizonImage: devundercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-TAG-VERSION>-rhosp16.1
## Backup target type nfs/s3, used to store snapshots taken by triliovault
BackupTargetType: 'nfs'
## For backup target 'nfs'
NfsShares: '192.168.122.101:/opt/tvault'
NfsOptions: 'nolock,soft,timeo=180,intr,lookupcache=none'
## For backup target 's3'
## S3 type: amazon_s3/ceph_s3
S3Type: 'amazon_s3'
## S3 access key
S3AccessKey: ''
## S3 secret key
S3SecretKey: ''
## S3 region, if your s3 does not have any region, just keep the parameter as it is
S3RegionName: ''
## S3 bucket name
S3Bucket: ''
## S3 endpoint url, not required for Amazon S3, keep it as it is
S3EndpointUrl: ''
## S3 signature version
S3SignatureVersion: 'default'
## S3 Auth version
S3AuthVersion: 'DEFAULT'
## If S3 backend is not Amazon S3 and SSL is enabled on S3 endpoint url then change it to 'True', otherwise keep it as 'False'
S3SslEnabled: False
## If S3 backend is not Amazon S3 and SSL is enabled on S3 endpoint URL and SSL certificates are self signed, then
## user need to set this parameter value to: '/etc/tvault-contego/s3-cert.pem', otherwise keep it's value as empty string.
S3SslCert: ''
## Don't edit following parameter
EnablePackageInstall: True
6. Advanced Settings/Configuration
6.1 Haproxy customized configuration for Trilio dmapi service __
The existing default haproxy configuration works fine with most of the environments. Only when timeout issues with the dmapi are observed or other reasons are known, change the configuration as described here.
Following is the haproxy conf file location on haproxy nodes of the overcloud.
Trilio datamover api service haproxy configuration gets added to this file.
To change these default values, you need to do the following steps.
i) On the undercloud node, open the following file for edit (Edit <RHOSP_RELEASE> with your cloud's release information. Valid values are - rhosp13, rhosp16, rhosp16.1)
If the containers are in restarting state or not listed by the following command then your deployment is not done correctly. Please recheck if you followed the complete documentation.
8.1] On Controller node
Make sure Trilio dmapi and horizon containers are in a running state and no other Trilio container is deployed on controller nodes.
When the role for these containers is not "controller" check on respective nodes according to configured roles_data.yaml.
[root@overcloud-controller-0 heat-admin]# podman ps | grep trilio
26fcb9194566 rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-datamover-api:<HOTFIX-REL-VERSION>-rhosp16.1 kolla_start 5 days ago Up 5 days ago trilio_dmapi
094971d0f5a9 rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-REL-VERSION>-rhosp16.1 kolla_start 5 days ago Up 5 days ago horizon
Make sure Trilio datamover container is in running state and no other Trilio container is deployed on compute nodes.
[root@overcloud-novacompute-0 heat-admin]# podman ps | grep trilio
b1840444cc59 rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-datamover:<HOTFIX-REL-VERSION>-rhosp16.1 kolla_start 5 days ago Up 5 days ago trilio_datamover
8.3] On the node with Horizon service
Make sure horizon container is in running state. Please note that 'Horizon' container is replaced with Trilio Horizon container. This container will have latest OpenStack horizon + Trilio's horizon plugin.
[root@overcloud-controller-0 heat-admin]# podman ps | grep horizon
094971d0f5a9 rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:<HOTFIX-REL-VERSION>-rhosp16.1 kolla_start 5 days ago Up 5 days ago horizon
9] Additional Steps on Trilio Appliance
9.1] Change the nova user id on the Trilio Nodes
In RHOSP, nova user id on nova-compute docker container is set to '42436'. The 'nova' user id on the Trilio nodes needs to be set the same. Do the following steps on all Trilio nodes:
Download the shell script that will change the user id
Assign executable permissions
Execute the script
Verify that nova user and group id have changed to '42436'
## Download the shell script
$ curl -O https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/master/common/nova_userid.sh
## Assign executable permissions
$ chmod +x nova_userid.sh
## Execute the shell script to change 'nova' user and group id to '42436'
$ ./nova_userid.sh
## Ignore any errors and verify that 'nova' user and group id has changed to '42436'
$ id nova
uid=42436(nova) gid=42436(nova) groups=42436(nova),990(libvirt),36(kvm)
10] Troubleshooting for overcloud deployment failures
Trilio components will be deployed using puppet scripts.
openstack stack failures list overcloud
heat stack-list --show-nested -f "status=FAILED"
heat resource-list --nested-depth 5 overcloud | grep FAILED
=> If trilio datamover api containers does not start well or in restarting state, use following logs to debug.
docker logs trilio_dmapi
tailf /var/log/containers/trilio-datamover-api/dmapi.log
=> If trilio datamover containers does not start well or in restarting state, use following logs to debug.
docker logs trilio_datamover
tailf /var/log/containers/trilio-datamover/tvault-contego.log
