T4O-4.0

Installing on RHOSP16.1

The Red Hat Openstack Platform Director is the supported and recommended method to deploy and maintain any RHOSP installation.

Trilio is integrating natively into the RHOSP Director. Manual deployment methods are not supported for RHOSP.

Prepare for deployment

Depending whether the RHOSP environment is already installed or is getting installed for the first time different steps are done to be able to deploy Trilio.

All commands need to be run as user 'stack'

The following command clones the Trilio configscripts github.

cd /home/stack
git clone -b v4.0maintenance https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/redhat-director-scripts/

If your backup target is ceph S3 with SSL and SSL certificates are self signed or authorized by private CA, then user needs to provide CA chain certificate to validate the SSL requests. For that, user needs to rename his ca chain cert file to 's3-cert.pem' and copy it into directory - 'triliovault-cfg-scripts/redhat-director-scripts/redhat-director-scripts/puppet/trilio/files'

cp s3-cert.pem /home/stack/triliovault-cfg-scripts/redhat-director-scripts/puppet5/trilio/files/

The following commands upload the Trilio puppet module to the overcloud registry. The actual upload happens upon the next deployment.

# ./upload_puppet_module_puppet5.sh
Creating tarball...
Tarball created.
Creating heat environment file: /home/stack/.tripleo/environments/puppet-modules-url.yaml
Uploading file to swift: /tmp/puppet-modules-8Qjya2X/puppet-modules.tar.gz
+-----------------------+---------------------+----------------------------------+
| object                | container           | etag                             |
+-----------------------+---------------------+----------------------------------+
| puppet-modules.tar.gz | overcloud-artifacts | 368951f6a4d39cfe53b5781797b133ad |
+-----------------------+---------------------+----------------------------------+

## Make sure that
ls -ll /home/stack/.tripleo/environments/puppet-modules-url.yaml

Update overcloud roles data file to include Trilio services

Trilio contains of multiple services. Add these services to your roles_data.yaml.

In case of uncostomized roles_data.yaml can the default file be found on the undercloud at:

/usr/share/openstack-tripleo-heat-templates/roles_data.yaml

Add the following services to the roles_data.yaml

Trilio Datamover Api Service

This service needs to share the same role as the nova-api service. In case of the pre-defined roles will the nova-api service run on the role Controller. In case of custom defined roles, it is necessary to use the role the nova-api service is using.

Add the following line to the identified role:

'OS::TripleO::Services::TrilioDatamoverApi'

Trilio Datamover Service

This service needs to share the same role as the nova-compute service. In case of the pre-defined roles will the nova-compute service run on the role Compute. In case of custom defined roles, it is necessary to use the role the nova-compute service is using.

Add the following line to the identified role:

'OS::TripleO::Services::TrilioDatamover'

Prepare Trilio container images

All commands need to be run as user 'stack'

Trilio containers are pushed to 'RedHat Container Registry'. Registry URL: 'registry.connect.redhat.com'. Container pull urls are given below.

Trilio Datamover container: registry.connect.redhat.com/trilio/trilio-datamover:4.0.116-rhosp16.1 Trilio Datamover Api Container: registry.connect.redhat.com/trilio/trilio-datamover-api:4.0.116-rhosp16.1 Trilio horizon plugin: registry.connect.redhat.com/trilio/trilio-horizon-plugin:4.0.116-rhosp16.1

There are three registry methods available in RedHat Openstack Platform.

  1. Remote Registry

  2. Local Registry

  3. Satellite Server

Remote Registry

Follow this section when 'Remote Registry' is used.

For this method it is not necessary to pull the containers in advance. It is only necessary to populate the trilio_env.yaml file with the Trilio container URLs from Redhat registry.

Populate the trilio_env_osp16.yaml with container urls for:

  • Trilio Datamover container

  • Trilio Datamover api container

  • Trilio Horizon Plugin

trilio_env_osp16.yaml will be available in triliovault-cfg-scripts/redhat-director-scripts/

(undercloud) [stack@undercloud redhat-director-scripts]$ grep 'Image' trilio_env_osp16.yaml
   DockerTrilioDatamoverImage: registry.connect.redhat.com/trilio/trilio-datamover:4.0.116-rhosp16.1
   DockerTrilioDmApiImage: registry.connect.redhat.com/trilio/trilio-datamover-api:4.0.116-rhosp16.1
   ContainerHorizonImage: registry.connect.redhat.com/trilio/trilio-horizon-plugin:4.0.116-rhosp16.1

Local Registry

Follow this section when 'local registry' is used on the undercloud.

In this case it is necessary to pull the Trilio containers to the undercloud registry. Trilio provides shell scripts which will pull the containers from 'registry.connect.redhat.com' to the undercloud and updates the trilio_env_osp16.yaml with the values for the datamover and datamover api containers.

## Command to prepare, push trilio container images to undercloud registry: 
sudo ./podman_prepare_images.sh <UNDERCLOUD_REGISTRY_HOSTNAME> <CONTAINER_TAG> 


## Run following command to find 'UNDERCLOUD_REGISTRY_HOSTNAME'. Hostname is higlighted in bold format in below example.
$ openstack tripleo container image list | grep keystone
| docker://trilio-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-keystone:16.0-82                       |
| docker://trilio-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-barbican-keystone-listener:16.0-84   

## 'CONTAINER_TAG' format: 4.0.116-rhosp16.1
## For example. If Trilio version='4.0.116',  then 'CONTAINER_TAG'=4.0.116-rhosp16.1

## Example:
sudo ./podman_prepare_images.sh trilio-undercloud.ctlplane.localdomain 4.0.116-rhosp16.1

The script assumes that the undercloud container registry is running on port 8787. If the registry is running on a different port, the script needs to be adjusted manually.

The changes can be verified using the following commands.

(undercloud) [stack@undercloud redhat-director-scripts]$ openstack tripleo container image list | grep trilio
| docker://undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:4.0.116-rhosp16.1                       |                        |
| docker://undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:4.0.116-rhosp16.1                   |                   |
| docker://undercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:4.0.116-rhosp16.1                  |

-----------------------------------------------------------------------------------------------------

(undercloud) [stack@undercloud redhat-director-scripts]$ grep 'Image' trilio_env_osp16.yaml
   DockerTrilioDatamoverImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:4.0.116-rhosp16.1
   DockerTrilioDmApiImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:4.0.116-rhosp16.1
   ContainerHorizonImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:4.0.116-rhosp16.1

Satellite Server

Follow this section when a Satellite Server is used for the container registry.

Pull the Trilio containers on the Red Hat Satellite using the given Red Hat registry URLs.

Populate the trilio_env_osp16.yaml with container urls for:

  • Trilio Datamover container

  • Trilio Datamover api container

(undercloud) [stack@undercloud redhat-director-scripts]$ grep 'Image' trilio_env_osp16.yaml
   DockerTrilioDatamoverImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:4.0.116-rhosp16.1
   DockerTrilioDmApiImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:4.0.116-rhosp16.1
   ContainerHorizonImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:4.0.116-rhosp16.1

Provide environment details in trilio-env_osp16.yaml

Provide backup target details and other necessary details in the provided environment file. This environment file will be used in the overcloud deployment to configure Trilio components. Container image names have already been populated in the preparation of the container images. Still it is recommended to verify the container URLs.

The following information are required additionally:

  • Network for the datamover api

  • Backup target type {nfs/s3}

  • In case of NFS

    • list of NFS Shares

    • NFS options

  • In case of S3

    • s3 type {amazon_s3/ceph_s3}

    • Access key

    • Secret key

    • S3 Region name

    • S3 Bucket name

    • S3 Endpoint URL

    • S3 SSL Enabled {true/false}

    • S3 SSL Cert

resource_registry:
  OS::TripleO::Services::TrilioDatamover: docker/services/trilio-datamover-osp16.yaml
  OS::TripleO::Services::TrilioDatamoverApi: docker/services/trilio-datamover-api-osp16.yaml

parameter_defaults:

   ## Enable Trilio's quota functionality on horizon
   ExtraConfig:
     horizon::customization_module: 'dashboards.overrides'

   ## Define network map for trilio datamover api service
   ServiceNetMap:
       TrilioDatamoverApiNetwork: internal_api

   ## Trilio container pull urls
   DockerTrilioDatamoverImage: devundercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:4.0.92-rhosp16
   DockerTrilioDmApiImage: devundercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:4.0.92-rhosp16

   ## If you do not want Trilio's horizon plugin to replace your horizon container, just comment following line.
   ContainerHorizonImage: devundercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:4.0.92-rhosp16


   ## Backup target type nfs/s3, used to store snapshots taken by triliovault
   BackupTargetType: 'nfs'

   ## For backup target 'nfs'
   NfsShares: '192.168.122.101:/opt/tvault'
   NfsOptions: 'nolock,soft,timeo=180,intr,lookupcache=none'

   ## For backup target 's3'
   ## S3 type: amazon_s3/ceph_s3
   S3Type: 'amazon_s3'

   ## S3 access key
   S3AccessKey: ''
  
   ## S3 secret key
   S3SecretKey: ''

   ## S3 region, if your s3 does not have any region, just keep the parameter as it is
   S3RegionName: ''

   ## S3 bucket name
   S3Bucket: ''

   ## S3 endpoint url, not required for Amazon S3, keep it as it is
   S3EndpointUrl: ''

   ## If SSL enabled on S3 url, not required for Amazon S3, just keep it as it is
   S3SslEnabled: false

   ## If S3 backend is not Amazon S3 and SSL is enabled on S3 endpoint URL and SSL certificates are self signed, then
   ## user need to set this parameter value to: '/etc/tvault-contego/s3-cert.pem', otherwise keep it's value as empty string.
   S3SslCert: ''

   ## Don't edit following parameter
   EnablePackageInstall: True

Deploy overcloud with trilio environment

Use the following heat environment file and roles data file in overcloud deploy command:

  1. trilio_env_osp16.yaml

  2. roles_data.yaml

To include new environment files use '-e' option and for roles data file use '-r' option. An example overcloud deploy command is shown below:

openstack overcloud deploy --templates \
--libvirt-type qemu \
--ntp-server 192.168.1.34 \
-e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/trilio_env_osp16.yaml \
-e /home/stack/templates/node-info.yaml \
-e /home/stack/containers-prepare-parameter.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-tls.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/inject-trust-anchor.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \
-r /usr/share/openstack-tripleo-heat-templates/roles_data.yaml

Verify deployment

If the containers are in restarting state or not listed by the following command then your deployment is not done correctly. Please recheck if you followed the complete documentation.

On Controller node

Make sure Trilio dmapi and horizon containers are in a running state and no other Trilio container is deployed on controller nodes.

[root@overcloud-controller-0 heat-admin]# podman ps | grep trilio
26fcb9194566  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-datamover-api:4.0.116-rhosp16.1  kolla_start  5 days ago  Up 5 days ago  trilio_dmapi
094971d0f5a9  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:4.0.116-rhosp16.1  kolla_start  5 days ago  Up 5 days ago  horizon

On Compute node

Make sure Trilio datamover container is in running state and no other Trilio container is deployed on compute nodes.

[root@overcloud-novacompute-0 heat-admin]# podman ps | grep trilio
b1840444cc59  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-datamover:4.0.116-rhosp16.1  kolla_start  5 days ago  Up 5 days ago  tilio_datamover

On the node with Horizon service

Make sure horizon container is in running state. Please note that 'Horizon' container is replaced with Trilio Horizon container. This container will have latest OpenStack horizon + Trilio's horizon plugin.

[root@overcloud-controller-0 heat-admin]# podman ps | grep horizon
094971d0f5a9  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:4.0.116-rhosp16.1  kolla_start  5 days ago  Up 5 days ago  horizon

Configure Trilio Appliance

Once RHOSP16.1 Installation steps have completed successfully, follow the instructions below to now configure the Trilio Appliance.

Change the nova user id on the Trilio Nodes

In RHOSP, 'nova' user id on nova-compute docker container is set to '42436'. The 'nova' user id on the Trilio nodes need to be set the same. Do the following steps on all Trilio nodes:

  1. Download the shell script that will change the user id

  2. Assign executable permissions

  3. Execute the script

  4. Verify that 'nova' user and group id has changed to '42436'

# curl -O https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/master/common/nova_userid.sh
# chmod +x nova_userid.sh
# ./nova_userid.sh
# id nova
  uid=42436(nova) gid=42436(nova) groups=42436(nova),990(libvirt),36(kvm)

Run the Trilio configurator

Follow this documentation to configure Trilio Appliance.

Change the workloadmgr.conf to use the right mountpoint

It is necessary to first configure the Trilio appliance, before the steps of this section can be done.

RHOSP16 is using a different mount point in its datamover containers than other Openstack distribution. It is necessary to adjust the mountpoint of the Trilio Nodes to match this. If the mountpoints are not getting aligned, will files created by the datamover and read by the Trilio appliance not match in their paths and backup and restore processes will fail.

Please follow these steps to align the mountpoints:

  1. Edit /etc/workloadmgr/workloadmgr.conf file

  2. Set parameter 'vault_data_directory' to '/var/lib/nova/triliovault-mounts'

  3. create the directory for the mountpoint

  4. assign the created directory to nova:nova

  5. unmount the old mountpoint

  6. Update the Trilio configurator

  7. Restart the Trilio services

  8. Verify the mountpoint has been configured correctly

# cat /etc/workloadmgr/workloadmgr.conf | grep vault_data_directory
vault_data_directory = /var/lib/nova/triliovault-mounts
vault_data_directory_old = /var/triliovault

# mkdir -p /var/lib/nova/triliovault-mounts

# chown nova:nova /var/lib/nova/triliovault-mounts

# umount /var/triliovault-mounts

# cat /home/stack/myansible/lib/python3.6/site-packages/workloadmgr/tvault_configurator/ansible-play/roles/ansible-workloadmgr/templates/workloadmgr.conf.j2 | grep vault_data_directory
vault_data_directory = /var/lib/nova/triliovault-mounts

#pcs resource restart wlm-cron 

#pcs resource restart wlm-scheduler 

#systemctl restart wlm-api 

#systemctl restart wlm-workloads

#df-h
Filesystem             Size  Used Avail Use% Mounted on
devtmpfs               3.8G     0  3.8G   0% /dev
tmpfs                  3.8G   38M  3.8G   1% /dev/shm
tmpfs                  3.8G  427M  3.4G  12% /run
tmpfs                  3.8G     0  3.8G   0% /sys/fs/cgroup
/dev/vda1               40G  8.8G   32G  22% /
tmpfs                  773M     0  773M   0% /run/user/996
tmpfs                  773M     0  773M   0% /run/user/0
10.10.2.20:/upstream  1008G  704G  254G  74% /var/lib/nova/triliovault-mounts/MTAuMTAuMi4yMDovdXBzdHJlYW0=

Troubleshooting for overcloud deployment failures

Trilio components will be deployed using puppet scripts.

In case of the overcloud deployment failing does the following command provide the list of errors:

openstack stack failures list overcloud

Further commands that can help identifying any errors.

heat stack-list --show-nested -f "status=FAILED"
heat resource-list --nested-depth 5 overcloud | grep FAILED

In case of the trilio datamover api container is failing to start or stuck in restart, check these logs:

sudo podman logs trilio_dmapi
tailf /var/log/containers/trilio-datamover-api/dmapi.log

In case of Trilio datamover container failing to start or being stuck in restart, check these logs:

sudo podman logs trilio_datamover
tailf /var/log/containers/trilio-datamover/tvault-contego.log

Cinder backend is Ceph - additional steps

Add Ceph details to configuration file

If Cinder backend is Ceph it is necessary to manually add the ceph details to tvault-contego.conf on all compute nodes.

The file can be found here: /var/lib/config-data/puppet-generated/triliodm/etc/tvault-contego/tvault-contego.conf

Add the following information:

[libvirt]
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = <rbd_user>

[ceph] 
keyring_ext = .<rbd_user>.keyring

The same block of information can be found in the nova.conf file.

Restart the datamover container

sudo podman restart trilio_datamover
PreviousInstalling on RHOSP16.0NextInstalling on Canonical Openstack Queens and Train

Last updated