Installing on RHOSP10

The Red Hat Openstack Platform Director is the supported and recommended method to deploy and maintain any RHOSP installation.

Trilio is integrating natively into the RHOSP Director. Manual deployment methods are not supported for RHOSP.

Prepare for deployment

Depending whether the RHOSP environment is already installed or is getting installed for the first time different steps are done to be able to deploy Trilio.

Overcloud is not yet deployed

If overcloud is not deployed already, in that case user should install trilio rpms on overcloud image before starting deployment. Trilio RPM packages are provided through yum repo hosted on the Trilio VM.

To inject the Trilio yum repository on the overcloud image do the following commands:

##Replace TrilioVault_VM_IP with actual Trilio vm IP
$ cat trilio.repo
[trilio]
name=Trilio Repository
baseurl=http://<TrilioVault_VM_IP>:8085/yum-repo/
enabled=1
gpgcheck=0

$ virt-customize --selinux-relabel -a overcloud-full.qcow2 --upload trilio.repo:/etc/yum.repos.d/
[  0.0] Examining the guest ...
[ 12.0] Setting a random seed
[ 12.0] Copying: opendaylight.repo to /etc/yum.repos.d/
[ 13.0] Finishing off

Afterwards will the overcloud image be created.

virt-customize --selinux-relabel / 
-a overcloud-full.qcow2 / 
--install puppet-triliovault, /
python-workloadmgrclient, /
tvault-contego, /
tvault-contego-api, /
tvault-horizon-plugin

Overcloud is already deployed

If overcloud is deployed already it is necessary to prepare the artifacts to install on the overcloud.

All commands need to be run as user 'stack'

Firstly the github repository needs to be synced with the undercloud.

git clone https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts
git checkout stable/3.0
cd redhat-director-scripts/

Afterwards are those artifacts created and pushed to the overcloud nodes using the upload-swift-artifacts tool. This tool is provided on the undercloud and the prepare_artifacts.sh is created for it.

## Get help of script
(undercloud) [stack@undercloud redhat-director-scripts]$ ./prepare_artifacts.sh --help

Error: Script takes exactly two arguments, 1 provided
    First argument:    undercloud rc file path
    Second argument:    Trilio VM/cluster IP
For Example:
    ./prepare_artifacts.sh /home/stack/stackrc 192.168.122.201

##Run shell script
[root@redhat-undercloud redhat-director-scripts]# ./prepare_artifacts.sh /home/stack/stackrc 192.168.122.201

Update overcloud roles data file to include Trilio services

Trilio contains of multiple services. Add these services to your roles_data.yaml.

In case of uncostomized roles_data.yaml can the default file be found on the undercloud at:

/usr/share/openstack-tripleo-heat-templates/roles_data.yaml

Add the following services to the roles_data.yaml

Trilio Datamover Api Service

This service needs to share the same role as the nova-api service. In case of the pre-defined roles will the nova-api service run on the role Controller. In case of custom defined roles, it is necessary to use the role the nova-api service is using.

Add the following line to the identified role:

'OS::TripleO::Services::TrilioDatamoverApi'

Trilio Horizon Plugin

This service needs to share the same role as the openstack horizon service. In case of the pre-defined roles will the openstack horizon service run on the role Controller. In case of custom defined roles, it is necessary to use the role the openstack horizon service is using.

Add the following line to the identified role:

'OS::TripleO::Services::TrilioHorizonPlugin'

Trilio Datamover Service

This service needs to share the same role as the nova-compute service. In case of the pre-defined roles will the nova-compute service run on the role Compute. In case of custom defined roles, it is necessary to use the role the nova-compute service is using.

Add the following line to the identified role:

'OS::TripleO::Services::TrilioDatamover' 

Provide environment details to trilio_env.yaml

Provide backup target details like NFS share, S3 bucket details and other necessary details in trilio_env.yaml environment file. This environment file will be used in overcloud deployment to configure trilio components.

Sample trilio_env.yaml

resource_registry:
  OS::TripleO::Services::TrilioDatamover: heat/trilio-datamover.yaml
  OS::TripleO::Services::TrilioDatamoverApi: heat/trilio-datamover-api.yaml
  OS::TripleO::Services::TrilioHorizonPlugin: heat/trilio-horizon-plugin.yaml


parameter_defaults:

   ##Edit following parameters only if you are using non-default locations for nova configuration files,
   ##Otherwise just keep them as it is
   NovaConfFile: '/etc/nova/nova.conf'
   NovaDistConfFile: '/usr/share/nova/nova-dist.conf'
   RedhatOpenstackVersion: '10'

   ##Backup target nfs/s3
   BackupTargetType: 'nfs'
    
   ##For backup target 'nfs'
   NfsShares: ''
   NfsOptions: 'nolock,soft,timeo=180,intr'
   
   ##For backup target 's3'
   #S3 type: amazon_s3/ceph_s3/minio_s3
   S3Type: 'amazon_s3'
   S3AccessKey: ''
   S3SecretKey: ''
   S3RegionName: ''
   S3Bucket: ''
   S3EndpointUrl: ''
   S3SslEnabled: false
   S3SignatureVersion: 's3v4'

   #Horizon directory on controller node, this location will be used to install trilio horizon plugin
   HorizonDir: '/usr/share/openstack-dashboard/'
   EnablePackageInstall: True

Deploy overcloud with trilio environment

Use the following heat environment file and roles data file in overcloud deploy command:

1. trilio_env.yaml

2. roles_data.yaml

To include new environment files use '-e' option and for roles data file use '-r' option. An example overcloud deploy command is shown below:

/openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/templates/network-environment.yaml \
-e /home/stack/templates/storage-environment.yaml \
-e ${basedir}/trilio_env.yaml \
-r ${basedir}/roles_data.yaml \
--control-scale 1 --compute-scale 1 --control-flavor control --compute-flavor compute \
--ntp-server 0.north-america.pool.ntp.org --neutron-network-type vxlan --neutron-tunnel-types vxlan \
--validation-errors-fatal --validation-warnings-fatal \
--log-file overcloud_deploy.log

Verify deployment

The following packages should be installed on nodes with the role that contains the nova-api:

# rpm -qa | grep -E 'tvault|workloadmgr'
tvault-contego-api-3.0.44-3.0.noarch

The following packages should be installed nodes with the role that contains the openstack horizon:

# rpm -qa | grep -E 'tvault|workloadmgr'
tvault-horizon-plugin-3.0.44-3.0.noarch
python-workloadmgrclient-3.0.44-3.0.noarch

The following packages should be installed on nodes with the role that contain nova-compute:

# rpm -qa | grep -E 'tvault|trilio'
tvault-contego-3.0.44-3.0.noarch
puppet-triliovault-3.0.44-3.0.noarch

On the same nodes that contain the nova-compute service a new systemd service tvault-contego should have been registered and running. Further should the following mount be visible: /var/triliovault-mounts/<hash>

[root@overcloud-compute-0 opt]# systemctl status tvault-contego
● tvault-contego.service - Tvault contego
   Loaded: loaded (/etc/systemd/system/tvault-contego.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2018-07-14 18:08:20 UTC; 1 day 16h ago
 Main PID: 373191 (python)
   CGroup: /system.slice/tvault-contego.service
           └─373191 /home/tvault/.virtenv/bin/python /home/tvault/.virtenv/bin/tvault-contego --config-file=/usr/share/nova/nova-dist.conf --config-file=/e...

Jul 14 18:08:24 overcloud-compute-0 python[373191]: Option "verbose" from group "DEFAULT" is deprecated for removal.  Its value may be silently ign... future.
Jul 14 18:08:24 overcloud-compute-0 python[373191]: Option "rpc_backend" from group "DEFAULT" is deprecated for removal.  Its value may be silently... future.
Jul 14 18:08:24 overcloud-compute-0 sudo[373247]:     nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/umount -l /var/triliovault-mounts/MTkyL...c2hhcmU=
Jul 14 18:08:24 overcloud-compute-0 python[373191]: umount: /var/triliovault-mounts/MTkyLjE2OC4xMjIuMjQwOi9uZnNmaWxlc2hhcmU=: not mounted
Jul 14 18:08:24 overcloud-compute-0 sudo[373249]:     nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/service tvault-object-store stop
Jul 14 18:08:25 overcloud-compute-0 python[373191]: Redirecting to /bin/systemctl stop tvault-object-store.service
Jul 14 18:08:25 overcloud-compute-0 python[373191]: Failed to stop tvault-object-store.service: Unit tvault-object-store.service not loaded.
Jul 14 18:08:25 overcloud-compute-0 sudo[373260]:     nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/mount -o nolock,soft,timeo=180,intr 192...c2hhcmU=
Jul 14 18:08:26 overcloud-compute-0 python[373191]: /home/tvault/.virtenv/lib/python2.7/site-packages/nova/rpc.py:175: FutureWarning: The access_policy arg...
Jul 14 18:08:26 overcloud-compute-0 python[373191]: serializer=serializer)
Hint: Some lines were ellipsized, use -l to show in full.

[root@overcloud-compute-0 opt]# df -h
Filesystem                     Size  Used Avail Use% Mounted on
/dev/vda2                      200G  5.5G  195G   3% /
devtmpfs                        12G     0   12G   0% /dev
tmpfs                           12G     0   12G   0% /dev/shm
tmpfs                           12G  2.1M   12G   1% /run
tmpfs                           12G     0   12G   0% /sys/fs/cgroup
tmpfs                          2.4G     0  2.4G   0% /run/user/981
tmpfs                          2.4G     0  2.4G   0% /run/user/1000
192.168.122.240:/nfsfileshare   20G  1.7G   19G   9% /var/triliovault-mounts/MTkyLjE2OC4xMjIuMjQwOi9uZnNmaWxlc2hhcmU=

Lastly login into the Horizon Dashboard as admin user. Two new tabs should be visible:

1. Backups

2. Backups-Admin

If any rpm packages are missing or other verification steps fail verify that the given steps have been followed.

Troubleshooting for overcloud deployment failures

Trilio components will be deployed using puppet scripts.

In case of the overcloud deployment failing does the following command provide the list of errors:

openstack stack failures list overcloud

Further commands that can help identifying any errors.

heat stack-list --show-nested -f "status=FAILED"
heat resource-list --nested-depth 5 overcloud | grep FAILED

Cinder backend is Ceph - additional steps

Add Ceph details to configuration file

If Cinder backend is Ceph it is necessary to manually add the ceph details to tvault-contego.conf on all compute nodes.

The file can be found here: /var/lib/config-data/puppet-generated/triliodm/etc/tvault-contego/tvault-contego.conf

Add the following information:

[libvirt]
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = <rbd_user>

[ceph] 
keyring_ext = .<rbd_user>.keyring

The same block of information can be found in the nova.conf file.

Give nova user read permission

In case that the user nova does not have permission to read and use the ceph conf and keyring files, run the following commands to provide the necessary access:

setfacl -m u:nova:r /etc/ceph/<ceph_keyring_file_path>
setfacl -m u:nova:r /etc/ceph/<ceph_conf_file_path>