1 of 3

Installing Trilio Components

Once the Trilio VM or the Cluster of Trilio VMs has been spun, the actual installation process can begin. This process contains of the following steps:

Install the Trilio dm-api service on the control plane
Install the Trilio datamover service on the compute plane
Install the Trilio Horizon plugin into the Horizon service

How these steps look in detail is depending on the Openstack distribution Trilio is installed in. Each supported Openstack distribution has its own deployment tools. Trilio is integrating into these deployment tools to provide a native integration from the beginning to the end.

Installing on RHOSP10

The Red Hat Openstack Platform Director is the supported and recommended method to deploy and maintain any RHOSP installation.

Trilio is integrating natively into the RHOSP Director. Manual deployment methods are not supported for RHOSP.

Prepare for deployment

Depending whether the RHOSP environment is already installed or is getting installed for the first time different steps are done to be able to deploy Trilio.

Overcloud is not yet deployed

If overcloud is not deployed already, in that case user should install trilio rpms on overcloud image before starting deployment. Trilio RPM packages are provided through yum repo hosted on the Trilio VM.

To inject the Trilio yum repository on the overcloud image do the following commands:

##Replace TrilioVault_VM_IP with actual Trilio vm IP
$ cat trilio.repo
[trilio]
name=Trilio Repository
baseurl=http://<TrilioVault_VM_IP>:8085/yum-repo/
enabled=1
gpgcheck=0

$ virt-customize --selinux-relabel -a overcloud-full.qcow2 --upload trilio.repo:/etc/yum.repos.d/
[  0.0] Examining the guest ...
[ 12.0] Setting a random seed
[ 12.0] Copying: opendaylight.repo to /etc/yum.repos.d/
[ 13.0] Finishing off

Afterwards will the overcloud image be created.

virt-customize --selinux-relabel / 
-a overcloud-full.qcow2 / 
--install puppet-triliovault, /
python-workloadmgrclient, /
tvault-contego, /
tvault-contego-api, /
tvault-horizon-plugin

Overcloud is already deployed

If overcloud is deployed already it is necessary to prepare the artifacts to install on the overcloud.

All commands need to be run as user 'stack'

Firstly the github repository needs to be synced with the undercloud.

git clone https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts
git checkout stable/3.0
cd redhat-director-scripts/

Afterwards are those artifacts created and pushed to the overcloud nodes using the upload-swift-artifacts tool. This tool is provided on the undercloud and the prepare_artifacts.sh is created for it.

## Get help of script
(undercloud) [stack@undercloud redhat-director-scripts]$ ./prepare_artifacts.sh --help

Error: Script takes exactly two arguments, 1 provided
    First argument:    undercloud rc file path
    Second argument:    Trilio VM/cluster IP
For Example:
    ./prepare_artifacts.sh /home/stack/stackrc 192.168.122.201

##Run shell script
[root@redhat-undercloud redhat-director-scripts]# ./prepare_artifacts.sh /home/stack/stackrc 192.168.122.201

Update overcloud roles data file to include Trilio services

Trilio contains of multiple services. Add these services to your roles_data.yaml.

In case of uncostomized roles_data.yaml can the default file be found on the undercloud at:

/usr/share/openstack-tripleo-heat-templates/roles_data.yaml

Add the following services to the roles_data.yaml

Trilio Datamover Api Service

This service needs to share the same role as the nova-api service. In case of the pre-defined roles will the nova-api service run on the role Controller. In case of custom defined roles, it is necessary to use the role the nova-api service is using.

Add the following line to the identified role:

'OS::TripleO::Services::TrilioDatamoverApi'

Trilio Horizon Plugin

This service needs to share the same role as the openstack horizon service. In case of the pre-defined roles will the openstack horizon service run on the role Controller. In case of custom defined roles, it is necessary to use the role the openstack horizon service is using.

Add the following line to the identified role:

'OS::TripleO::Services::TrilioHorizonPlugin'

Trilio Datamover Service

This service needs to share the same role as the nova-compute service. In case of the pre-defined roles will the nova-compute service run on the role Compute. In case of custom defined roles, it is necessary to use the role the nova-compute service is using.

Add the following line to the identified role:

'OS::TripleO::Services::TrilioDatamover'

Provide environment details to trilio_env.yaml

Provide backup target details like NFS share, S3 bucket details and other necessary details in trilio_env.yaml environment file. This environment file will be used in overcloud deployment to configure trilio components.

Sample trilio_env.yaml

resource_registry:
  OS::TripleO::Services::TrilioDatamover: heat/trilio-datamover.yaml
  OS::TripleO::Services::TrilioDatamoverApi: heat/trilio-datamover-api.yaml
  OS::TripleO::Services::TrilioHorizonPlugin: heat/trilio-horizon-plugin.yaml


parameter_defaults:

   ##Edit following parameters only if you are using non-default locations for nova configuration files,
   ##Otherwise just keep them as it is
   NovaConfFile: '/etc/nova/nova.conf'
   NovaDistConfFile: '/usr/share/nova/nova-dist.conf'
   RedhatOpenstackVersion: '10'

   ##Backup target nfs/s3
   BackupTargetType: 'nfs'
    
   ##For backup target 'nfs'
   NfsShares: ''
   NfsOptions: 'nolock,soft,timeo=180,intr'
   
   ##For backup target 's3'
   #S3 type: amazon_s3/ceph_s3/minio_s3
   S3Type: 'amazon_s3'
   S3AccessKey: ''
   S3SecretKey: ''
   S3RegionName: ''
   S3Bucket: ''
   S3EndpointUrl: ''
   S3SslEnabled: false
   S3SignatureVersion: 's3v4'

   #Horizon directory on controller node, this location will be used to install trilio horizon plugin
   HorizonDir: '/usr/share/openstack-dashboard/'
   EnablePackageInstall: True

Deploy overcloud with trilio environment

Use the following heat environment file and roles data file in overcloud deploy command:

trilio_env.yaml
roles_data.yaml

To include new environment files use '-e' option and for roles data file use '-r' option. An example overcloud deploy command is shown below:

/openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/templates/network-environment.yaml \
-e /home/stack/templates/storage-environment.yaml \
-e ${basedir}/trilio_env.yaml \
-r ${basedir}/roles_data.yaml \
--control-scale 1 --compute-scale 1 --control-flavor control --compute-flavor compute \
--ntp-server 0.north-america.pool.ntp.org --neutron-network-type vxlan --neutron-tunnel-types vxlan \
--validation-errors-fatal --validation-warnings-fatal \
--log-file overcloud_deploy.log

Verify deployment

The following packages should be installed on nodes with the role that contains the nova-api:

# rpm -qa | grep -E 'tvault|workloadmgr'
tvault-contego-api-3.0.44-3.0.noarch

The following packages should be installed nodes with the role that contains the openstack horizon:

# rpm -qa | grep -E 'tvault|workloadmgr'
tvault-horizon-plugin-3.0.44-3.0.noarch
python-workloadmgrclient-3.0.44-3.0.noarch

The following packages should be installed on nodes with the role that contain nova-compute:

# rpm -qa | grep -E 'tvault|trilio'
tvault-contego-3.0.44-3.0.noarch
puppet-triliovault-3.0.44-3.0.noarch

On the same nodes that contain the nova-compute service a new systemd service tvault-contego should have been registered and running. Further should the following mount be visible: /var/triliovault-mounts/<hash>

[root@overcloud-compute-0 opt]# systemctl status tvault-contego
● tvault-contego.service - Tvault contego
   Loaded: loaded (/etc/systemd/system/tvault-contego.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2018-07-14 18:08:20 UTC; 1 day 16h ago
 Main PID: 373191 (python)
   CGroup: /system.slice/tvault-contego.service
           └─373191 /home/tvault/.virtenv/bin/python /home/tvault/.virtenv/bin/tvault-contego --config-file=/usr/share/nova/nova-dist.conf --config-file=/e...

Jul 14 18:08:24 overcloud-compute-0 python[373191]: Option "verbose" from group "DEFAULT" is deprecated for removal.  Its value may be silently ign... future.
Jul 14 18:08:24 overcloud-compute-0 python[373191]: Option "rpc_backend" from group "DEFAULT" is deprecated for removal.  Its value may be silently... future.
Jul 14 18:08:24 overcloud-compute-0 sudo[373247]:     nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/umount -l /var/triliovault-mounts/MTkyL...c2hhcmU=
Jul 14 18:08:24 overcloud-compute-0 python[373191]: umount: /var/triliovault-mounts/MTkyLjE2OC4xMjIuMjQwOi9uZnNmaWxlc2hhcmU=: not mounted
Jul 14 18:08:24 overcloud-compute-0 sudo[373249]:     nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/service tvault-object-store stop
Jul 14 18:08:25 overcloud-compute-0 python[373191]: Redirecting to /bin/systemctl stop tvault-object-store.service
Jul 14 18:08:25 overcloud-compute-0 python[373191]: Failed to stop tvault-object-store.service: Unit tvault-object-store.service not loaded.
Jul 14 18:08:25 overcloud-compute-0 sudo[373260]:     nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/mount -o nolock,soft,timeo=180,intr 192...c2hhcmU=
Jul 14 18:08:26 overcloud-compute-0 python[373191]: /home/tvault/.virtenv/lib/python2.7/site-packages/nova/rpc.py:175: FutureWarning: The access_policy arg...
Jul 14 18:08:26 overcloud-compute-0 python[373191]: serializer=serializer)
Hint: Some lines were ellipsized, use -l to show in full.

[root@overcloud-compute-0 opt]# df -h
Filesystem                     Size  Used Avail Use% Mounted on
/dev/vda2                      200G  5.5G  195G   3% /
devtmpfs                        12G     0   12G   0% /dev
tmpfs                           12G     0   12G   0% /dev/shm
tmpfs                           12G  2.1M   12G   1% /run
tmpfs                           12G     0   12G   0% /sys/fs/cgroup
tmpfs                          2.4G     0  2.4G   0% /run/user/981
tmpfs                          2.4G     0  2.4G   0% /run/user/1000
192.168.122.240:/nfsfileshare   20G  1.7G   19G   9% /var/triliovault-mounts/MTkyLjE2OC4xMjIuMjQwOi9uZnNmaWxlc2hhcmU=

Lastly login into the Horizon Dashboard as admin user. Two new tabs should be visible:

Backups
Backups-Admin

If any rpm packages are missing or other verification steps fail verify that the given steps have been followed.

Troubleshooting for overcloud deployment failures

Trilio components will be deployed using puppet scripts.

In case of the overcloud deployment failing does the following command provide the list of errors:

openstack stack failures list overcloud

Further commands that can help identifying any errors.

heat stack-list --show-nested -f "status=FAILED"
heat resource-list --nested-depth 5 overcloud | grep FAILED

Cinder backend is Ceph - additional steps

Add Ceph details to configuration file

If Cinder backend is Ceph it is necessary to manually add the ceph details to tvault-contego.conf on all compute nodes.

The file can be found here: /var/lib/config-data/puppet-generated/triliodm/etc/tvault-contego/tvault-contego.conf

Add the following information:

[libvirt]
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = <rbd_user>

[ceph] 
keyring_ext = .<rbd_user>.keyring

The same block of information can be found in the nova.conf file.

Give nova user read permission

In case that the user nova does not have permission to read and use the ceph conf and keyring files, run the following commands to provide the necessary access:

setfacl -m u:nova:r /etc/ceph/<ceph_keyring_file_path>
setfacl -m u:nova:r /etc/ceph/<ceph_conf_file_path>

Installing on other supported Openstack distributions

Installing Trilio Nova API Extension on Controller Node¶

The next step in the installation process is to install the Trilio Nova API extension on each of the Nova API controller nodes. The purpose of the API extension is to route RESTful API calls to a Trilio Data Mover (which will be discussed next).

In case of multiple Controller nodes it is necessary to repeat these steps on all controller nodes running the Nova API service.

To install the Trilio Nova Extension on each Nova controller node, follow these steps after logging into a controller node.

[root@controller]# wget https://<ipaddress-of-triliovault-controller>/tvault-contego-install.sh --no-check-certificate
[root@controller]# chmod +x tvault-contego-install.sh
[root@controller]# ./tvault-contego-install.sh --install

Once the script has been started does it prompt for the following information:

Trilio IP This IP will be used to download the Trilio Nova API Extension pip packages.
Request whether the script is running on a Controller or Compute Node To install the Trilio Nova API Extension choose Controller

The Trilio Nova API extension is a download from Trilio appliance and installed on the controller node. After installation, the Nova-API service is restarted to load the Trilio Nova API extension into Nova-API process so Nova-API now can handle Data Mover requests from the Trilio appliance.

Installing Trilio Data Mover on Computer Node

Now it is time to install the Trilio Data Mover service on the compute nodes where the Nova Compute service is running. The purpose of the Data Mover service is, as the name says, to move the actual disk data to and from the backup target.

The Trilio Data Mover needs to be installed on all Compute Nodes to guarantee the possible protection of all VMs running inside the Openstack Cluster.

To install the Trilio Data Mover on a Compute node, follow these steps after logging into a compute node.

[root@compute]# wget https://<ipaddress-of-triliovault-controller>/tvault-contego-install.sh --no-check-certificate
[root@compute]# chmod +x tvault-contego-install.sh

At this point it is possible to run the installation either in an automated fashion or in an interactive mode.

Interactive install of the Trilio Data Mover

The following command will start the interactive installation.

[root@compute]# ./tvault-contego-install.sh  --install

Once the script has been started does it prompt for the following information:

Trilio IP This IP will be used to download the Trilio Nova API Extension pip packages.
Request whether the script is running on a Controller or Compute Node To install the Trilio Data Mover service choose Compute Node.
Request of the path to the compute.filters file Choose matching your distribution or provide the exact path.
Acceptance of Trilio changing the sudoers file for nova nova ALL = (root) NOPASSWD: /home/tvault/.virtenv/bin/privsep-helper *
Choose between NFS or S3 as backup target
In case of NFS
1. Provide the NFS Volume path
2. check and change the default NFS options if necessary
In case of S3
1. Choose the S3 profile to be used
2. provide access credentials according to the chosen profile

Automated installation of Trilio Data Mover¶

The Trilio Data Mover can be installed in an unattended way using a configuration file. The configuration file contains all the data that is required while the script is being executed.

The first step for the automated installation is to generate the answers file. A predefined file can be downloaded from any Trilio VM.

[root@compute]# wget https://<ipaddress-of-triliovault-controller>/tvault-contego-install.answers --no-check-certificate

Edit tvault-contego-install.answers file as required for compute nodes. An example can be seen below:

####IP address of Tvault appliance###

IP_ADDRESS=192.168.1.122
TVAULT_CONTEGO_EXT_USER=nova
TVAULT_CONTEGO_EXT_GROUP=nova

#Uncomment following node which you want to configure##
#controller=True
compute=True

#Need to add the following statement in nova sudoers file
#nova ALL = (root) NOPASSWD: /home/tvault/.virtenv/bin/privsep-helper *
#These changes require for Datamover, Otherwise Datamover will not work
# if you are agree then please uncomment the below line   NOVA_SUDOERS=require
#NOVA_SUDOERS=require

#Nova compute.filters file path
#Uncomment following line as per the OS distribution, you can edit the path as per your nova configuration
###For RHEL systems
NOVA_COMPUTE_FILTERS_FILE=/usr/share/nova/rootwrap/compute.filters

###For Debian systems
#NOVA_COMPUTE_FILTERS_FILE=/etc/nova/rootwrap.d/compute.filters

##For any other system
##NOVA_COMPUTE_FILTERS_FILE=


##### Select snapshot storage type #####
#Uncomment following two lines and edit for NFS as snapshot storage, Provide multiple nfs shares separated by comma
NFS=True
NFS_SHARES=192.168.1.33:/mnt/tvault,192.168.1.33:/mnt/tvault_1

#### Uncomment following lines and edit for Swift as snapshot storage
#Swift=False
#VAULT_SWIFT_AUTH_URL=
#VAULT_SWIFT_TENANT=
#VAULT_SWIFT_USERNAME=
#VAULT_SWIFT_PASSWORD=
#VAULT_SWIFT_DOMAIN_ID=
### TEMPAUTH / KEYSTONEV2/ KEYSTONEV3
#VAULT_SWIFT_AUTH_VERSION=TEMPAUTH

#### Uncomment following lines and edit for S3 as snapshot storage
#S3=True
#VAULT_S3_ACCESS_KEY=
#VAULT_S3_SECRET_ACCESS_KEY=
#VAULT_S3_REGION_NAME=
#VAULT_S3_BUCKET=
#### S3 Specific Backend Configurations
#### Uncomment one of the following for the specific S3 type
#Amazon=True
#other_S3_compatible_storage=True

#### Required field(s) for all S3 backends except Amazon
### IP Address:Port Ex: http://192.168.1.33:9000
#VAULT_S3_ENDPOINT_URL=
### True / False
#VAULT_S3_SECURE=


#### Required field(s) for Minio
### Required Signature version s3v4
#VAULT_S3_SIGNATURE_VERSION=s3v4

Once the answersfile has been created can it be used to run the Data Mover installation unattended, which allows the automation of the installation process.

[root@compute]# ./tvault-contego-install.sh –install –file <path to answersfile>

Start the Trilio Data Mover service

At the end of the installation is the Data Mover service deactivated.

To start the service using the installation script run:

[root@compute]# ./tvault-contego-install.sh  --start

Alternatively can the operating system service control commands be used to start the service tvault-contego.service

Install Trilio Horizon Plugin¶

The last component to install is the Trilio Horizon Plugin on the Controllers where the Openstack Horizon service is running. The purpose of the Horizon Plugin is, to provide the native integration of TrlioVault into the GUI of Openstack.

To install the Trilio Horizon Plugin on each Openstack Horizon controller node, follow these steps after logging into a controller node.

[root@controller]# wget https://<ipaddress-of-triliovault-controller>/tvault-horizon-plugin-install.sh --no-check-certificate
[root@controller]# chmod +x tvault-horizon-plugin-install.sh
[root@controller]# ./tvault-horizon-plugin-install.sh

The installation of the Trilio Horizon Plugin will restart the Openstack Horizon service.

Installing on other supported Openstack distributions

Installing Trilio Nova API Extension on Controller Node¶

In case of multiple Controller nodes it is necessary to repeat these steps on all controller nodes running the Nova API service.

To install the Trilio Nova Extension on each Nova controller node, follow these steps after logging into a controller node.

[root@controller]# wget https://<ipaddress-of-triliovault-controller>/tvault-contego-install.sh --no-check-certificate
[root@controller]# chmod +x tvault-contego-install.sh
[root@controller]# ./tvault-contego-install.sh --install

Once the script has been started does it prompt for the following information:

Trilio IP This IP will be used to download the Trilio Nova API Extension pip packages.
Request whether the script is running on a Controller or Compute Node To install the Trilio Nova API Extension choose Controller

Installing Trilio Data Mover on Computer Node

The Trilio Data Mover needs to be installed on all Compute Nodes to guarantee the possible protection of all VMs running inside the Openstack Cluster.

To install the Trilio Data Mover on a Compute node, follow these steps after logging into a compute node.

[root@compute]# wget https://<ipaddress-of-triliovault-controller>/tvault-contego-install.sh --no-check-certificate
[root@compute]# chmod +x tvault-contego-install.sh

At this point it is possible to run the installation either in an automated fashion or in an interactive mode.

Interactive install of the Trilio Data Mover

The following command will start the interactive installation.

[root@compute]# ./tvault-contego-install.sh  --install

Once the script has been started does it prompt for the following information:

Trilio IP This IP will be used to download the Trilio Nova API Extension pip packages.
Request whether the script is running on a Controller or Compute Node To install the Trilio Data Mover service choose Compute Node.
Request of the path to the compute.filters file Choose matching your distribution or provide the exact path.
Acceptance of Trilio changing the sudoers file for nova nova ALL = (root) NOPASSWD: /home/tvault/.virtenv/bin/privsep-helper *
Choose between NFS or S3 as backup target
In case of NFS
1. Provide the NFS Volume path
2. check and change the default NFS options if necessary
In case of S3
1. Choose the S3 profile to be used
2. provide access credentials according to the chosen profile

Automated installation of Trilio Data Mover¶

The Trilio Data Mover can be installed in an unattended way using a configuration file. The configuration file contains all the data that is required while the script is being executed.

The first step for the automated installation is to generate the answers file. A predefined file can be downloaded from any Trilio VM.

[root@compute]# wget https://<ipaddress-of-triliovault-controller>/tvault-contego-install.answers --no-check-certificate

Edit tvault-contego-install.answers file as required for compute nodes. An example can be seen below:

####IP address of Tvault appliance###

IP_ADDRESS=192.168.1.122
TVAULT_CONTEGO_EXT_USER=nova
TVAULT_CONTEGO_EXT_GROUP=nova

#Uncomment following node which you want to configure##
#controller=True
compute=True

#Need to add the following statement in nova sudoers file
#nova ALL = (root) NOPASSWD: /home/tvault/.virtenv/bin/privsep-helper *
#These changes require for Datamover, Otherwise Datamover will not work
# if you are agree then please uncomment the below line   NOVA_SUDOERS=require
#NOVA_SUDOERS=require

#Nova compute.filters file path
#Uncomment following line as per the OS distribution, you can edit the path as per your nova configuration
###For RHEL systems
NOVA_COMPUTE_FILTERS_FILE=/usr/share/nova/rootwrap/compute.filters

###For Debian systems
#NOVA_COMPUTE_FILTERS_FILE=/etc/nova/rootwrap.d/compute.filters

##For any other system
##NOVA_COMPUTE_FILTERS_FILE=


##### Select snapshot storage type #####
#Uncomment following two lines and edit for NFS as snapshot storage, Provide multiple nfs shares separated by comma
NFS=True
NFS_SHARES=192.168.1.33:/mnt/tvault,192.168.1.33:/mnt/tvault_1

#### Uncomment following lines and edit for Swift as snapshot storage
#Swift=False
#VAULT_SWIFT_AUTH_URL=
#VAULT_SWIFT_TENANT=
#VAULT_SWIFT_USERNAME=
#VAULT_SWIFT_PASSWORD=
#VAULT_SWIFT_DOMAIN_ID=
### TEMPAUTH / KEYSTONEV2/ KEYSTONEV3
#VAULT_SWIFT_AUTH_VERSION=TEMPAUTH

#### Uncomment following lines and edit for S3 as snapshot storage
#S3=True
#VAULT_S3_ACCESS_KEY=
#VAULT_S3_SECRET_ACCESS_KEY=
#VAULT_S3_REGION_NAME=
#VAULT_S3_BUCKET=
#### S3 Specific Backend Configurations
#### Uncomment one of the following for the specific S3 type
#Amazon=True
#other_S3_compatible_storage=True

#### Required field(s) for all S3 backends except Amazon
### IP Address:Port Ex: http://192.168.1.33:9000
#VAULT_S3_ENDPOINT_URL=
### True / False
#VAULT_S3_SECURE=


#### Required field(s) for Minio
### Required Signature version s3v4
#VAULT_S3_SIGNATURE_VERSION=s3v4

Once the answersfile has been created can it be used to run the Data Mover installation unattended, which allows the automation of the installation process.

[root@compute]# ./tvault-contego-install.sh –install –file <path to answersfile>

Start the Trilio Data Mover service

At the end of the installation is the Data Mover service deactivated.

To start the service using the installation script run:

[root@compute]# ./tvault-contego-install.sh  --start

Alternatively can the operating system service control commands be used to start the service tvault-contego.service

Install Trilio Horizon Plugin¶

To install the Trilio Horizon Plugin on each Openstack Horizon controller node, follow these steps after logging into a controller node.

[root@controller]# wget https://<ipaddress-of-triliovault-controller>/tvault-horizon-plugin-install.sh --no-check-certificate
[root@controller]# chmod +x tvault-horizon-plugin-install.sh
[root@controller]# ./tvault-horizon-plugin-install.sh

The installation of the Trilio Horizon Plugin will restart the Openstack Horizon service.

Installing on RHOSP10

The Red Hat Openstack Platform Director is the supported and recommended method to deploy and maintain any RHOSP installation.

Trilio is integrating natively into the RHOSP Director. Manual deployment methods are not supported for RHOSP.

Prepare for deployment

Depending whether the RHOSP environment is already installed or is getting installed for the first time different steps are done to be able to deploy Trilio.

Overcloud is not yet deployed

To inject the Trilio yum repository on the overcloud image do the following commands:

##Replace TrilioVault_VM_IP with actual Trilio vm IP
$ cat trilio.repo
[trilio]
name=Trilio Repository
baseurl=http://<TrilioVault_VM_IP>:8085/yum-repo/
enabled=1
gpgcheck=0

$ virt-customize --selinux-relabel -a overcloud-full.qcow2 --upload trilio.repo:/etc/yum.repos.d/
[  0.0] Examining the guest ...
[ 12.0] Setting a random seed
[ 12.0] Copying: opendaylight.repo to /etc/yum.repos.d/
[ 13.0] Finishing off

Afterwards will the overcloud image be created.

virt-customize --selinux-relabel / 
-a overcloud-full.qcow2 / 
--install puppet-triliovault, /
python-workloadmgrclient, /
tvault-contego, /
tvault-contego-api, /
tvault-horizon-plugin

Overcloud is already deployed

If overcloud is deployed already it is necessary to prepare the artifacts to install on the overcloud.

All commands need to be run as user 'stack'

Firstly the github repository needs to be synced with the undercloud.

git clone https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts
git checkout stable/3.0
cd redhat-director-scripts/

Afterwards are those artifacts created and pushed to the overcloud nodes using the upload-swift-artifacts tool. This tool is provided on the undercloud and the prepare_artifacts.sh is created for it.

## Get help of script
(undercloud) [stack@undercloud redhat-director-scripts]$ ./prepare_artifacts.sh --help

Error: Script takes exactly two arguments, 1 provided
    First argument:    undercloud rc file path
    Second argument:    Trilio VM/cluster IP
For Example:
    ./prepare_artifacts.sh /home/stack/stackrc 192.168.122.201

##Run shell script
[root@redhat-undercloud redhat-director-scripts]# ./prepare_artifacts.sh /home/stack/stackrc 192.168.122.201

Update overcloud roles data file to include Trilio services

Trilio contains of multiple services. Add these services to your roles_data.yaml.

In case of uncostomized roles_data.yaml can the default file be found on the undercloud at:

/usr/share/openstack-tripleo-heat-templates/roles_data.yaml

Add the following services to the roles_data.yaml

Trilio Datamover Api Service

Add the following line to the identified role:

'OS::TripleO::Services::TrilioDatamoverApi'

Trilio Horizon Plugin

Add the following line to the identified role:

'OS::TripleO::Services::TrilioHorizonPlugin'

Trilio Datamover Service

Add the following line to the identified role:

'OS::TripleO::Services::TrilioDatamover'

Provide environment details to trilio_env.yaml

Sample trilio_env.yaml

resource_registry:
  OS::TripleO::Services::TrilioDatamover: heat/trilio-datamover.yaml
  OS::TripleO::Services::TrilioDatamoverApi: heat/trilio-datamover-api.yaml
  OS::TripleO::Services::TrilioHorizonPlugin: heat/trilio-horizon-plugin.yaml


parameter_defaults:

   ##Edit following parameters only if you are using non-default locations for nova configuration files,
   ##Otherwise just keep them as it is
   NovaConfFile: '/etc/nova/nova.conf'
   NovaDistConfFile: '/usr/share/nova/nova-dist.conf'
   RedhatOpenstackVersion: '10'

   ##Backup target nfs/s3
   BackupTargetType: 'nfs'
    
   ##For backup target 'nfs'
   NfsShares: ''
   NfsOptions: 'nolock,soft,timeo=180,intr'
   
   ##For backup target 's3'
   #S3 type: amazon_s3/ceph_s3/minio_s3
   S3Type: 'amazon_s3'
   S3AccessKey: ''
   S3SecretKey: ''
   S3RegionName: ''
   S3Bucket: ''
   S3EndpointUrl: ''
   S3SslEnabled: false
   S3SignatureVersion: 's3v4'

   #Horizon directory on controller node, this location will be used to install trilio horizon plugin
   HorizonDir: '/usr/share/openstack-dashboard/'
   EnablePackageInstall: True

Deploy overcloud with trilio environment

Use the following heat environment file and roles data file in overcloud deploy command:

trilio_env.yaml
roles_data.yaml

To include new environment files use '-e' option and for roles data file use '-r' option. An example overcloud deploy command is shown below:

/openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/templates/network-environment.yaml \
-e /home/stack/templates/storage-environment.yaml \
-e ${basedir}/trilio_env.yaml \
-r ${basedir}/roles_data.yaml \
--control-scale 1 --compute-scale 1 --control-flavor control --compute-flavor compute \
--ntp-server 0.north-america.pool.ntp.org --neutron-network-type vxlan --neutron-tunnel-types vxlan \
--validation-errors-fatal --validation-warnings-fatal \
--log-file overcloud_deploy.log

Verify deployment

The following packages should be installed on nodes with the role that contains the nova-api:

# rpm -qa | grep -E 'tvault|workloadmgr'
tvault-contego-api-3.0.44-3.0.noarch

The following packages should be installed nodes with the role that contains the openstack horizon:

# rpm -qa | grep -E 'tvault|workloadmgr'
tvault-horizon-plugin-3.0.44-3.0.noarch
python-workloadmgrclient-3.0.44-3.0.noarch

The following packages should be installed on nodes with the role that contain nova-compute:

# rpm -qa | grep -E 'tvault|trilio'
tvault-contego-3.0.44-3.0.noarch
puppet-triliovault-3.0.44-3.0.noarch

[root@overcloud-compute-0 opt]# systemctl status tvault-contego
● tvault-contego.service - Tvault contego
   Loaded: loaded (/etc/systemd/system/tvault-contego.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2018-07-14 18:08:20 UTC; 1 day 16h ago
 Main PID: 373191 (python)
   CGroup: /system.slice/tvault-contego.service
           └─373191 /home/tvault/.virtenv/bin/python /home/tvault/.virtenv/bin/tvault-contego --config-file=/usr/share/nova/nova-dist.conf --config-file=/e...

Jul 14 18:08:24 overcloud-compute-0 python[373191]: Option "verbose" from group "DEFAULT" is deprecated for removal.  Its value may be silently ign... future.
Jul 14 18:08:24 overcloud-compute-0 python[373191]: Option "rpc_backend" from group "DEFAULT" is deprecated for removal.  Its value may be silently... future.
Jul 14 18:08:24 overcloud-compute-0 sudo[373247]:     nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/umount -l /var/triliovault-mounts/MTkyL...c2hhcmU=
Jul 14 18:08:24 overcloud-compute-0 python[373191]: umount: /var/triliovault-mounts/MTkyLjE2OC4xMjIuMjQwOi9uZnNmaWxlc2hhcmU=: not mounted
Jul 14 18:08:24 overcloud-compute-0 sudo[373249]:     nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/service tvault-object-store stop
Jul 14 18:08:25 overcloud-compute-0 python[373191]: Redirecting to /bin/systemctl stop tvault-object-store.service
Jul 14 18:08:25 overcloud-compute-0 python[373191]: Failed to stop tvault-object-store.service: Unit tvault-object-store.service not loaded.
Jul 14 18:08:25 overcloud-compute-0 sudo[373260]:     nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/mount -o nolock,soft,timeo=180,intr 192...c2hhcmU=
Jul 14 18:08:26 overcloud-compute-0 python[373191]: /home/tvault/.virtenv/lib/python2.7/site-packages/nova/rpc.py:175: FutureWarning: The access_policy arg...
Jul 14 18:08:26 overcloud-compute-0 python[373191]: serializer=serializer)
Hint: Some lines were ellipsized, use -l to show in full.

[root@overcloud-compute-0 opt]# df -h
Filesystem                     Size  Used Avail Use% Mounted on
/dev/vda2                      200G  5.5G  195G   3% /
devtmpfs                        12G     0   12G   0% /dev
tmpfs                           12G     0   12G   0% /dev/shm
tmpfs                           12G  2.1M   12G   1% /run
tmpfs                           12G     0   12G   0% /sys/fs/cgroup
tmpfs                          2.4G     0  2.4G   0% /run/user/981
tmpfs                          2.4G     0  2.4G   0% /run/user/1000
192.168.122.240:/nfsfileshare   20G  1.7G   19G   9% /var/triliovault-mounts/MTkyLjE2OC4xMjIuMjQwOi9uZnNmaWxlc2hhcmU=

Lastly login into the Horizon Dashboard as admin user. Two new tabs should be visible:

Backups
Backups-Admin

If any rpm packages are missing or other verification steps fail verify that the given steps have been followed.

Troubleshooting for overcloud deployment failures

Trilio components will be deployed using puppet scripts.

In case of the overcloud deployment failing does the following command provide the list of errors:

openstack stack failures list overcloud

Further commands that can help identifying any errors.

heat stack-list --show-nested -f "status=FAILED"
heat resource-list --nested-depth 5 overcloud | grep FAILED

Cinder backend is Ceph - additional steps

Add Ceph details to configuration file

If Cinder backend is Ceph it is necessary to manually add the ceph details to tvault-contego.conf on all compute nodes.

The file can be found here: /var/lib/config-data/puppet-generated/triliodm/etc/tvault-contego/tvault-contego.conf

Add the following information:

[libvirt]
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = <rbd_user>

[ceph] 
keyring_ext = .<rbd_user>.keyring

The same block of information can be found in the nova.conf file.

Give nova user read permission

In case that the user nova does not have permission to read and use the ceph conf and keyring files, run the following commands to provide the necessary access:

setfacl -m u:nova:r /etc/ceph/<ceph_keyring_file_path>
setfacl -m u:nova:r /etc/ceph/<ceph_conf_file_path>