Installing on Kolla Train

Change the nova user id on the Trilio Nodes

In Kolla, 'nova' user id on nova-compute docker container is set to '42436'. The 'nova' user id on the Trilio nodes need to be set the same. Do the following steps on all compute nodes:

1. Download the shell script that will change the user id

2. Assign executable permissions

3. Execute the script

4. Verify that 'nova' user and group id has changed to '42436'

curl -O https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/master/common/nova_userid.sh 
chmod +x nova_userid.sh
./nova_userid.sh
id nova
# uid=42436(nova) gid=42436(nova) groups=42436(nova),990(libvirt),36(kvm)

Deploy Trilio Datamover API on all Openstack controller nodes

Trilio Datamover Api container should be deployed on all nodes where nova_api container is running. In standard deployment, we can call these nodes as OpenStack controller nodes.

Pull TrilioVaullt Datamover API container image

The very first step is to pull the container image from docker.io.

Login to docker using credentials: triliodocker/triliopassword

Login to docker and pull the Trilio Datamover API container.

docker login docker.io

## Pull Trilio Datamover Api container using following command.
## You need to edit OS_PLATFORM, TVAULT_VERSION, OPENSTACK_RELEASE_NAME.
## <OS_PLATFORM> ubuntu or centos.
## <OPENSTACK_RELEASE_NAME> train
## <TVAULT_VERSION> 4.0.92 or 4.0.115

docker pull docker.io/trilio/<OS_PLATFORM>-source-trilio-datamover-api:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME>;

Example command for train openstack on ubuntu platform with triliovault 4.0 GA release: docker pull docker.io/trilio/ubuntu-source-trilio-datamover-api:4.0.92-train

Create the Trilio Datamover API configuration

In this part of the process is the configuration file for the Trilio Datamover API created.

The following steps need to be done:

1. Create config directory

2. get default config file dmapi.conf

3. edit dmapi.conf

4. copy nova.conf to config directory

Create config directory for trilio-datamover-api service

mkdir -p /etc/kolla/trilio-datamover-api

Get the trilio-datamover-api default config file

git clone https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/
git checkout stable/4.0 

cp kolla-ansible/trilio-datamover-api/dmapi.conf.sample /etc/kolla/trilio-datamover-api/dmapi.conf

Edit dmapi.conf

The dmapi.conf located in /etc/kolla/trilio-datamover-api/ needs to be edited to adjust to the Openstack environment.

Nearly all required values can be copied from the nova.conf located at: /etc/kolla/nova-api/

Follow comments inside the dmapi.conf to learn which parameters are the minimum needed.

An example dmapi.conf can be seen here:

[DEFAULT]
dmapi_workers = 2
# Get this parameter value from nova-api.conf
transport_url = rabbit://openstack:aklGvwxJ2GE9nnp1ivEOBPTCfwXev6TNGj2RIUHc@11.11.11.95:5672//

##In the following paramater, edit IP address only. Use dmapi/nova-api nodes fixed ip.
dmapi_link_prefix = http://11.11.11.95:8784
dmapi_enabled_ssl_apis =
dmapi_listen_port = 8784
dmapi_enabled_apis = dmapi
bindir = /usr/bin
instance_name_template = instance-%08x
## dmapi_listen should be assigned with nova-api/dmapi node's fixed ip
dmapi_listen = 11.11.11.95
my_ip = 11.11.11.95
rootwrap_config = /etc/dmapi/rootwrap.conf
debug = True
log_file = /var/log/kolla/trilio-datamover-api/dmapi.log
log_dir = /var/log/kolla/trilio-datamover-api

[wsgi]
ssl_cert_file =
ssl_key_file =
api_paste_config = /etc/dmapi/api-paste.ini

[database]
# Get this parameter value from nova-api.conf
connection = mysql+pymysql://nova:IJP6ryxvAvQDZdNxosJr2OKDRqEofTN4Zbet638U@11.11.11.96:3306/nova

[api_database]
# Get this parameter value from nova-api.conf
connection = mysql+pymysql://nova_api:YoHe1umK1pso0xINCmcmvmK7IkTLPT9QBn3aMduF@11.11.11.96:3306/nova_api

[keystone_authtoken]
signing_dir = /var/cache/dmapi
# Get these parameter values from nova-api.conf
cafile =
user_domain_name = Default
insecure = True
auth_uri = http://11.11.11.96:5000
auth_url = http://11.11.11.96:35357
auth_type = password
project_domain_id = default
user_domain_id = default
project_name = service
username = nova
password = XZ2Do9MYEWivxZoiuKKHAc6wOJOPVGHzgXyYu9ic
memcache_security_strategy = ENCRYPT
memcache_secret_key = W8NudaXmPejmCcefr0I0WEcZExFnGrbd6I6ZOkOv
memcached_servers = 11.11.11.95:11211

[oslo_messaging_notifications]
# Get this parameter value from nova-api.conf
transport_url = rabbit://openstack:aklGvwxJ2GE9nnp1ivEOBPTCfwXev6TNGj2RIUHc@11.11.11.95:5672//
driver = noop

[oslo_middleware]
enable_proxy_headers_parsing = true

Copy nova.conf of nova-api service to trilio-datamover-api directory

cp /etc/kolla/nova-api/nova.conf /etc/kolla/trilio-datamover-api/
chmod -R 744 /etc/kolla/trilio-datamover-api/

Create trilio-datamover-api log directory

For CentOS, we need nova user ownership on datamover api log directory where as for Ubuntu, we need dmapi user's ownership on datmover api log directory.

mkdir -p /var/log/kolla/trilio-datamover-api

1. If openstack is based on CentOS :
##Change ownership of log directory to 42436:42436 - nova user and group id on container
chmod 755 /var/log/kolla/trilio-datamover-api
chown 42436:42436 /var/log/kolla/trilio-datamover-api

2. If openstack is based on Ubuntu :
##Change ownership of log directory to 42486:42487- dmapi user and group id on container 
chmod 755 /var/log/kolla/trilio-datamover-api 
chown 42486:42487 /var/log/kolla/trilio-datamover-api

Add trilio-datamover-api to haproxy.cfg

## Edit haproxy.cfg 
vi /etc/kolla/haproxy/haproxy.cfg

## If SSL enabled on public interface of dmapi
listen trilio_datamover_api
  bind <Keystone_virtual_ip>:8784  ssl crt /etc/haproxy/haproxy.pem
  server <controller_hostname_1> <controller_IP1>:8784 check inter 2000 rise 2 fall 5
  server <controller_hostname_2> <controller_IP2>:8784 check inter 2000 rise 2 fall 5
  server <controller_hostname_3> <controller_IP3>:8784 check inter 2000 rise 2 fall 5

## If SSL is not enabled on any interface
listen trilio_datamover_api
  bind <Keystone_virtual_ip>:8784
  server <controller_hostname_1> <controller_IP1>:8784 check inter 2000 rise 2 fall 5
  server <controller_hostname_2> <controller_IP2>:8784 check inter 2000 rise 2 fall 5 
  server <controller_hostname_3> <controller_IP3>:8784 check inter 2000 rise 2 fall 5

## Restart haproxy container
docker restart haproxy

Run trilio-datamover-api container

Now the trilio-datamover-api container can be deployed and started.

## You need to edit OS_PLATFORM, TVAULT_VERSION, OPENSTACK_RELEASE_NAME. 
## <OS_PLATFORM> ubuntu or centos.
## <OPENSTACK_RELEASE_NAME> train
## <TVAULT_VERSION> : 4.0.92

docker run --network host --name trilio_datamover_api -d --restart always -v /etc/kolla/trilio-datamover-api/nova.conf:/etc/nova/nova.conf \
-v /etc/kolla/trilio-datamover-api/dmapi.conf:/etc/dmapi/dmapi.conf \
-v /var/log/kolla/:/var/log/kolla/ \
trilio/<OS_PLATFORM>-source-trilio-datamover-api:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME> /var/lib/kolla/venv/bin/python /usr/bin/dmapi-api

Verify deployment of trilio-datamover-api container

To verify the deployment was successful check the container status using docker ps.

root@trilio-Standard-PC-i440FX-PIIX-1996:~# docker ps | grep trilio_datamover_api
736e4ac462c9  trilio/ubuntu-source-trilio-datamover-api:4.0.92  "/var/lib/kolla/venv/"  3 days ago  Up 23 hours  trilio_datamover_api

Deploy Trilio Datamover container on all Compute nodes

Trilio Datamover container should be deployed on all nodes where nova_compute container is running. In standard deployment, we can call these nodes as openstack compute nodes.

At this stage it is necessary to know if the deployment shall use NFS or S3 protocol for the backup target.

Pull Trilio Datamover container

The very first step is to pull the container image from docker.io.

Login to docker using credentials: triliodocker/triliopassword

Login to docker and pull the Trilio Datamover API container.

docker login docker.io

## Pull Trilio Datamover container using following command.
## You need to edit OS_PLATFORM, TVAULT_VERSION, OPENSTACK_RELEASE_NAME.
## <OS_PLATFORM> ubuntu or centos.
## <OPENSTACK_RELEASE_NAME> train
## <TVAULT_VERSION> 4.0.92 or 4.0.115

docker pull docker.io/trilio/<OS_PLATFORM>-source-trilio-datamover:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME>;

Example command for train openstack on ubuntu platform with Trilio 4.0 GA release: docker pull docker.io/trilio/ubuntu-source-trilio-datamover:4.0.92-train

Create the Trilio Datamover configuration

In this part of the process is the configuration file for the Trilio Datamover created.

The following steps need to be done:

1. Create config directory

2. copy nova.conf to config directory

3. get default config file tvault-contego.conf

4. edit tvault.conf

Create service config directory for trilio-datamover service

mkdir -p /etc/kolla/trilio-datamover

Copy nova.conf of nova-compute service to trilio-datamover config directory

cp /etc/kolla/nova-compute/nova.conf /etc/kolla/trilio-datamover/

Get the trilio-datamover default config file

## Clone triliovault devops repository. It's public github repository.
git clone https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/ 
git checkout <GITHUB_BRANCH>          // Check section: 1.i) plan for deployment for correct github branch name

## If backup storage type you want to use is 'NFS'
cp kolla-ansible/trilio-datamover/tvault-contego.conf.nfs.sample /etc/kolla/trilio-datamover/tvault-contego.conf

## If backup storage type you want to use is 'S3'
cp kolla-ansible/trilio-datamover/tvault-contego.conf.s3.sample /etc/kolla/trilio-datamover/tvault-contego.conf

## Set correct permissions
chmod -R 744 /etc/kolla/trilio-datamover/

Edit tvault-contego.conf

Edit /etc/kolla/trilio-datamover/tvault-contego.conf config file to provide NFS/S3 details as per backup storage selected.

In case of NFS backup target, only nfs share details need to provided. No other conf parameters need to edit, unless you know the details of it.

[DEFAULT]
vault_storage_type = nfs
vault_storage_nfs_export = 192.168.1.34:/mnt/tvault/tvm
vault_data_directory_old = /var/triliovault
vault_data_directory = /var/triliovault-mounts
log_file = /var/log/kolla/trilio-datamover/tvault-contego.log
debug = False
verbose = True
max_uploads_pending = 3
max_commit_pending = 3
vault_s3_auth_version = DEFAULT
vault_s3_access_key_id =
vault_s3_secret_access_key =
vault_s3_region_name = us-east-2
vault_s3_bucket =
qemu_agent_ping_timeout = 900

[contego_sys_admin]
helper_command = sudo /var/lib/kolla/venv/bin/privsep-helper

[conductor]
use_local = True

Create trilio-datamover log directory

mkdir -p /var/log/kolla/trilio-datamover

##Change ownership of log directory to 42436:42436 - nova user and group id on container
chmod 755 /var/log/kolla/trilio-datamover
chown 42436:42436 /var/log/kolla/trilio-datamover

If Ceph is used for Nova/Cinder Storage

If ceph is getting used for cinder/nova, the correct permissions for ceph.conf and keyrings files need to be assigned. The trilio_datamover container will be using ceph.conf and keyring files with the 'nova' user.

chmod o+rx /etc/ceph
chmod o+r /etc/ceph/*

If nova/cinder backend is ceph, you need to add ceph user and keyring details to /etc/kolla/trilio-datamover/tvault-contego.conf file. Add the following sections to the tvault-contego.conf file. In the provided example is ceph's 'cinder' user configured to use for trilio read/write operations.

[libvirt]
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = cinder

[ceph]
keyring_ext = .cinder.keyring

Mount /etc/ceph on trilio_datamover container in read only mode. Check docker run command provided in the next step. The ceph user (example -'cinder') should have read and write permissions on ceph pool used for nova/cinder backend. Verify nova user (uid - 42436) on trilio_datamover container is able to read ceph user's keyring file and ceph.conf after mounting /etc/ceph on the container. Set appropriate permissions for /etc/ceph/ files on the host itself.

Run trilio-datamover container

If the cloud does not use 'ceph' storage for nova/cinder, remove '/etc/ceph' volume mount option from below commands.

## In following docker run command you need to edit OS_PLATFORM, TVAULT_VERSION,vOPENSTACK_RELEASE_NAME.

## <OS_PLATFORM> ubuntu or centos.
## <OPENSTACK_RELEASE_NAME> train
## <TVAULT_VERSION> : 4.0.92 or 4.0.115

##If backup storage type is 'NFS'
docker run --privileged --network host --name trilio_datamover -d \
--restart always \
-v /etc/kolla/trilio-datamover/tvault-contego.conf:/etc/tvault-contego/tvault-contego.conf \
-v /etc/kolla/trilio-datamover/nova.conf:/etc/nova/nova.conf \
-v /dev:/dev:rw \
-v /etc/ceph:/etc/ceph:ro \
-v nova_compute:/var/lib/nova/:rw \
-v /var/log/kolla/:/var/log/kolla/ \
-v iscsi_info:/etc/iscsi:rw -v /var/run/libvirt \
trilio/<OS_PLATFORM>-source-trilio-datamover:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME> /opt/tvault/start_datamover_nfs

##If backup storage type is 'S3'
docker run --privileged --network host --name trilio_datamover -d \
--restart always \
-v /etc/kolla/trilio-datamover/tvault-contego.conf:/etc/tvault-contego/tvault-contego.conf \
-v /etc/kolla/trilio-datamover/nova.conf:/etc/nova/nova.conf \
-v /dev:/dev:rw \
-v /etc/ceph:/etc/ceph:ro \
-v nova_compute:/var/lib/nova/:rw \
-v /var/log/kolla/:/var/log/kolla/ \
-v iscsi_info:/etc/iscsi:rw -v /var/run/libvirt \
trilio/<OS_PLATFORM>-source-trilio-datamover:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME> /opt/tvault/start_datamover_s3

Verify deployment of trilio-datamover container

To verify the deployment was successful check the container status using docker ps.

root@trilio-Standard-PC-i440FX-PIIX-1996:~# docker ps | grep trilio_datamover
9d572974c75d  trilio/centos-source-trilio-datamover:4.0.92  "/opt/tvault/start_da"  2 days ago  Up 23 hours  trilio_datamover

Installing Trilio Horizon plugin

Trilio Horizon plugin needs to be installed inside the OpenStack horizon container. Once installed, the Trilio dashboard will be visible in OpenStack Horizon.

The following steps need to be done:

1. Download installation shell script

2. Run the shell script

3. Edit Horizon settings

4. Restart the Horizon container

Download the installation shell script

To download the shell script directly into the Horizon container do:

docker exec -it horizon /bin/bash
curl -kO https://<TVAULT_VM_IP>/tvault-horizon-plugin-install.sh
chmod +x tvault-horizon-plugin-install.sh

Run the shell script and restart Horizon container

You have to run the script inside the Horizon container as root.

Run the shell script and restart horizon container. This will restart apache service, which may enforce a log out of the container.

# Login to horizon container if not already.
docker exec -itu root horizon /bin/bash

## Run the trilio horizon plugin shell script 
- Script will ask for python2/python3, you need to select the option as per your enviornment
- Trilio horizon install script will ask for horizon's openstack_dashboard directory path if it's not at the default
 location - '/usr/shar/openstack-dashboard' For train ubuntu bionic, it's : '/var/lib/kolla/venv/lib/python2.7/site-packages'

./tvault-horizon-plugin-install.sh

## Exit the horizon container
Ctrl + D

Edit Horizon settings

The following line needs to be aded in 'local_settings' of Openstack's Horizon file to enable workloadmanager quota feature in the Horizon dashboard.

## vi /etc/kolla/horizon/local_settings

Add this line : HORIZON_CONFIG['customization_module'] = 'dashboards.overrides'

Restart the Horizon container

To enable the done changes restart the Horizon container

## Restart the horizon container
docker restart horizon

Known issues for Horizon plugin installation

If OpenStack is based on 'Centos' platform

This issue has not been observed in all CentOS based Kolla Train installations. Please verify before disabling grafana repository.

Grafana yum repository has an issue on the latest horizon containers of OpenStack (not Trilio). To confirm the issue, you can just run yum repolist, it will fail. Use the following command to disable the grafana repository.

yum --disablerepo=grafana

If Openstack is based on Ubuntu platform

Trilio horizon install script will ask for horizon's openstack_dashboard directory path if it's not at the default location - /usr/shar/openstack-dashboard

For train ubuntu bionic, it's : /var/lib/kolla/venv/lib/python2.7/site-packages

If Trilio Horizon tabs are not accessible but Openstack Horizon works

If Trilio Horizon tabs are not accessible but OpenStack Horizon is working fine, make sure that endpoints for service 'TrilioVaultWLM' are created correctly. The root cause of this issue is typically, that SSL is enabled on all three endpoint types of 'TrilioVaultWLM' service.

If SSL is enable only on public 'keystone' service endpoints, then create 'TrilioVaultWLM' service endpoints in the same fashion. Endpoints for service 'TrilioVaultWLM' get created during Trilio configuration step. If these endpoints need to be edited reconfigure the Trilio.

Enabling Snapshout Mount

To make 'snapshot mount' functionality work, the cloud administrator needs to complete the following steps.

1. Identify backup target mount point on Trilio VM

2. install nfs-common on nova_compute and nova_libvirt containers

3. Mount backup target nfs share on nova_compute and nova_libvirt containers

Identify backup target mount point in Trilio VM

The following command will provide the active mountpoint on the Trilio VM

[stack@tvm ~]$ mount | grep triliovault
192.168.1.33:/mnt/tvault on /var/triliovault-mounts/MTkyLjE2OC4xLjMzOi9tbnQvdHZhdWx0 type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=180,retrans=2,sec=sys,clientaddr=192.168.10.10,lookupcache=none,local_lock=none,addr=192.168.1.33)

This example gives the following information:

Backup target is NFS share: 192.168.1.33:/mnt/tvault Mountpoint is: /var/triliovault-mounts/MTkyLjE2OC4xLjMzOi9tbnQvdHZhdWx0

Install nfs-common on nova_compute and nova_libvirt containers

It is necessary to install nfs-common package on both nova_compute and nova_libvirt containers.

# Login to nova_compute container with root user
docker exec -itu root nova_compute /bin/bash

# Install nfs-utils/nfs-common package using
=> If it's centos
yum install nfs-utils
=> If it's ubuntu
apt-get install nfs-common

# Login to nova_libvirt container with root user
docker exec -itu root nova_libvirt /bin/bash

# Install nfs-utils/nfs-common package using yum
=> If it's centos 
yum install nfs-utils 
=> If it's ubuntu
apt-get install nfs-common

Mount backup target nfs share on nova_compute and nova_libvirt containers

Mount the backup target nfs share on 'nova_compute' and 'nova_libvirt' container at exactly same mount point as done on triliovault VM.

Create the mountpoint directory as necessary.

mount -t nfs 192.168.1.33:/mnt/tvault /var/triliovault-mounts/MTkyLjE2OC4xLjMzOi9tbnQvdHZhdWx0

Troubleshoot installation

If any triliovault container is stuck in restarting state the following logs can be checked.

# Get docker run logs for datamover container
docker logs trilio_datamover

## Get docker logs for datamover api container
docker logs trilio_datamover_api

Possible issues for trilio-datamover container failure are for example NFS mount issues or S3 credentials might be wrong. If it's Amazon S3, then network connectivity between compute node and AWS s3 is needed. The docker logs will clearly tell the exact error.

If the above logs do not help OR If containers running well but, backups fail, following service logs will help:

/var/log/kolla/trilio-datamover/tvault-contego.log
/var/log/kolla/trilio-datamover-api/dmapi.log

If the Trilio Horizon tabs are not visible on Openstack, verify the following:

• Make sure trilio horizon plugin is installed on OpenStack horizon container

• Trilio configuration step needs to be completed to see the triliovault dashboard on OpenStack

• Make sure correct openstack_dashboard directory got provided and the triliovault horizon plugin files got successfully copied there.