Installing on Kolla Train
Change the nova user id on the Trilio Nodes
In Kolla, 'nova' user id on nova-compute docker container is set to '42436'. The 'nova' user id on the Trilio nodes need to be set the same. Do the following steps on all compute nodes:
Download the shell script that will change the user id
Assign executable permissions
Execute the script
Verify that 'nova' user and group id has changed to '42436'
curl -O https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/master/common/nova_userid.sh
chmod +x nova_userid.sh
./nova_userid.sh
id nova
# uid=42436(nova) gid=42436(nova) groups=42436(nova),990(libvirt),36(kvm)
Deploy Trilio Datamover API on all Openstack controller nodes
Trilio Datamover Api container should be deployed on all nodes where nova_api
container is running. In standard deployment, we can call these nodes as OpenStack controller nodes.
Pull TrilioVaullt Datamover API container image
The very first step is to pull the container image from docker.io.
Login to docker and pull the Trilio Datamover API container.
docker login docker.io
## Pull Trilio Datamover Api container using following command.
## You need to edit OS_PLATFORM, TVAULT_VERSION, OPENSTACK_RELEASE_NAME.
## <OS_PLATFORM> ubuntu or centos.
## <OPENSTACK_RELEASE_NAME> train
## <TVAULT_VERSION> 4.0.92 or 4.0.115
docker pull docker.io/trilio/<OS_PLATFORM>-source-trilio-datamover-api:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME>;
Create the Trilio Datamover API configuration
In this part of the process is the configuration file for the Trilio Datamover API created.
The following steps need to be done:
Create config directory
get default config file dmapi.conf
edit dmapi.conf
copy nova.conf to config directory
Create config directory for trilio-datamover-api service
mkdir -p /etc/kolla/trilio-datamover-api
Get the trilio-datamover-api default config file
git clone https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/
git checkout stable/4.0
cp kolla-ansible/trilio-datamover-api/dmapi.conf.sample /etc/kolla/trilio-datamover-api/dmapi.conf
Edit dmapi.conf
The dmapi.conf located in /etc/kolla/trilio-datamover-api/
needs to be edited to adjust to the Openstack environment.
An example dmapi.conf can be seen here:
[DEFAULT]
dmapi_workers = 2
# Get this parameter value from nova-api.conf
transport_url = rabbit://openstack:[email protected]:5672//
##In the following paramater, edit IP address only. Use dmapi/nova-api nodes fixed ip.
dmapi_link_prefix = http://11.11.11.95:8784
dmapi_enabled_ssl_apis =
dmapi_listen_port = 8784
dmapi_enabled_apis = dmapi
bindir = /usr/bin
instance_name_template = instance-%08x
## dmapi_listen should be assigned with nova-api/dmapi node's fixed ip
dmapi_listen = 11.11.11.95
my_ip = 11.11.11.95
rootwrap_config = /etc/dmapi/rootwrap.conf
debug = True
log_file = /var/log/kolla/trilio-datamover-api/dmapi.log
log_dir = /var/log/kolla/trilio-datamover-api
[wsgi]
ssl_cert_file =
ssl_key_file =
api_paste_config = /etc/dmapi/api-paste.ini
[database]
# Get this parameter value from nova-api.conf
connection = mysql+pymysql://nova:[email protected]:3306/nova
[api_database]
# Get this parameter value from nova-api.conf
connection = mysql+pymysql://nova_api:[email protected]:3306/nova_api
[keystone_authtoken]
signing_dir = /var/cache/dmapi
# Get these parameter values from nova-api.conf
cafile =
user_domain_name = Default
insecure = True
auth_uri = http://11.11.11.96:5000
auth_url = http://11.11.11.96:35357
auth_type = password
project_domain_id = default
user_domain_id = default
project_name = service
username = nova
password = XZ2Do9MYEWivxZoiuKKHAc6wOJOPVGHzgXyYu9ic
memcache_security_strategy = ENCRYPT
memcache_secret_key = W8NudaXmPejmCcefr0I0WEcZExFnGrbd6I6ZOkOv
memcached_servers = 11.11.11.95:11211
[oslo_messaging_notifications]
# Get this parameter value from nova-api.conf
transport_url = rabbit://openstack:[email protected]:5672//
driver = noop
[oslo_middleware]
enable_proxy_headers_parsing = true
Copy nova.conf of nova-api service to trilio-datamover-api directory
cp /etc/kolla/nova-api/nova.conf /etc/kolla/trilio-datamover-api/
chmod -R 744 /etc/kolla/trilio-datamover-api/
Create trilio-datamover-api log directory
For CentOS, we need nova user ownership on datamover api log directory where as for Ubuntu, we need dmapi user's ownership on datmover api log directory.
mkdir -p /var/log/kolla/trilio-datamover-api
1. If openstack is based on CentOS :
##Change ownership of log directory to 42436:42436 - nova user and group id on container
chmod 755 /var/log/kolla/trilio-datamover-api
chown 42436:42436 /var/log/kolla/trilio-datamover-api
2. If openstack is based on Ubuntu :
##Change ownership of log directory to 42486:42487- dmapi user and group id on container
chmod 755 /var/log/kolla/trilio-datamover-api
chown 42486:42487 /var/log/kolla/trilio-datamover-api
Add trilio-datamover-api to haproxy.cfg
## Edit haproxy.cfg
vi /etc/kolla/haproxy/haproxy.cfg
## If SSL enabled on public interface of dmapi
listen trilio_datamover_api
bind <Keystone_virtual_ip>:8784 ssl crt /etc/haproxy/haproxy.pem
server <controller_hostname_1> <controller_IP1>:8784 check inter 2000 rise 2 fall 5
server <controller_hostname_2> <controller_IP2>:8784 check inter 2000 rise 2 fall 5
server <controller_hostname_3> <controller_IP3>:8784 check inter 2000 rise 2 fall 5
## If SSL is not enabled on any interface
listen trilio_datamover_api
bind <Keystone_virtual_ip>:8784
server <controller_hostname_1> <controller_IP1>:8784 check inter 2000 rise 2 fall 5
server <controller_hostname_2> <controller_IP2>:8784 check inter 2000 rise 2 fall 5
server <controller_hostname_3> <controller_IP3>:8784 check inter 2000 rise 2 fall 5
## Restart haproxy container
docker restart haproxy
Run trilio-datamover-api container
Now the trilio-datamover-api container can be deployed and started.
## You need to edit OS_PLATFORM, TVAULT_VERSION, OPENSTACK_RELEASE_NAME.
## <OS_PLATFORM> ubuntu or centos.
## <OPENSTACK_RELEASE_NAME> train
## <TVAULT_VERSION> : 4.0.92
docker run --network host --name trilio_datamover_api -d --restart always -v /etc/kolla/trilio-datamover-api/nova.conf:/etc/nova/nova.conf \
-v /etc/kolla/trilio-datamover-api/dmapi.conf:/etc/dmapi/dmapi.conf \
-v /var/log/kolla/:/var/log/kolla/ \
trilio/<OS_PLATFORM>-source-trilio-datamover-api:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME> /var/lib/kolla/venv/bin/python /usr/bin/dmapi-api
Verify deployment of trilio-datamover-api container
To verify the deployment was successful check the container status using docker ps
.
root@trilio-Standard-PC-i440FX-PIIX-1996:~# docker ps | grep trilio_datamover_api
736e4ac462c9 trilio/ubuntu-source-trilio-datamover-api:4.0.92 "/var/lib/kolla/venv/" 3 days ago Up 23 hours trilio_datamover_api
Deploy Trilio Datamover container on all Compute nodes
Trilio Datamover container should be deployed on all nodes where nova_compute
container is running. In standard deployment, we can call these nodes as openstack compute nodes.
Pull Trilio Datamover container
The very first step is to pull the container image from docker.io.
Login to docker and pull the Trilio Datamover API container.
docker login docker.io
## Pull Trilio Datamover container using following command.
## You need to edit OS_PLATFORM, TVAULT_VERSION, OPENSTACK_RELEASE_NAME.
## <OS_PLATFORM> ubuntu or centos.
## <OPENSTACK_RELEASE_NAME> train
## <TVAULT_VERSION> 4.0.92 or 4.0.115
docker pull docker.io/trilio/<OS_PLATFORM>-source-trilio-datamover:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME>;
Create the Trilio Datamover configuration
In this part of the process is the configuration file for the Trilio Datamover created.
The following steps need to be done:
Create config directory
copy nova.conf to config directory
get default config file tvault-contego.conf
edit tvault.conf
Create service config directory for trilio-datamover service
mkdir -p /etc/kolla/trilio-datamover
Copy nova.conf of nova-compute service to trilio-datamover config directory
cp /etc/kolla/nova-compute/nova.conf /etc/kolla/trilio-datamover/
Get the trilio-datamover default config file
## Clone triliovault devops repository. It's public github repository.
git clone https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/
git checkout <GITHUB_BRANCH> // Check section: 1.i) plan for deployment for correct github branch name
## If backup storage type you want to use is 'NFS'
cp kolla-ansible/trilio-datamover/tvault-contego.conf.nfs.sample /etc/kolla/trilio-datamover/tvault-contego.conf
## If backup storage type you want to use is 'S3'
cp kolla-ansible/trilio-datamover/tvault-contego.conf.s3.sample /etc/kolla/trilio-datamover/tvault-contego.conf
## Set correct permissions
chmod -R 744 /etc/kolla/trilio-datamover/
Edit tvault-contego.conf
Edit /etc/kolla/trilio-datamover/tvault-contego.conf
config file to provide NFS/S3 details as per backup storage selected.
[DEFAULT]
vault_storage_type = nfs
vault_storage_nfs_export = 192.168.1.34:/mnt/tvault/tvm
vault_data_directory_old = /var/triliovault
vault_data_directory = /var/triliovault-mounts
log_file = /var/log/kolla/trilio-datamover/tvault-contego.log
debug = False
verbose = True
max_uploads_pending = 3
max_commit_pending = 3
vault_s3_auth_version = DEFAULT
vault_s3_access_key_id =
vault_s3_secret_access_key =
vault_s3_region_name = us-east-2
vault_s3_bucket =
qemu_agent_ping_timeout = 900
[contego_sys_admin]
helper_command = sudo /var/lib/kolla/venv/bin/privsep-helper
[conductor]
use_local = True
Create trilio-datamover log directory
mkdir -p /var/log/kolla/trilio-datamover
##Change ownership of log directory to 42436:42436 - nova user and group id on container
chmod 755 /var/log/kolla/trilio-datamover
chown 42436:42436 /var/log/kolla/trilio-datamover
If Ceph is used for Nova/Cinder Storage
If ceph is getting used for cinder/nova, the correct permissions for ceph.conf and keyrings files need to be assigned. The trilio_datamover container will be using ceph.conf and keyring files with the 'nova' user.
chmod o+rx /etc/ceph
chmod o+r /etc/ceph/*
If nova/cinder backend is ceph, you need to add ceph user and keyring details to /etc/kolla/trilio-datamover/tvault-contego.conf file. Add the following sections to the tvault-contego.conf file. In the provided example is ceph's 'cinder' user configured to use for trilio read/write operations.
[libvirt]
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = cinder
[ceph]
keyring_ext = .cinder.keyring
Mount /etc/ceph on trilio_datamover container in read only mode. Check docker run command provided in the next step. The ceph user (example -'cinder') should have read and write permissions on ceph pool used for nova/cinder backend. Verify nova user (uid - 42436) on trilio_datamover container is able to read ceph user's keyring file and ceph.conf after mounting /etc/ceph on the container. Set appropriate permissions for /etc/ceph/ files on the host itself.
Run trilio-datamover container
If the cloud does not use 'ceph' storage for nova/cinder, remove '/etc/ceph' volume mount option from below commands.
## In following docker run command you need to edit OS_PLATFORM, TVAULT_VERSION,vOPENSTACK_RELEASE_NAME.
## <OS_PLATFORM> ubuntu or centos.
## <OPENSTACK_RELEASE_NAME> train
## <TVAULT_VERSION> : 4.0.92 or 4.0.115
##If backup storage type is 'NFS'
docker run --privileged --network host --name trilio_datamover -d \
--restart always \
-v /etc/kolla/trilio-datamover/tvault-contego.conf:/etc/tvault-contego/tvault-contego.conf \
-v /etc/kolla/trilio-datamover/nova.conf:/etc/nova/nova.conf \
-v /dev:/dev:rw \
-v /etc/ceph:/etc/ceph:ro \
-v nova_compute:/var/lib/nova/:rw \
-v /var/log/kolla/:/var/log/kolla/ \
-v iscsi_info:/etc/iscsi:rw -v /var/run/libvirt \
trilio/<OS_PLATFORM>-source-trilio-datamover:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME> /opt/tvault/start_datamover_nfs
##If backup storage type is 'S3'
docker run --privileged --network host --name trilio_datamover -d \
--restart always \
-v /etc/kolla/trilio-datamover/tvault-contego.conf:/etc/tvault-contego/tvault-contego.conf \
-v /etc/kolla/trilio-datamover/nova.conf:/etc/nova/nova.conf \
-v /dev:/dev:rw \
-v /etc/ceph:/etc/ceph:ro \
-v nova_compute:/var/lib/nova/:rw \
-v /var/log/kolla/:/var/log/kolla/ \
-v iscsi_info:/etc/iscsi:rw -v /var/run/libvirt \
trilio/<OS_PLATFORM>-source-trilio-datamover:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME> /opt/tvault/start_datamover_s3
Verify deployment of trilio-datamover container
To verify the deployment was successful check the container status using docker ps
.
root@trilio-Standard-PC-i440FX-PIIX-1996:~# docker ps | grep trilio_datamover
9d572974c75d trilio/centos-source-trilio-datamover:4.0.92 "/opt/tvault/start_da" 2 days ago Up 23 hours trilio_datamover
Installing Trilio Horizon plugin
Trilio Horizon plugin needs to be installed inside the OpenStack horizon container. Once installed, the Trilio dashboard will be visible in OpenStack Horizon.
The following steps need to be done:
Download installation shell script
Run the shell script
Edit Horizon settings
Restart the Horizon container
Download the installation shell script
To download the shell script directly into the Horizon container do:
docker exec -it horizon /bin/bash
curl -kO https://<TVAULT_VM_IP>/tvault-horizon-plugin-install.sh
chmod +x tvault-horizon-plugin-install.sh
Run the shell script and restart Horizon container
Run the shell script and restart horizon container. This will restart apache service, which may enforce a log out of the container.
# Login to horizon container if not already.
docker exec -itu root horizon /bin/bash
## Run the trilio horizon plugin shell script
- Script will ask for python2/python3, you need to select the option as per your enviornment
- Trilio horizon install script will ask for horizon's openstack_dashboard directory path if it's not at the default
location - '/usr/shar/openstack-dashboard' For train ubuntu bionic, it's : '/var/lib/kolla/venv/lib/python2.7/site-packages'
./tvault-horizon-plugin-install.sh
## Exit the horizon container
Ctrl + D
Edit Horizon settings
The following line needs to be aded in 'local_settings' of Openstack's Horizon file to enable workloadmanager quota feature in the Horizon dashboard.
## vi /etc/kolla/horizon/local_settings
Add this line : HORIZON_CONFIG['customization_module'] = 'dashboards.overrides'
Restart the Horizon container
To enable the done changes restart the Horizon container
## Restart the horizon container
docker restart horizon
Known issues for Horizon plugin installation
If OpenStack is based on 'Centos' platform
Grafana yum repository has an issue on the latest horizon containers of OpenStack (not Trilio). To confirm the issue, you can just run yum repolist
, it will fail. Use the following command to disable the grafana repository.
yum --disablerepo=grafana
If Openstack is based on Ubuntu platform
Trilio horizon install script will ask for horizon's openstack_dashboard directory path if it's not at the default location - /usr/shar/openstack-dashboard
For train ubuntu bionic, it's : /var/lib/kolla/venv/lib/python2.7/site-packages
If Trilio Horizon tabs are not accessible but Openstack Horizon works
If Trilio Horizon tabs are not accessible but OpenStack Horizon is working fine, make sure that endpoints for service 'TrilioVaultWLM' are created correctly. The root cause of this issue is typically, that SSL is enabled on all three endpoint types of 'TrilioVaultWLM' service.
If SSL is enable only on public 'keystone' service endpoints, then create 'TrilioVaultWLM' service endpoints in the same fashion. Endpoints for service 'TrilioVaultWLM' get created during Trilio configuration step. If these endpoints need to be edited reconfigure the Trilio.
Enabling Snapshout Mount
To make 'snapshot mount' functionality work, the cloud administrator needs to complete the following steps.
Identify backup target mount point on Trilio VM
install nfs-common on nova_compute and nova_libvirt containers
Mount backup target nfs share on nova_compute and nova_libvirt containers
Identify backup target mount point in Trilio VM
The following command will provide the active mountpoint on the Trilio VM
[stack@tvm ~]$ mount | grep triliovault
192.168.1.33:/mnt/tvault on /var/triliovault-mounts/MTkyLjE2OC4xLjMzOi9tbnQvdHZhdWx0 type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=180,retrans=2,sec=sys,clientaddr=192.168.10.10,lookupcache=none,local_lock=none,addr=192.168.1.33)
This example gives the following information:
Backup target is NFS share: 192.168.1.33:/mnt/tvault
Mountpoint is: /var/triliovault-mounts/MTkyLjE2OC4xLjMzOi9tbnQvdHZhdWx0
Install nfs-common on nova_compute and nova_libvirt containers
It is necessary to install nfs-common
package on both nova_compute
and nova_libvirt
containers.
# Login to nova_compute container with root user
docker exec -itu root nova_compute /bin/bash
# Install nfs-utils/nfs-common package using
=> If it's centos
yum install nfs-utils
=> If it's ubuntu
apt-get install nfs-common
# Login to nova_libvirt container with root user
docker exec -itu root nova_libvirt /bin/bash
# Install nfs-utils/nfs-common package using yum
=> If it's centos
yum install nfs-utils
=> If it's ubuntu
apt-get install nfs-common
Mount backup target nfs share on nova_compute and nova_libvirt containers
Mount the backup target nfs share on 'nova_compute' and 'nova_libvirt' container at exactly same mount point as done on triliovault VM.
mount -t nfs 192.168.1.33:/mnt/tvault /var/triliovault-mounts/MTkyLjE2OC4xLjMzOi9tbnQvdHZhdWx0
Troubleshoot installation
If any triliovault container is stuck in restarting state the following logs can be checked.
# Get docker run logs for datamover container
docker logs trilio_datamover
## Get docker logs for datamover api container
docker logs trilio_datamover_api
Possible issues for trilio-datamover container failure are for example NFS mount issues or S3 credentials might be wrong. If it's Amazon S3, then network connectivity between compute node and AWS s3 is needed. The docker logs will clearly tell the exact error.
If the above logs do not help OR If containers running well but, backups fail, following service logs will help:
/var/log/kolla/trilio-datamover/tvault-contego.log
/var/log/kolla/trilio-datamover-api/dmapi.log
If the Trilio Horizon tabs are not visible on Openstack, verify the following:
Make sure trilio horizon plugin is installed on OpenStack horizon container
Trilio configuration step needs to be completed to see the triliovault dashboard on OpenStack
Make sure correct openstack_dashboard directory got provided and the triliovault horizon plugin files got successfully copied there.
Last updated
Was this helpful?