In Kolla, 'nova' user id on nova-compute docker container is set to '42436'. The 'nova' user id on the Trilio nodes need to be set the same. Do the following steps on all compute nodes:
Download the shell script that will change the user id
Assign executable permissions
Execute the script
Verify that 'nova' user and group id has changed to '42436'
curl -O https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/master/common/nova_userid.sh
chmod +x nova_userid.sh
./nova_userid.sh
id nova
# uid=42436(nova) gid=42436(nova) groups=42436(nova),990(libvirt),36(kvm)
Deploy Trilio Datamover API on all Openstack controller nodes
Trilio Datamover Api container should be deployed on all nodes where nova_api container is running. In standard deployment, we can call these nodes as OpenStack controller nodes.
Pull TrilioVaullt Datamover API container image
The very first step is to pull the container image from docker.io.
Login to docker using credentials: triliodocker/triliopassword
Login to docker and pull the Trilio Datamover API container.
docker login docker.io
## Pull Trilio Datamover Api container using following command.
## You need to edit OS_PLATFORM, TVAULT_VERSION, OPENSTACK_RELEASE_NAME.
## <OS_PLATFORM> ubuntu or centos.
## <OPENSTACK_RELEASE_NAME> train
## <TVAULT_VERSION> 4.0.92 or 4.0.115
docker pull docker.io/trilio/<OS_PLATFORM>-source-trilio-datamover-api:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME>;
Example command for train openstack on ubuntu platform with triliovault 4.0 GA release: docker pull docker.io/trilio/ubuntu-source-trilio-datamover-api:4.0.92-train
Create the Trilio Datamover API configuration
In this part of the process is the configuration file for the Trilio Datamover API created.
The following steps need to be done:
Create config directory
get default config file dmapi.conf
edit dmapi.conf
copy nova.conf to config directory
Create config directory for trilio-datamover-api service
For CentOS, we need nova user ownership on datamover api log directory where as for Ubuntu, we need dmapi user's ownership on datmover api log directory.
mkdir -p /var/log/kolla/trilio-datamover-api
1. If openstack is based on CentOS :
##Change ownership of log directory to 42436:42436 - nova user and group id on container
chmod 755 /var/log/kolla/trilio-datamover-api
chown 42436:42436 /var/log/kolla/trilio-datamover-api
2. If openstack is based on Ubuntu :
##Change ownership of log directory to 42486:42487- dmapi user and group id on container
chmod 755 /var/log/kolla/trilio-datamover-api
chown 42486:42487 /var/log/kolla/trilio-datamover-api
Add trilio-datamover-api to haproxy.cfg
## Edit haproxy.cfg
vi /etc/kolla/haproxy/haproxy.cfg
## If SSL enabled on public interface of dmapi
listen trilio_datamover_api
bind <Keystone_virtual_ip>:8784 ssl crt /etc/haproxy/haproxy.pem
server <controller_hostname_1> <controller_IP1>:8784 check inter 2000 rise 2 fall 5
server <controller_hostname_2> <controller_IP2>:8784 check inter 2000 rise 2 fall 5
server <controller_hostname_3> <controller_IP3>:8784 check inter 2000 rise 2 fall 5
## If SSL is not enabled on any interface
listen trilio_datamover_api
bind <Keystone_virtual_ip>:8784
server <controller_hostname_1> <controller_IP1>:8784 check inter 2000 rise 2 fall 5
server <controller_hostname_2> <controller_IP2>:8784 check inter 2000 rise 2 fall 5
server <controller_hostname_3> <controller_IP3>:8784 check inter 2000 rise 2 fall 5
## Restart haproxy container
docker restart haproxy
Run trilio-datamover-api container
Now the trilio-datamover-api container can be deployed and started.
## You need to edit OS_PLATFORM, TVAULT_VERSION, OPENSTACK_RELEASE_NAME.
## <OS_PLATFORM> ubuntu or centos.
## <OPENSTACK_RELEASE_NAME> train
## <TVAULT_VERSION> : 4.0.92
docker run --network host --name trilio_datamover_api -d --restart always -v /etc/kolla/trilio-datamover-api/nova.conf:/etc/nova/nova.conf \
-v /etc/kolla/trilio-datamover-api/dmapi.conf:/etc/dmapi/dmapi.conf \
-v /var/log/kolla/:/var/log/kolla/ \
trilio/<OS_PLATFORM>-source-trilio-datamover-api:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME> /var/lib/kolla/venv/bin/python /usr/bin/dmapi-api
Verify deployment of trilio-datamover-api container
To verify the deployment was successful check the container status using docker ps.
root@trilio-Standard-PC-i440FX-PIIX-1996:~# docker ps | grep trilio_datamover_api
736e4ac462c9 trilio/ubuntu-source-trilio-datamover-api:4.0.92 "/var/lib/kolla/venv/" 3 days ago Up 23 hours trilio_datamover_api
Deploy Trilio Datamover container on all Compute nodes
Trilio Datamover container should be deployed on all nodes where nova_compute container is running. In standard deployment, we can call these nodes as openstack compute nodes.
At this stage it is necessary to know if the deployment shall use NFS or S3 protocol for the backup target.
Pull Trilio Datamover container
The very first step is to pull the container image from docker.io.
Login to docker using credentials: triliodocker/triliopassword
Login to docker and pull the Trilio Datamover API container.
docker login docker.io
## Pull Trilio Datamover container using following command.
## You need to edit OS_PLATFORM, TVAULT_VERSION, OPENSTACK_RELEASE_NAME.
## <OS_PLATFORM> ubuntu or centos.
## <OPENSTACK_RELEASE_NAME> train
## <TVAULT_VERSION> 4.0.92 or 4.0.115
docker pull docker.io/trilio/<OS_PLATFORM>-source-trilio-datamover:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME>;
Example command for train openstack on ubuntu platform with Trilio 4.0 GA release: docker pull docker.io/trilio/ubuntu-source-trilio-datamover:4.0.92-train
Create the Trilio Datamover configuration
In this part of the process is the configuration file for the Trilio Datamover created.
The following steps need to be done:
Create config directory
copy nova.conf to config directory
get default config file tvault-contego.conf
edit tvault.conf
Create service config directory for trilio-datamover service
mkdir -p /etc/kolla/trilio-datamover
Copy nova.conf of nova-compute service to trilio-datamover config directory
## Clone triliovault devops repository. It's public github repository.
git clone https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/
git checkout <GITHUB_BRANCH> // Check section: 1.i) plan for deployment for correct github branch name
## If backup storage type you want to use is 'NFS'
cp kolla-ansible/trilio-datamover/tvault-contego.conf.nfs.sample /etc/kolla/trilio-datamover/tvault-contego.conf
## If backup storage type you want to use is 'S3'
cp kolla-ansible/trilio-datamover/tvault-contego.conf.s3.sample /etc/kolla/trilio-datamover/tvault-contego.conf
## Set correct permissions
chmod -R 744 /etc/kolla/trilio-datamover/
Edit tvault-contego.conf
Edit /etc/kolla/trilio-datamover/tvault-contego.conf config file to provide NFS/S3 details as per backup storage selected.
In case of NFS backup target, only nfs share details need to provided. No other conf parameters need to edit, unless you know the details of it.
mkdir -p /var/log/kolla/trilio-datamover
##Change ownership of log directory to 42436:42436 - nova user and group id on container
chmod 755 /var/log/kolla/trilio-datamover
chown 42436:42436 /var/log/kolla/trilio-datamover
If Ceph is used for Nova/Cinder Storage
If ceph is getting used for cinder/nova, the correct permissions for ceph.conf and keyrings files need to be assigned. The trilio_datamover container will be using ceph.conf and keyring files with the 'nova' user.
chmod o+rx /etc/ceph
chmod o+r /etc/ceph/*
If nova/cinder backend is ceph, you need to add ceph user and keyring details to /etc/kolla/trilio-datamover/tvault-contego.conf file. Add the following sections to the tvault-contego.conf file. In the provided example is ceph's 'cinder' user configured to use for trilio read/write operations.
Mount /etc/ceph on trilio_datamover container in read only mode.
Check docker run command provided in the next step. The ceph user (example -'cinder') should have read and write permissions on ceph pool used for nova/cinder backend.
Verify nova user (uid - 42436) on trilio_datamover container is able to read ceph user's keyring file and ceph.conf after mounting /etc/ceph on the container.
Set appropriate permissions for /etc/ceph/ files on the host itself.
Run trilio-datamover container
If the cloud does not use 'ceph' storage for nova/cinder, remove '/etc/ceph' volume mount option from below commands.
## In following docker run command you need to edit OS_PLATFORM, TVAULT_VERSION,vOPENSTACK_RELEASE_NAME.
## <OS_PLATFORM> ubuntu or centos.
## <OPENSTACK_RELEASE_NAME> train
## <TVAULT_VERSION> : 4.0.92 or 4.0.115
##If backup storage type is 'NFS'
docker run --privileged --network host --name trilio_datamover -d \
--restart always \
-v /etc/kolla/trilio-datamover/tvault-contego.conf:/etc/tvault-contego/tvault-contego.conf \
-v /etc/kolla/trilio-datamover/nova.conf:/etc/nova/nova.conf \
-v /dev:/dev:rw \
-v /etc/ceph:/etc/ceph:ro \
-v nova_compute:/var/lib/nova/:rw \
-v /var/log/kolla/:/var/log/kolla/ \
-v iscsi_info:/etc/iscsi:rw -v /var/run/libvirt \
trilio/<OS_PLATFORM>-source-trilio-datamover:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME> /opt/tvault/start_datamover_nfs
##If backup storage type is 'S3'
docker run --privileged --network host --name trilio_datamover -d \
--restart always \
-v /etc/kolla/trilio-datamover/tvault-contego.conf:/etc/tvault-contego/tvault-contego.conf \
-v /etc/kolla/trilio-datamover/nova.conf:/etc/nova/nova.conf \
-v /dev:/dev:rw \
-v /etc/ceph:/etc/ceph:ro \
-v nova_compute:/var/lib/nova/:rw \
-v /var/log/kolla/:/var/log/kolla/ \
-v iscsi_info:/etc/iscsi:rw -v /var/run/libvirt \
trilio/<OS_PLATFORM>-source-trilio-datamover:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME> /opt/tvault/start_datamover_s3
Verify deployment of trilio-datamover container
To verify the deployment was successful check the container status using docker ps.
root@trilio-Standard-PC-i440FX-PIIX-1996:~# docker ps | grep trilio_datamover
9d572974c75d trilio/centos-source-trilio-datamover:4.0.92 "/opt/tvault/start_da" 2 days ago Up 23 hours trilio_datamover
Installing Trilio Horizon plugin
Trilio Horizon plugin needs to be installed inside the OpenStack horizon container. Once installed, the Trilio dashboard will be visible in OpenStack Horizon.
The following steps need to be done:
Download installation shell script
Run the shell script
Edit Horizon settings
Restart the Horizon container
Download the installation shell script
To download the shell script directly into the Horizon container do:
Run the shell script and restart Horizon container
You have to run the script inside the Horizon container as root.
Run the shell script and restart horizon container. This will restart apache service, which may enforce a log out of the container.
# Login to horizon container if not already.
docker exec -itu root horizon /bin/bash
## Run the trilio horizon plugin shell script
- Script will ask for python2/python3, you need to select the option as per your enviornment
- Trilio horizon install script will ask for horizon's openstack_dashboard directory path if it's not at the default
location - '/usr/shar/openstack-dashboard' For train ubuntu bionic, it's : '/var/lib/kolla/venv/lib/python2.7/site-packages'
./tvault-horizon-plugin-install.sh
## Exit the horizon container
Ctrl + D
Edit Horizon settings
The following line needs to be aded in 'local_settings' of Openstack's Horizon file to enable workloadmanager quota feature in the Horizon dashboard.
## vi /etc/kolla/horizon/local_settings
Add this line : HORIZON_CONFIG['customization_module'] = 'dashboards.overrides'
Restart the Horizon container
To enable the done changes restart the Horizon container
## Restart the horizon container
docker restart horizon
Known issues for Horizon plugin installation
If OpenStack is based on 'Centos' platform
This issue has not been observed in all CentOS based Kolla Train installations.
Please verify before disabling grafana repository.
Grafana yum repository has an issue on the latest horizon containers of OpenStack (not Trilio). To confirm the issue, you can just run yum repolist, it will fail. Use the following command to disable the grafana repository.
yum --disablerepo=grafana
If Openstack is based on Ubuntu platform
Trilio horizon install script will ask for horizon's openstack_dashboard directory path if it's not at the default location - /usr/shar/openstack-dashboard
For train ubuntu bionic, it's : /var/lib/kolla/venv/lib/python2.7/site-packages
If Trilio Horizon tabs are not accessible but Openstack Horizon works
If Trilio Horizon tabs are not accessible but OpenStack Horizon is working fine, make sure that endpoints for service 'TrilioVaultWLM' are created correctly. The root cause of this issue is typically, that SSL is enabled on all three endpoint types of 'TrilioVaultWLM' service.
If SSL is enable only on public 'keystone' service endpoints, then create 'TrilioVaultWLM' service endpoints in the same fashion. Endpoints for service 'TrilioVaultWLM' get created during Trilio configuration step. If these endpoints need to be edited reconfigure the Trilio.
Enabling Snapshout Mount
To make 'snapshot mount' functionality work, the cloud administrator needs to complete the following steps.
Identify backup target mount point on Trilio VM
install nfs-common on nova_compute and nova_libvirt containers
Mount backup target nfs share on nova_compute and nova_libvirt containers
Identify backup target mount point in Trilio VM
The following command will provide the active mountpoint on the Trilio VM
[stack@tvm ~]$ mount | grep triliovault
192.168.1.33:/mnt/tvault on /var/triliovault-mounts/MTkyLjE2OC4xLjMzOi9tbnQvdHZhdWx0 type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=180,retrans=2,sec=sys,clientaddr=192.168.10.10,lookupcache=none,local_lock=none,addr=192.168.1.33)
This example gives the following information:
Backup target is NFS share: 192.168.1.33:/mnt/tvault
Mountpoint is: /var/triliovault-mounts/MTkyLjE2OC4xLjMzOi9tbnQvdHZhdWx0
Install nfs-common on nova_compute and nova_libvirt containers
It is necessary to install nfs-common package on both nova_compute and nova_libvirt containers.
# Login to nova_compute container with root user
docker exec -itu root nova_compute /bin/bash
# Install nfs-utils/nfs-common package using
=> If it's centos
yum install nfs-utils
=> If it's ubuntu
apt-get install nfs-common
# Login to nova_libvirt container with root user
docker exec -itu root nova_libvirt /bin/bash
# Install nfs-utils/nfs-common package using yum
=> If it's centos
yum install nfs-utils
=> If it's ubuntu
apt-get install nfs-common
Mount backup target nfs share on nova_compute and nova_libvirt containers
Mount the backup target nfs share on 'nova_compute' and 'nova_libvirt' container at exactly same mount point as done on triliovault VM.
Create the mountpoint directory as necessary.
mount -t nfs 192.168.1.33:/mnt/tvault /var/triliovault-mounts/MTkyLjE2OC4xLjMzOi9tbnQvdHZhdWx0
Troubleshoot installation
If any triliovault container is stuck in restarting state the following logs can be checked.
# Get docker run logs for datamover container
docker logs trilio_datamover
## Get docker logs for datamover api container
docker logs trilio_datamover_api
Possible issues for trilio-datamover container failure are for example NFS mount issues or S3 credentials might be wrong. If it's Amazon S3, then network connectivity between compute node and AWS s3 is needed. The docker logs will clearly tell the exact error.
If the above logs do not help OR If containers running well but, backups fail, following service logs will help:
