arrow-left

All pages
gitbookPowered by GitBook
1 of 7

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Installing Trilio Components

Once the Trilio VM or the Cluster of Trilio VMs has been spun, the actual installation process can begin. This process contains of the following steps:

  1. Install the Trilio dm-api service on the control plane

  2. Install the Trilio datamover service on the compute plane

  3. Install the Trilio Horizon plugin into the Horizon service

How these steps look in detail is depending on the Openstack distribution Trilio is installed in. Each supported Openstack distribution has its own deployment tools. Trilio is integrating into these deployment tools to provide a native integration from the beginning to the end.

Installing on Canonical Openstack Queens and Train

Trilio and Canonical have started a partnership to ensure a native deployment of Trilio using JuJu Charms.

Those JuJu Charms are publicly available as Open Source Charms.

circle-info

The charms are currently listed as tech-preview. They will be fully supported by Canonical with the next charm release.

circle-check

Canonical Openstack doesn't require the Trilio Cluster. The required services are installed and managed via JuJu Charms.

The following charms exist:

  • ➡️ Installs and manages Trilio Controller services.

  • ➡️Installs and manages the Trilio Datamover API service.

  • ➡️

The documentation of the charms can be found here:

Installs and manages the Trilio Datamover service.
  • trilio-horizon-pluginarrow-up-right ➡️ Installs and manages the Trilio Horizon Plugin.

  • trilio-wlmarrow-up-right
    trilio-dm-apiarrow-up-right
    trilio-data-moverarrow-up-right

    Installing on RHOSP13

    The Red Hat Openstack Platform Directorarrow-up-right is the supported and recommended method to deploy and maintain any RHOSP installation.

    Trilio is integrating natively into the RHOSP Director. Manual deployment methods are not supported for RHOSP.

    hashtag
    Prepare for deployment

    Depending whether the RHOSP environment is already installed or is getting installed for the first time different steps are done to be able to deploy Trilio.

    hashtag
    Clone triliovault-cfg-scripts repository and upload Trilio puppet module

    circle-exclamation

    All commands need to be run as user 'stack'

    The following commands upload the Trilio puppet module to the overcloud. The actual upload happens upon the next deployment.

    circle-info

    Gitub branches to use:

    Trilio 4.0 GA == stable/4.0 Trilio 4.0 SP1 == v4.0maintenance

    hashtag
    Update overcloud roles data file to include Trilio services

    Trilio contains of multiple services. Add these services to your roles_data.yaml.

    circle-info

    In case of uncostomized roles_data.yaml can the default file be found on the undercloud at:

    /usr/share/openstack-tripleo-heat-templates/roles_data.yaml

    Add the following services to the roles_data.yaml

    hashtag
    Trilio Datamover Api Service

    This service needs to share the same role as the nova-api service. In case of the pre-defined roles will the nova-api service run on the role Controller. In case of custom defined roles, it is necessary to use the role the nova-api service is using.

    Add the following line to the identified role:

    hashtag
    Trilio Datamover Service

    This service needs to share the same role as the nova-compute service. In case of the pre-defined roles will the nova-compute service run on the role Compute. In case of custom defined roles, it is necessary to use the role the nova-compute service is using.

    Add the following line to the identified role:

    hashtag
    Prepare Trilio container images

    circle-exclamation

    All commands need to be run as user 'stack'

    Trilio containers are pushed to 'RedHat Container Registry'. Registry URL: 'registry.connect.redhat.com'. Container pull urls are given below.

    Trilio Datamover container: registry.connect.redhat.com/trilio/trilio-datamover:<container-tag> Trilio Datamover Api Container: registry.connect.redhat.com/trilio/trilio-datamover-api:<container-tag> Trilio horizon plugin: registry.connect.redhat.com/trilio/trilio-horizon-plugin:<container-tag>

    circle-info

    '4.0.92' is Trilio 4.0 build version. Container tag: 4.0.92-rhosp13 '4.0.115' is Trilio 4.0 SP1 build version. Container tag: 4.0.115-rhosp13

    There are three registry methods available in RedHat Openstack Platform.

    1. Remote Registry

    2. Local Registry

    3. Satellite Server

    hashtag
    Remote Registry

    Follow this section when 'Remote Registry' is used.

    For this method it is not necessary to pull the containers in advance. It is only necessary to populate the trilio_env.yaml and overcloud_images.yaml with the RedHat Registry URLs to the right containers.

    Populate the trilio_env.yaml with container urls for:

    • Trilio Datamover container

    • Trilio Datamover api container

    In the overcloud_images.yaml replace the standard Horizon Container with the Trilio Horizon container. This is necessary to gain access to the Trilio Horizon Dashboard.

    hashtag
    Local Registry

    Follow this section when 'local registry' is used on the undercloud.

    In this case it is necessary to pull the Trilio containers to the undercloud registry. Trilio provides shell scripts which will pull the containers from 'registry.connect.redhat.com' to the undercloud and updates the trilio_env.yaml with the values for the datamover and datamover api containers.

    circle-info

    The script assumes that the undercloud container registry is running on port 8787. If the registry is running on a different port, the script needs to be adjusted manually.

    The changes can be verified in the trilio_env.yaml.

    In the overcloud_images.yaml replace the standard Horizon Container with the Trilio Horizon container. This is necessary to gain access to the Trilio Horizon Dashboard.

    hashtag
    Satellite Server

    Follow this section when a Satellite Server is used for the container registry.

    Pull the Trilio containers on the Red Hat Satellite using the given Red Hat registry URLs.

    Populate the trilio_env.yaml with container urls for:

    • Trilio Datamover container

    • Trilio Datamover api container

    In the overcloud_images.yaml replace the standard Horizon Container with the Trilio Horizon container. This is necessary to gain access to the Trilio Horizon Dashboard.

    hashtag
    Provide environment details in trilio-env.yaml

    Provide backup target details and other necessary details in the provided environment file. This environment file will be used in the overcloud deployment to configure Trilio components. Container image names have already been populated in the preparation of the container images. Still it is recommended to verify the container URLs.

    The following information are required additionally:

    • Network for the datamover api

    • Backup target type {nfs/s3}

    • In case of NFS

    hashtag
    Deploy overcloud with trilio environment

    Use the following heat environment file and roles data file in overcloud deploy command:

    1. trilio_env.yaml

    2. roles_data.yaml

    To include new environment files use '-e' option and for roles data file use '-r' option. An example overcloud deploy command is shown below:

    hashtag
    Verify deployment

    hashtag
    On Controller node

    Make sure Trilio dmapi and horizon containers are in a running state and no other Trilio container is deployed on controller nodes.

    triangle-exclamation

    If the containers are in restarting state or not listed by the following command then your deployment is not done correctly. Please recheck if you followed the complete documentation.

    hashtag
    On Compute node

    Make sure Trilio datamover container is in running state and no other Trilio container is deployed on compute nodes.

    hashtag
    Configure Trilio Appliance

    Once RHOSP13 Installation steps have completed successfully, follow the instructions below to now configure the Trilio Appliance.

    hashtag
    Change the nova user id on the Trilio Nodes

    In RHOSP, 'nova' user id on nova-compute docker container is set to '42436'. The 'nova' user id on the Trilio nodes need to be set the same. Do the following steps on all Trilio nodes:

    1. Download the shell script that will change the user id

    2. Assign executable permissions

    3. Execute the script

    hashtag
    Run the Trilio configurator

    Follow .

    hashtag
    Change the workloadmgr.conf to use the right mountpoint

    circle-exclamation

    It is necessary to first configure the Trilio appliance, before the steps of this section can be done.

    RHOSP13 is using a different mount point in its datamover containers than other Openstack distribution. It is necessary to adjust the mountpoint of the Trilio Nodes to match this. If the mountpoints are not getting aligned, will files created by the datamover and read by the Trilio appliance not match in their paths and backup and restore processes will fail.

    Please follow these steps to align the mountpoints:

    1. Edit /etc/workloadmgr/workloadmgr.conf file

    2. Set parameter 'vault_data_directory' to '/var/lib/nova/triliovault-mounts'

    3. create the directory for the mountpoint

    hashtag
    Troubleshooting for overcloud deployment failures

    Trilio components will be deployed using puppet scripts.

    In case of the overcloud deployment failing does the following command provide the list of errors:

    Further commands that can help identifying any errors.

    In case of the trilio datamover api container is failing to start or stuck in restart, check these logs:

    In case of Trilio datamover container failing to start or being stuck in restart, check these logs:

    hashtag
    Cinder backend is Ceph - additional steps

    hashtag
    Add Ceph details to configuration file

    If Cinder backend is Ceph it is necessary to manually add the ceph details to tvault-contego.conf on all compute nodes.

    The file can be found here: /var/lib/config-data/puppet-generated/triliodm/etc/tvault-contego/tvault-contego.conf

    Add the following information:

    circle-info

    The same block of information can be found in the nova.conf file.

    hashtag
    Restart the datamover container

    list of NFS Shares

  • NFS options

  • In case of S3

    • s3 type {amazon_s3/ceph_s3}

    • Access key

    • Secret key

    • S3 Region name

    • S3 Bucket name

    • S3 Endpoint URL

    • S3 SSL Enabled {true/false}

  • Verify that 'nova' user and group id has changed to '42436'
    assign the created directory to nova:nova
  • unmount the old mountpoint

  • Update the Trilio configurator

  • Restart the Trilio services

  • Verify the mountpoint has been configured correctly

  • this documentation to configure Trilio Appliance
    cd /home/stack
    git clone -b <branch> https://github.com/trilioData/triliovault-cfg-scripts.git
    cd triliovault-cfg-scripts/redhat-director-scripts/
    # ./upload_puppet_module.sh
    Creating tarball...
    Tarball created.
    Creating heat environment file: /home/stack/.tripleo/environments/puppet-modules-url.yaml
    Uploading file to swift: /tmp/puppet-modules-8Qjya2X/puppet-modules.tar.gz
    +-----------------------+---------------------+----------------------------------+
    | object                | container           | etag                             |
    +-----------------------+---------------------+----------------------------------+
    | puppet-modules.tar.gz | overcloud-artifacts | 368951f6a4d39cfe53b5781797b133ad |
    +-----------------------+---------------------+----------------------------------+
    
    ## Make sure that
    ls -ll /home/stack/.tripleo/environments/puppet-modules-url.yaml
    'OS::TripleO::Services::TrilioDatamoverApi'
    'OS::TripleO::Services::TrilioDatamover' 
    $ grep '<container-tag>' trilio_env.yaml
       DockerTrilioDatamoverImage: registry.connect.redhat.com/trilio/trilio-datamover:<container-tag>
       DockerTrilioDmApiImage: registry.connect.redhat.com/trilio/trilio-datamover-api:<container-tag>
    $ grep 'trilio' /home/stack/templates/overcloud_images.yaml
       DockerHorizonImage: registry.connect.redhat.com/trilio/trilio-horizon-plugin:<container-tag>
    ## Script argument details: 
    ./prepare_trilio_images.sh <undercloud_ip> <container_tag>
    
    ## Example execution:
    ./prepare_trilio_images.sh 192.168.13.34 4.0.92-rhosp13
    [stack@undercloud$ grep '4.0.92-rhosp13' trilio_env.yaml
       DockerTrilioDatamoverImage: 192.168.122.10:8787/trilio/trilio-datamover:4.0.92-rhosp13
       DockerTrilioDmApiImage: 192.168.122.10:8787/trilio/trilio-datamover-api:4.0.92-rhosp13
    $ grep 'trilio' /home/stack/templates/overcloud_images.yaml
      DockerHorizonImage: 192.168.122.10:8787/trilio/trilio-horizon-plugin:4.0.92-rhosp13
    $ grep '4.0.92-rhosp13' trilio_env.yaml
       DockerTrilioDatamoverImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover:4.0.92-rhosp13
       DockerTrilioDmApiImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-datamover-api:4.0.92-rhosp13
    $ grep 'trilio' /home/stack/templates/overcloud_images.yaml
       DockerHorizonImage: <SATELLITE_REGISTRY_URL>/trilio/trilio-horizon-plugin:4.0.92-rhosp13
    resource_registry:
      OS::TripleO::Services::TrilioDatamover: docker/services/trilio-datamover.yaml
      OS::TripleO::Services::TrilioDatamoverApi: docker/services/trilio-datamover-api.yaml
    
    
    parameter_defaults:
    
       ## Define network map for trilio datamover api service
       ServiceNetMap:
           TrilioDatamoverApiNetwork: internal_api
    
       ## Container locations
       DockerTrilioDatamoverImage: 192.168.122.10:8787/trilio/trilio-datamover:3.1.58-queens
    
       DockerTrilioDmApiImage: 192.168.122.10:8787/trilio/trilio-datamover-api:3.1.58-queens
    
       ## Backup target type nfs/s3, used to store snapshots taken by triliovault
       BackupTargetType: 'nfs'
    
       ## For backup target 'nfs'
       NfsShares: '192.168.122.101:/opt/tvault'
       NfsOptions: 'nolock,soft,timeo=180,intr,lookupcache=none'
    
       ## For backup target 's3'
       ## S3 type: amazon_s3/ceph_s3
       S3Type: 'amazon_s3'
    
       ## S3 access key
       S3AccessKey: ''
      
       ## S3 secret key
       S3SecretKey: ''
    
       ## S3 region, if your s3 does not have any region, just keep the parameter as it is
       S3RegionName: ''
    
       ## S3 bucket name
       S3Bucket: ''
    
       ## S3 endpoint url, not required for Amazon S3, keep it as it is
       S3EndpointUrl: ''
    
       ## If SSL enabled on S3 url, not required for Amazon S3, just keep it as it is
       S3SslEnabled: false
    
       ## Don't edit following parameter
       EnablePackageInstall: True
    openstack overcloud deploy --templates \
    -e /home/stack/templates/overcloud_images.yaml
    -e ${basedir}/trilio_env.yaml \
    -r ${basedir}/roles_data.yaml \
    --control-scale 1 --compute-scale 1 --control-flavor control --compute-flavor compute \
    --ntp-server 0.north-america.pool.ntp.org \
    [root@overcloud-controller-0 trilio]# docker container ls | grep trilio
    cad4b68a6436 192.168.122.151:8787/trilio/trilio-datamover-api:4.0.92-rhosp13  "kolla_start"  2 days ago  Up 2 days  trilio_dmapi
    10b95b501092 192.168.122.151:8787/trilio/trilio-horizon-plugin:4.0.92-rhosp13  "kolla_start"  2 days ago  Up 2 days  horizon
    # docker container ls | grep trilio
    2598963695c7  192.168.122.151:8787/trilio/trilio-datamover:4.0.92-rhosp13  "kolla_start"  2 days ago  Up 2 days  trilio_datamover
    # curl -O https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/master/common/nova_userid.sh
    # chmod +x nova_userid.sh
    # ./nova_userid.sh
    # id nova
      uid=42436(nova) gid=42436(nova) groups=42436(nova),990(libvirt),36(kvm)
    # cat /etc/workloadmgr/workloadmgr.conf | grep vault_data_directory
    vault_data_directory = /var/lib/nova/triliovault-mounts
    vault_data_directory_old = /var/triliovault
    
    # mkdir -p /var/lib/nova/triliovault-mounts
    
    # chown nova:nova /var/lib/nova/triliovault-mounts
    
    # umount /var/triliovault-mounts
    
    # cat /home/stack/myansible/lib/python3.6/site-packages/workloadmgr/tvault_configurator/ansible-play/roles/ansible-workloadmgr/templates/workloadmgr.conf.j2 | grep vault_data_directory
    vault_data_directory = /var/lib/nova/triliovault-mounts
    
    #pcs resource restart wlm-cron 
    
    #pcs resource restart wlm-scheduler 
    
    #systemctl restart wlm-api 
    
    #systemctl restart wlm-workloads
    
    #df-h
    Filesystem             Size  Used Avail Use% Mounted on
    devtmpfs               3.8G     0  3.8G   0% /dev
    tmpfs                  3.8G   38M  3.8G   1% /dev/shm
    tmpfs                  3.8G  427M  3.4G  12% /run
    tmpfs                  3.8G     0  3.8G   0% /sys/fs/cgroup
    /dev/vda1               40G  8.8G   32G  22% /
    tmpfs                  773M     0  773M   0% /run/user/996
    tmpfs                  773M     0  773M   0% /run/user/0
    10.10.2.20:/upstream  1008G  704G  254G  74% /var/lib/nova/triliovault-mounts/MTAuMTAuMi4yMDovdXBzdHJlYW0= 
    openstack stack failures list overcloud
    heat stack-list --show-nested -f "status=FAILED"
    heat resource-list --nested-depth 5 overcloud | grep FAILED
    docker logs trilio_dmapi
    tailf /var/log/containers/trilio-datamover-api/dmapi.log
    docker logs trilio_datamover
    tailf /var/log/containers/trilio-datamover/tvault-contego.log
    [libvirt]
    images_rbd_ceph_conf = /etc/ceph/ceph.conf
    rbd_user = <rbd_user>
    
    [ceph] 
    keyring_ext = .<rbd_user>.keyring
    sudo docker restart trilio_datamover

    Installing on RHOSP16.0

    The Red Hat Openstack Platform Directorarrow-up-right is the supported and recommended method to deploy and maintain any RHOSP installation.

    Trilio is integrating natively into the RHOSP Director. Manual deployment methods are not supported for RHOSP.

    hashtag
    Prepare for deployment

    Depending whether the RHOSP environment is already installed or is getting installed for the first time different steps are done to be able to deploy Trilio.

    circle-exclamation

    All commands need to be run as user 'stack'

    The following commands upload the Trilio puppet module to the overcloud. The actual upload happens upon the next deployment.

    circle-info

    Gitub branches to use:

    Trilio 4.0 GA == stable/4.0 Trilio 4.0 SP1 == v4.0maintenance

    hashtag
    Update overcloud roles data file to include Trilio services

    Trilio contains of multiple services. Add these services to your roles_data.yaml.

    circle-info

    In case of uncostomized roles_data.yaml can the default file be found on the undercloud at:

    /usr/share/openstack-tripleo-heat-templates/roles_data.yaml

    Add the following services to the roles_data.yaml

    hashtag
    Trilio Datamover Api Service

    This service needs to share the same role as the nova-api service. In case of the pre-defined roles will the nova-api service run on the role Controller. In case of custom defined roles, it is necessary to use the role the nova-api service is using.

    Add the following line to the identified role:

    hashtag
    Trilio Datamover Service

    This service needs to share the same role as the nova-compute service. In case of the pre-defined roles will the nova-compute service run on the role Compute. In case of custom defined roles, it is necessary to use the role the nova-compute service is using.

    Add the following line to the identified role:

    hashtag
    Prepare Trilio container images

    circle-exclamation

    All commands need to be run as user 'stack'

    Trilio containers are pushed to 'RedHat Container Registry'. Registry URL: 'registry.connect.redhat.com'. Container pull urls are given below.

    Trilio Datamover container: registry.connect.redhat.com/trilio/trilio-datamover:<container-tag> Trilio Datamover Api Container: registry.connect.redhat.com/trilio/trilio-datamover-api:<container-tag> Trilio horizon plugin: registry.connect.redhat.com/trilio/trilio-horizon-plugin:<container-tag>

    circle-info

    '4.0.92' is Trilio 4.0 build version. Container tag: 4.0.92-rhosp16 '4.0.115' is Trilio 4.0 SP1 build version. Container tag: 4.0.115-rhosp16

    There are three registry methods available in RedHat Openstack Platform.

    1. Remote Registry

    2. Local Registry

    3. Satellite Server

    hashtag
    Remote Registry

    Follow this section when 'Remote Registry' is used.

    For this method it is not necessary to pull the containers in advance. It is only necessary to populate the trilio_env.yaml file with the Trilio container URLs from Redhat registry.

    Populate the trilio_env_osp16.yaml with container urls for:

    • Trilio Datamover container

    • Trilio Datamover api container

    • Trilio Horizon Plugin

    circle-info

    trilio_env_osp16.yaml will be available in triliovault-cfg-scripts/redhat-director-scripts/

    hashtag
    Local Registry

    Follow this section when 'local registry' is used on the undercloud.

    In this case it is necessary to pull the Trilio containers to the undercloud registry. Trilio provides shell scripts which will pull the containers from 'registry.connect.redhat.com' to the undercloud and updates the trilio_env_osp16.yaml with the values for the datamover and datamover api containers.

    circle-info

    The script assumes that the undercloud container registry is running on port 8787. If the registry is running on a different port, the script needs to be adjusted manually.

    The changes can be verified using the following commands.

    hashtag
    Satellite Server

    Follow this section when a Satellite Server is used for the container registry.

    Pull the Trilio containers on the Red Hat Satellite using the given Red Hat registry URLs.

    Populate the trilio_env_osp16.yaml with container urls for:

    • Trilio Datamover container

    • Trilio Datamover api container

    hashtag
    Provide environment details in trilio-env_osp16.yaml

    Provide backup target details and other necessary details in the provided environment file. This environment file will be used in the overcloud deployment to configure Trilio components. Container image names have already been populated in the preparation of the container images. Still it is recommended to verify the container URLs.

    The following information are required additionally:

    • Network for the datamover api

    • Backup target type {nfs/s3}

    • In case of NFS

    hashtag
    Deploy overcloud with trilio environment

    Use the following heat environment file and roles data file in overcloud deploy command:

    1. trilio_env_osp16.yaml

    2. roles_data.yaml

    To include new environment files use '-e' option and for roles data file use '-r' option. An example overcloud deploy command is shown below:

    hashtag
    Verify deployment

    triangle-exclamation

    If the containers are in restarting state or not listed by the following command then your deployment is not done correctly. Please recheck if you followed the complete documentation.

    hashtag
    On Controller node

    Make sure Trilio dmapi and horizon containers are in a running state and no other Trilio container is deployed on controller nodes.

    hashtag
    On Compute node

    Make sure Trilio datamover container is in running state and no other Trilio container is deployed on compute nodes.

    hashtag
    On the node with Horizon service

    Make sure horizon container is in running state. Please note that 'Horizon' container is replaced with Trilio Horizon container. This container will have latest OpenStack horizon + Trilio's horizon plugin.

    hashtag
    Configure Trilio Appliance

    Once RHOSP16.0 Installation steps have completed successfully, follow the instructions below to now configure the Trilio Appliance

    hashtag
    Change the nova user id on the Trilio Nodes

    In RHOSP, 'nova' user id on nova-compute docker container is set to '42436'. The 'nova' user id on the Trilio nodes need to be set the same. Do the following steps on all Trilio nodes:

    1. Download the shell script that will change the user id

    2. Assign executable permissions

    3. Execute the script

    hashtag
    Run the Trilio configurator

    Follow .

    hashtag
    Change the workloadmgr.conf to use the right mountpoint

    circle-exclamation

    It is necessary to first configure the Trilio appliance, before the steps of this section can be done.

    RHOSP16 is using a different mount point in its datamover containers than other Openstack distribution. It is necessary to adjust the mountpoint of the Trilio Nodes to match this. If the mountpoints are not getting aligned, will files created by the datamover and read by the Trilio appliance not match in their paths and backup and restore processes will fail.

    Please follow these steps to align the mountpoints:

    1. Edit /etc/workloadmgr/workloadmgr.conf file

    2. Set parameter 'vault_data_directory' to '/var/lib/nova/triliovault-mounts'

    3. create the directory for the mountpoint

    hashtag
    Troubleshooting for overcloud deployment failures

    Trilio components will be deployed using puppet scripts.

    In case of the overcloud deployment failing does the following command provide the list of errors:

    Further commands that can help identifying any errors.

    In case of the trilio datamover api container is failing to start or stuck in restart, check these logs:

    In case of Trilio datamover container failing to start or being stuck in restart, check these logs:

    hashtag
    Cinder backend is Ceph - additional steps

    hashtag
    Add Ceph details to configuration file

    If Cinder backend is Ceph it is necessary to manually add the ceph details to tvault-contego.conf on all compute nodes.

    The file can be found here: /var/lib/config-data/puppet-generated/triliodm/etc/tvault-contego/tvault-contego.conf

    Add the following information:

    circle-info

    The same block of information can be found in the nova.conf file.

    hashtag
    Restart the datamover container

    list of NFS Shares

  • NFS options

  • In case of S3

    • s3 type {amazon_s3/ceph_s3}

    • Access key

    • Secret key

    • S3 Region name

    • S3 Bucket name

    • S3 Endpoint URL

    • S3 SSL Enabled {true/false}

  • Verify that 'nova' user and group id has changed to '42436'
    assign the created directory to nova:nova
  • unmount the old mountpoint

  • Update the Trilio configurator

  • Restart the Trilio services

  • Verify the mountpoint has been configured correctly

  • this documentation to configure Trilio Appliance
    cd /home/stack
    git clone -b <branch> https://github.com/trilioData/triliovault-cfg-scripts.git
    cd triliovault-cfg-scripts/redhat-director-scripts/
    # ./upload_puppet_module_puppet5.sh
    Creating tarball...
    Tarball created.
    Creating heat environment file: /home/stack/.tripleo/environments/puppet-modules-url.yaml
    Uploading file to swift: /tmp/puppet-modules-8Qjya2X/puppet-modules.tar.gz
    +-----------------------+---------------------+----------------------------------+
    | object                | container           | etag                             |
    +-----------------------+---------------------+----------------------------------+
    | puppet-modules.tar.gz | overcloud-artifacts | 368951f6a4d39cfe53b5781797b133ad |
    +-----------------------+---------------------+----------------------------------+
    
    ## Make sure that
    ls -ll /home/stack/.tripleo/environments/puppet-modules-url.yaml
    'OS::TripleO::Services::TrilioDatamoverApi'
    'OS::TripleO::Services::TrilioDatamover' 
    (undercloud) [stack@undercloud redhat-director-scripts]$ grep 'Image' trilio_env_osp16.yaml
       DockerTrilioDatamoverImage: registry.connect.redhat.com/trilio/trilio-datamover:<container-tag>
       DockerTrilioDmApiImage: registry.connect.redhat.com/trilio/trilio-datamover-api:<container-tag>
       ContainerHorizonImage: registry.connect.redhat.com/trilio/trilio-horizon-plugin:<container-tag>
    ## Command to prepare, push trilio container images to undercloud registry: 
    sudo ./podman_prepare_images.sh <UNDERCLOUD_REGISTRY_HOSTNAME> <CONTAINER_TAG> 
    
    
    ## Run following command to find 'UNDERCLOUD_REGISTRY_HOSTNAME'. Hostname is higlighted in bold format in below example.
    $ openstack tripleo container image list | grep keystone
    | docker://trilio-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-keystone:16.0-82                       |
    | docker://trilio-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-barbican-keystone-listener:16.0-84   
    
    ## 'CONTAINER_TAG' format: <TRILIOVAULT_VERSION>-rhosp16
    ## For example. If Trilio version='4.0.92',  then 'CONTAINER_TAG'=4.0.92-rhosp16
    
    ## Example:
    sudo ./podman_prepare_images.sh trilio-undercloud.ctlplane.localdomain 4.0.92-rhosp16
    (undercloud) [stack@undercloud redhat-director-scripts]$ openstack tripleo container image list | grep trilio
    | docker://undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:4.0.92-rhosp16                       |                        |
    | docker://undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:4.0.92-rhosp16                   |                   |
    | docker://undercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:4.0.92-rhosp16                  |
    
    -----------------------------------------------------------------------------------------------------
    
    (undercloud) [stack@undercloud redhat-director-scripts]$ grep 'Image' trilio_env_osp16.yaml
       DockerTrilioDatamoverImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:4.0.92-rhosp16
       DockerTrilioDmApiImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:4.0.92-rhosp16
       ContainerHorizonImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:4.0.92-rhosp16
    
    (undercloud) [stack@undercloud redhat-director-scripts]$ grep 'Image' trilio_env_osp16.yaml
       DockerTrilioDatamoverImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:4.0.92-rhosp16
       DockerTrilioDmApiImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:4.0.92-rhosp16
       ContainerHorizonImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:4.0.92-rhosp16
    resource_registry:
      OS::TripleO::Services::TrilioDatamover: docker/services/trilio-datamover-osp16.yaml
      OS::TripleO::Services::TrilioDatamoverApi: docker/services/trilio-datamover-api-osp16.yaml
    
    parameter_defaults:
    
       ## Enable Trilio's quota functionality on horizon
       ExtraConfig:
         horizon::customization_module: 'dashboards.overrides'
    
       ## Define network map for trilio datamover api service
       ServiceNetMap:
           TrilioDatamoverApiNetwork: internal_api
    
       ## Trilio container pull urls
       DockerTrilioDatamoverImage: devundercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:4.0.92-rhosp16
       DockerTrilioDmApiImage: devundercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:4.0.92-rhosp16
    
       ## If you do not want Trilio's horizon plugin to replace your horizon container, just comment following line.
       ContainerHorizonImage: devundercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:4.0.92-rhosp16
    
    
       ## Backup target type nfs/s3, used to store snapshots taken by triliovault
       BackupTargetType: 'nfs'
    
       ## For backup target 'nfs'
       NfsShares: '192.168.122.101:/opt/tvault'
       NfsOptions: 'nolock,soft,timeo=180,intr,lookupcache=none'
    
       ## For backup target 's3'
       ## S3 type: amazon_s3/ceph_s3
       S3Type: 'amazon_s3'
    
       ## S3 access key
       S3AccessKey: ''
      
       ## S3 secret key
       S3SecretKey: ''
    
       ## S3 region, if your s3 does not have any region, just keep the parameter as it is
       S3RegionName: ''
    
       ## S3 bucket name
       S3Bucket: ''
    
       ## S3 endpoint url, not required for Amazon S3, keep it as it is
       S3EndpointUrl: ''
    
       ## If SSL enabled on S3 url, not required for Amazon S3, just keep it as it is
       S3SslEnabled: false
    
       ## Don't edit following parameter
       EnablePackageInstall: True
    openstack overcloud deploy --templates \
    --libvirt-type qemu \
    --ntp-server 192.168.1.34 \
    -e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/trilio_env_osp16.yaml \
    -e /home/stack/templates/node-info.yaml \
    -e /home/stack/containers-prepare-parameter.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-tls.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/inject-trust-anchor.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \
    -r /usr/share/openstack-tripleo-heat-templates/roles_data.yaml
    [root@overcloud-controller-0 heat-admin]# podman ps | grep trilio
    26fcb9194566  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-datamover-api:4.0.92-rhosp16  kolla_start  5 days ago  Up 5 days ago  trilio_dmapi
    094971d0f5a9  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:4.0.92-rhosp16  kolla_start  5 days ago  Up 5 days ago  horizon
    [root@overcloud-novacompute-0 heat-admin]# podman ps | grep trilio
    b1840444cc59  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-datamover:4.0.92-rhosp16  kolla_start  5 days ago  Up 5 days ago  tilio_datamover
    [root@overcloud-controller-0 heat-admin]# podman ps | grep horizon
    094971d0f5a9  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:4.0.92-rhosp16  kolla_start  5 days ago  Up 5 days ago  horizon
    # curl -O https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/master/common/nova_userid.sh
    # chmod +x nova_userid.sh
    # ./nova_userid.sh
    # id nova
      uid=42436(nova) gid=42436(nova) groups=42436(nova),990(libvirt),36(kvm)
    # cat /etc/workloadmgr/workloadmgr.conf | grep vault_data_directory
    vault_data_directory = /var/lib/nova/triliovault-mounts
    vault_data_directory_old = /var/triliovault
    
    # mkdir -p /var/lib/nova/triliovault-mounts
    
    # chown nova:nova /var/lib/nova/triliovault-mounts
    
    # umount /var/triliovault-mounts
    
    # cat /home/stack/myansible/lib/python3.6/site-packages/workloadmgr/tvault_configurator/ansible-play/roles/ansible-workloadmgr/templates/workloadmgr.conf.j2 | grep vault_data_directory
    vault_data_directory = /var/lib/nova/triliovault-mounts
    
    #pcs resource restart wlm-cron 
    
    #pcs resource restart wlm-scheduler 
    
    #systemctl restart wlm-api 
    
    #systemctl restart wlm-workloads
    
    #df-h
    Filesystem             Size  Used Avail Use% Mounted on
    devtmpfs               3.8G     0  3.8G   0% /dev
    tmpfs                  3.8G   38M  3.8G   1% /dev/shm
    tmpfs                  3.8G  427M  3.4G  12% /run
    tmpfs                  3.8G     0  3.8G   0% /sys/fs/cgroup
    /dev/vda1               40G  8.8G   32G  22% /
    tmpfs                  773M     0  773M   0% /run/user/996
    tmpfs                  773M     0  773M   0% /run/user/0
    10.10.2.20:/upstream  1008G  704G  254G  74% /var/lib/nova/triliovault-mounts/MTAuMTAuMi4yMDovdXBzdHJlYW0= 
    openstack stack failures list overcloud
    heat stack-list --show-nested -f "status=FAILED"
    heat resource-list --nested-depth 5 overcloud | grep FAILED
    sudo podman logs trilio_dmapi
    tailf /var/log/containers/trilio-datamover-api/dmapi.log
    sudo podman logs trilio_datamover
    tailf /var/log/containers/trilio-datamover/tvault-contego.log
    [libvirt]
    images_rbd_ceph_conf = /etc/ceph/ceph.conf
    rbd_user = <rbd_user>
    
    [ceph] 
    keyring_ext = .<rbd_user>.keyring
    sudo podman restart trilio_datamover
    TrilioVault data protection — charm-guide 0.0.1.dev819 documentationdocs.openstack.orgchevron-right
    Logo

    Installing on Kolla Train

    hashtag
    Change the nova user id on the Trilio Nodes

    In Kolla, 'nova' user id on nova-compute docker container is set to '42436'. The 'nova' user id on the Trilio nodes need to be set the same. Do the following steps on all compute nodes:

    1. Download the shell script that will change the user id

    2. Assign executable permissions

    3. Execute the script

    4. Verify that 'nova' user and group id has changed to '42436'

    hashtag
    Deploy Trilio Datamover API on all Openstack controller nodes

    Trilio Datamover Api container should be deployed on all nodes where nova_api container is running. In standard deployment, we can call these nodes as OpenStack controller nodes.

    hashtag
    Pull TrilioVaullt Datamover API container image

    The very first step is to pull the container image from docker.io.

    circle-info

    Login to docker using credentials: triliodocker/triliopassword

    Login to docker and pull the Trilio Datamover API container.

    circle-info

    Example command for train openstack on ubuntu platform with triliovault 4.0 GA release: docker pull docker.io/trilio/ubuntu-source-trilio-datamover-api:4.0.92-train

    hashtag
    Create the Trilio Datamover API configuration

    In this part of the process is the configuration file for the Trilio Datamover API created.

    The following steps need to be done:

    1. Create config directory

    2. get default config file dmapi.conf

    3. edit dmapi.conf

    hashtag
    Create config directory for trilio-datamover-api service

    hashtag
    Get the trilio-datamover-api default config file

    hashtag
    Edit dmapi.conf

    The dmapi.conf located in /etc/kolla/trilio-datamover-api/ needs to be edited to adjust to the Openstack environment.

    circle-info

    Nearly all required values can be copied from the nova.conf located at: /etc/kolla/nova-api/

    circle-info

    Follow comments inside the dmapi.conf to learn which parameters are the minimum needed.

    An example dmapi.conf can be seen here:

    hashtag
    Copy nova.conf of nova-api service to trilio-datamover-api directory

    hashtag
    Create trilio-datamover-api log directory

    circle-exclamation

    For CentOS, we need nova user ownership on datamover api log directory where as for Ubuntu, we need dmapi user's ownership on datmover api log directory.

    hashtag
    Add trilio-datamover-api to haproxy.cfg

    hashtag
    Run trilio-datamover-api container

    Now the trilio-datamover-api container can be deployed and started.

    hashtag
    Verify deployment of trilio-datamover-api container

    To verify the deployment was successful check the container status using docker ps.

    hashtag
    Deploy Trilio Datamover container on all Compute nodes

    Trilio Datamover container should be deployed on all nodes where nova_compute container is running. In standard deployment, we can call these nodes as openstack compute nodes.

    circle-info

    At this stage it is necessary to know if the deployment shall use NFS or S3 protocol for the backup target.

    hashtag
    Pull Trilio Datamover container

    The very first step is to pull the container image from docker.io.

    circle-info

    Login to docker using credentials: triliodocker/triliopassword

    Login to docker and pull the Trilio Datamover API container.

    circle-info

    Example command for train openstack on ubuntu platform with Trilio 4.0 GA release: docker pull docker.io/trilio/ubuntu-source-trilio-datamover:4.0.92-train

    hashtag
    Create the Trilio Datamover configuration

    In this part of the process is the configuration file for the Trilio Datamover created.

    The following steps need to be done:

    1. Create config directory

    2. copy nova.conf to config directory

    3. get default config file tvault-contego.conf

    hashtag
    Create service config directory for trilio-datamover service

    hashtag
    Copy nova.conf of nova-compute service to trilio-datamover config directory

    hashtag
    Get the trilio-datamover default config file

    hashtag
    Edit tvault-contego.conf

    Edit /etc/kolla/trilio-datamover/tvault-contego.conf config file to provide NFS/S3 details as per backup storage selected.

    circle-info

    In case of NFS backup target, only nfs share details need to provided. No other conf parameters need to edit, unless you know the details of it.

    hashtag
    Create trilio-datamover log directory

    hashtag
    If Ceph is used for Nova/Cinder Storage

    If ceph is getting used for cinder/nova, the correct permissions for ceph.conf and keyrings files need to be assigned. The trilio_datamover container will be using ceph.conf and keyring files with the 'nova' user.

    If nova/cinder backend is ceph, you need to add ceph user and keyring details to /etc/kolla/trilio-datamover/tvault-contego.conf file. Add the following sections to the tvault-contego.conf file. In the provided example is ceph's 'cinder' user configured to use for trilio read/write operations.

    circle-exclamation

    Mount /etc/ceph on trilio_datamover container in read only mode. Check docker run command provided in the next step. The ceph user (example -'cinder') should have read and write permissions on ceph pool used for nova/cinder backend. Verify nova user (uid - 42436) on trilio_datamover container is able to read ceph user's keyring file and ceph.conf after mounting /etc/ceph on the container. Set appropriate permissions for /etc/ceph/ files on the host itself.

    hashtag
    Run trilio-datamover container

    circle-exclamation

    If the cloud does not use 'ceph' storage for nova/cinder, remove '/etc/ceph' volume mount option from below commands.

    hashtag
    Verify deployment of trilio-datamover container

    To verify the deployment was successful check the container status using docker ps.

    hashtag
    Installing Trilio Horizon plugin

    Trilio Horizon plugin needs to be installed inside the OpenStack horizon container. Once installed, the Trilio dashboard will be visible in OpenStack Horizon.

    The following steps need to be done:

    1. Download installation shell script

    2. Run the shell script

    3. Edit Horizon settings

    hashtag
    Download the installation shell script

    To download the shell script directly into the Horizon container do:

    hashtag
    Run the shell script and restart Horizon container

    circle-info

    You have to run the script inside the Horizon container as root.

    Run the shell script and restart horizon container. This will restart apache service, which may enforce a log out of the container.

    hashtag
    Edit Horizon settings

    The following line needs to be aded in 'local_settings' of Openstack's Horizon file to enable workloadmanager quota feature in the Horizon dashboard.

    hashtag
    Restart the Horizon container

    To enable the done changes restart the Horizon container

    hashtag
    Known issues for Horizon plugin installation

    hashtag
    If OpenStack is based on 'Centos' platform

    circle-info

    This issue has not been observed in all CentOS based Kolla Train installations. Please verify before disabling grafana repository.

    Grafana yum repository has an issue on the latest horizon containers of OpenStack (not Trilio). To confirm the issue, you can just run yum repolist, it will fail. Use the following command to disable the grafana repository.

    hashtag
    If Openstack is based on Ubuntu platform

    Trilio horizon install script will ask for horizon's openstack_dashboard directory path if it's not at the default location - /usr/shar/openstack-dashboard

    For train ubuntu bionic, it's : /var/lib/kolla/venv/lib/python2.7/site-packages

    hashtag
    If Trilio Horizon tabs are not accessible but Openstack Horizon works

    If Trilio Horizon tabs are not accessible but OpenStack Horizon is working fine, make sure that endpoints for service 'TrilioVaultWLM' are created correctly. The root cause of this issue is typically, that SSL is enabled on all three endpoint types of 'TrilioVaultWLM' service.

    If SSL is enable only on public 'keystone' service endpoints, then create 'TrilioVaultWLM' service endpoints in the same fashion. Endpoints for service 'TrilioVaultWLM' get created during step. If these endpoints need to be edited reconfigure the Trilio.

    hashtag
    Enabling Snapshout Mount

    To make 'snapshot mount' functionality work, the cloud administrator needs to complete the following steps.

    1. Identify backup target mount point on Trilio VM

    2. install nfs-common on nova_compute and nova_libvirt containers

    3. Mount backup target nfs share on nova_compute and nova_libvirt containers

    hashtag
    Identify backup target mount point in Trilio VM

    The following command will provide the active mountpoint on the Trilio VM

    This example gives the following information:

    Backup target is NFS share: 192.168.1.33:/mnt/tvault Mountpoint is: /var/triliovault-mounts/MTkyLjE2OC4xLjMzOi9tbnQvdHZhdWx0

    hashtag
    Install nfs-common on nova_compute and nova_libvirt containers

    It is necessary to install nfs-common package on both nova_compute and nova_libvirt containers.

    hashtag
    Mount backup target nfs share on nova_compute and nova_libvirt containers

    Mount the backup target nfs share on 'nova_compute' and 'nova_libvirt' container at exactly same mount point as done on triliovault VM.

    circle-info

    Create the mountpoint directory as necessary.

    hashtag
    Troubleshoot installation

    If any triliovault container is stuck in restarting state the following logs can be checked.

    Possible issues for trilio-datamover container failure are for example NFS mount issues or S3 credentials might be wrong. If it's Amazon S3, then network connectivity between compute node and AWS s3 is needed. The docker logs will clearly tell the exact error.

    If the above logs do not help OR If containers running well but, backups fail, following service logs will help:

    If the Trilio Horizon tabs are not visible on Openstack, verify the following:

    • Make sure trilio horizon plugin is installed on OpenStack horizon container

    • Trilio configuration step needs to be completed to see the triliovault dashboard on OpenStack

    • Make sure correct openstack_dashboard directory got provided and the triliovault horizon plugin files got successfully copied there.

    copy nova.conf to config directory
    edit tvault.conf
    Restart the Horizon container
    Trilio configuration
    curl -O https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/master/common/nova_userid.sh 
    chmod +x nova_userid.sh
    ./nova_userid.sh
    id nova
    # uid=42436(nova) gid=42436(nova) groups=42436(nova),990(libvirt),36(kvm)
    docker login docker.io
    
    ## Pull Trilio Datamover Api container using following command.
    ## You need to edit OS_PLATFORM, TVAULT_VERSION, OPENSTACK_RELEASE_NAME.
    ## <OS_PLATFORM> ubuntu or centos.
    ## <OPENSTACK_RELEASE_NAME> train
    ## <TVAULT_VERSION> 4.0.92 or 4.0.115
    
    docker pull docker.io/trilio/<OS_PLATFORM>-source-trilio-datamover-api:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME>;
    mkdir -p /etc/kolla/trilio-datamover-api
    git clone https://github.com/trilioData/triliovault-cfg-scripts.git
    cd triliovault-cfg-scripts/
    git checkout stable/4.0 
    
    cp kolla-ansible/trilio-datamover-api/dmapi.conf.sample /etc/kolla/trilio-datamover-api/dmapi.conf
    [DEFAULT]
    dmapi_workers = 2
    # Get this parameter value from nova-api.conf
    transport_url = rabbit://openstack:[email protected]:5672//
    
    ##In the following paramater, edit IP address only. Use dmapi/nova-api nodes fixed ip.
    dmapi_link_prefix = http://11.11.11.95:8784
    dmapi_enabled_ssl_apis =
    dmapi_listen_port = 8784
    dmapi_enabled_apis = dmapi
    bindir = /usr/bin
    instance_name_template = instance-%08x
    ## dmapi_listen should be assigned with nova-api/dmapi node's fixed ip
    dmapi_listen = 11.11.11.95
    my_ip = 11.11.11.95
    rootwrap_config = /etc/dmapi/rootwrap.conf
    debug = True
    log_file = /var/log/kolla/trilio-datamover-api/dmapi.log
    log_dir = /var/log/kolla/trilio-datamover-api
    
    [wsgi]
    ssl_cert_file =
    ssl_key_file =
    api_paste_config = /etc/dmapi/api-paste.ini
    
    [database]
    # Get this parameter value from nova-api.conf
    connection = mysql+pymysql://nova:[email protected]:3306/nova
    
    [api_database]
    # Get this parameter value from nova-api.conf
    connection = mysql+pymysql://nova_api:[email protected]:3306/nova_api
    
    [keystone_authtoken]
    signing_dir = /var/cache/dmapi
    # Get these parameter values from nova-api.conf
    cafile =
    user_domain_name = Default
    insecure = True
    auth_uri = http://11.11.11.96:5000
    auth_url = http://11.11.11.96:35357
    auth_type = password
    project_domain_id = default
    user_domain_id = default
    project_name = service
    username = nova
    password = XZ2Do9MYEWivxZoiuKKHAc6wOJOPVGHzgXyYu9ic
    memcache_security_strategy = ENCRYPT
    memcache_secret_key = W8NudaXmPejmCcefr0I0WEcZExFnGrbd6I6ZOkOv
    memcached_servers = 11.11.11.95:11211
    
    [oslo_messaging_notifications]
    # Get this parameter value from nova-api.conf
    transport_url = rabbit://openstack:[email protected]:5672//
    driver = noop
    
    [oslo_middleware]
    enable_proxy_headers_parsing = true
    cp /etc/kolla/nova-api/nova.conf /etc/kolla/trilio-datamover-api/
    chmod -R 744 /etc/kolla/trilio-datamover-api/
    mkdir -p /var/log/kolla/trilio-datamover-api
    
    1. If openstack is based on CentOS :
    ##Change ownership of log directory to 42436:42436 - nova user and group id on container
    chmod 755 /var/log/kolla/trilio-datamover-api
    chown 42436:42436 /var/log/kolla/trilio-datamover-api
    
    2. If openstack is based on Ubuntu :
    ##Change ownership of log directory to 42486:42487- dmapi user and group id on container 
    chmod 755 /var/log/kolla/trilio-datamover-api 
    chown 42486:42487 /var/log/kolla/trilio-datamover-api
    ## Edit haproxy.cfg 
    vi /etc/kolla/haproxy/haproxy.cfg
    
    ## If SSL enabled on public interface of dmapi
    listen trilio_datamover_api
      bind <Keystone_virtual_ip>:8784  ssl crt /etc/haproxy/haproxy.pem
      server <controller_hostname_1> <controller_IP1>:8784 check inter 2000 rise 2 fall 5
      server <controller_hostname_2> <controller_IP2>:8784 check inter 2000 rise 2 fall 5
      server <controller_hostname_3> <controller_IP3>:8784 check inter 2000 rise 2 fall 5
    
    ## If SSL is not enabled on any interface
    listen trilio_datamover_api
      bind <Keystone_virtual_ip>:8784
      server <controller_hostname_1> <controller_IP1>:8784 check inter 2000 rise 2 fall 5
      server <controller_hostname_2> <controller_IP2>:8784 check inter 2000 rise 2 fall 5 
      server <controller_hostname_3> <controller_IP3>:8784 check inter 2000 rise 2 fall 5
    
    ## Restart haproxy container
    docker restart haproxy
    ## You need to edit OS_PLATFORM, TVAULT_VERSION, OPENSTACK_RELEASE_NAME. 
    ## <OS_PLATFORM> ubuntu or centos.
    ## <OPENSTACK_RELEASE_NAME> train
    ## <TVAULT_VERSION> : 4.0.92
    
    docker run --network host --name trilio_datamover_api -d --restart always -v /etc/kolla/trilio-datamover-api/nova.conf:/etc/nova/nova.conf \
    -v /etc/kolla/trilio-datamover-api/dmapi.conf:/etc/dmapi/dmapi.conf \
    -v /var/log/kolla/:/var/log/kolla/ \
    trilio/<OS_PLATFORM>-source-trilio-datamover-api:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME> /var/lib/kolla/venv/bin/python /usr/bin/dmapi-api
    root@trilio-Standard-PC-i440FX-PIIX-1996:~# docker ps | grep trilio_datamover_api
    736e4ac462c9  trilio/ubuntu-source-trilio-datamover-api:4.0.92  "/var/lib/kolla/venv/"  3 days ago  Up 23 hours  trilio_datamover_api
    docker login docker.io
    
    ## Pull Trilio Datamover container using following command.
    ## You need to edit OS_PLATFORM, TVAULT_VERSION, OPENSTACK_RELEASE_NAME.
    ## <OS_PLATFORM> ubuntu or centos.
    ## <OPENSTACK_RELEASE_NAME> train
    ## <TVAULT_VERSION> 4.0.92 or 4.0.115
    
    docker pull docker.io/trilio/<OS_PLATFORM>-source-trilio-datamover:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME>;
    mkdir -p /etc/kolla/trilio-datamover
    cp /etc/kolla/nova-compute/nova.conf /etc/kolla/trilio-datamover/
    ## Clone triliovault devops repository. It's public github repository.
    git clone https://github.com/trilioData/triliovault-cfg-scripts.git
    cd triliovault-cfg-scripts/ 
    git checkout <GITHUB_BRANCH>          // Check section: 1.i) plan for deployment for correct github branch name
    
    ## If backup storage type you want to use is 'NFS'
    cp kolla-ansible/trilio-datamover/tvault-contego.conf.nfs.sample /etc/kolla/trilio-datamover/tvault-contego.conf
    
    ## If backup storage type you want to use is 'S3'
    cp kolla-ansible/trilio-datamover/tvault-contego.conf.s3.sample /etc/kolla/trilio-datamover/tvault-contego.conf
    
    ## Set correct permissions
    chmod -R 744 /etc/kolla/trilio-datamover/
    [DEFAULT]
    vault_storage_type = nfs
    vault_storage_nfs_export = 192.168.1.34:/mnt/tvault/tvm
    vault_data_directory_old = /var/triliovault
    vault_data_directory = /var/triliovault-mounts
    log_file = /var/log/kolla/trilio-datamover/tvault-contego.log
    debug = False
    verbose = True
    max_uploads_pending = 3
    max_commit_pending = 3
    vault_s3_auth_version = DEFAULT
    vault_s3_access_key_id =
    vault_s3_secret_access_key =
    vault_s3_region_name = us-east-2
    vault_s3_bucket =
    qemu_agent_ping_timeout = 900
    
    [contego_sys_admin]
    helper_command = sudo /var/lib/kolla/venv/bin/privsep-helper
    
    [conductor]
    use_local = True
    mkdir -p /var/log/kolla/trilio-datamover
    
    ##Change ownership of log directory to 42436:42436 - nova user and group id on container
    chmod 755 /var/log/kolla/trilio-datamover
    chown 42436:42436 /var/log/kolla/trilio-datamover
    chmod o+rx /etc/ceph
    chmod o+r /etc/ceph/*
    [libvirt]
    images_rbd_ceph_conf = /etc/ceph/ceph.conf
    rbd_user = cinder
    
    [ceph]
    keyring_ext = .cinder.keyring
    ## In following docker run command you need to edit OS_PLATFORM, TVAULT_VERSION,vOPENSTACK_RELEASE_NAME.
    
    ## <OS_PLATFORM> ubuntu or centos.
    ## <OPENSTACK_RELEASE_NAME> train
    ## <TVAULT_VERSION> : 4.0.92 or 4.0.115
    
    ##If backup storage type is 'NFS'
    docker run --privileged --network host --name trilio_datamover -d \
    --restart always \
    -v /etc/kolla/trilio-datamover/tvault-contego.conf:/etc/tvault-contego/tvault-contego.conf \
    -v /etc/kolla/trilio-datamover/nova.conf:/etc/nova/nova.conf \
    -v /dev:/dev:rw \
    -v /etc/ceph:/etc/ceph:ro \
    -v nova_compute:/var/lib/nova/:rw \
    -v /var/log/kolla/:/var/log/kolla/ \
    -v iscsi_info:/etc/iscsi:rw -v /var/run/libvirt \
    trilio/<OS_PLATFORM>-source-trilio-datamover:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME> /opt/tvault/start_datamover_nfs
    
    ##If backup storage type is 'S3'
    docker run --privileged --network host --name trilio_datamover -d \
    --restart always \
    -v /etc/kolla/trilio-datamover/tvault-contego.conf:/etc/tvault-contego/tvault-contego.conf \
    -v /etc/kolla/trilio-datamover/nova.conf:/etc/nova/nova.conf \
    -v /dev:/dev:rw \
    -v /etc/ceph:/etc/ceph:ro \
    -v nova_compute:/var/lib/nova/:rw \
    -v /var/log/kolla/:/var/log/kolla/ \
    -v iscsi_info:/etc/iscsi:rw -v /var/run/libvirt \
    trilio/<OS_PLATFORM>-source-trilio-datamover:<TVAULT_VERSION>-<OPENSTACK_RELEASE_NAME> /opt/tvault/start_datamover_s3
    root@trilio-Standard-PC-i440FX-PIIX-1996:~# docker ps | grep trilio_datamover
    9d572974c75d  trilio/centos-source-trilio-datamover:4.0.92  "/opt/tvault/start_da"  2 days ago  Up 23 hours  trilio_datamover
    docker exec -it horizon /bin/bash
    curl -kO https://<TVAULT_VM_IP>/tvault-horizon-plugin-install.sh
    chmod +x tvault-horizon-plugin-install.sh
    # Login to horizon container if not already.
    docker exec -itu root horizon /bin/bash
    
    ## Run the trilio horizon plugin shell script 
    - Script will ask for python2/python3, you need to select the option as per your enviornment
    - Trilio horizon install script will ask for horizon's openstack_dashboard directory path if it's not at the default
     location - '/usr/shar/openstack-dashboard' For train ubuntu bionic, it's : '/var/lib/kolla/venv/lib/python2.7/site-packages'
    
    ./tvault-horizon-plugin-install.sh
    
    ## Exit the horizon container
    Ctrl + D
    ## vi /etc/kolla/horizon/local_settings
    
    Add this line : HORIZON_CONFIG['customization_module'] = 'dashboards.overrides'
    ## Restart the horizon container
    docker restart horizon
    yum --disablerepo=grafana
    [stack@tvm ~]$ mount | grep triliovault
    192.168.1.33:/mnt/tvault on /var/triliovault-mounts/MTkyLjE2OC4xLjMzOi9tbnQvdHZhdWx0 type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=180,retrans=2,sec=sys,clientaddr=192.168.10.10,lookupcache=none,local_lock=none,addr=192.168.1.33)
    # Login to nova_compute container with root user
    docker exec -itu root nova_compute /bin/bash
    
    # Install nfs-utils/nfs-common package using
    => If it's centos
    yum install nfs-utils
    => If it's ubuntu
    apt-get install nfs-common
    
    # Login to nova_libvirt container with root user
    docker exec -itu root nova_libvirt /bin/bash
    
    # Install nfs-utils/nfs-common package using yum
    => If it's centos 
    yum install nfs-utils 
    => If it's ubuntu
    apt-get install nfs-common
    mount -t nfs 192.168.1.33:/mnt/tvault /var/triliovault-mounts/MTkyLjE2OC4xLjMzOi9tbnQvdHZhdWx0
    # Get docker run logs for datamover container
    docker logs trilio_datamover
    
    ## Get docker logs for datamover api container
    docker logs trilio_datamover_api
    /var/log/kolla/trilio-datamover/tvault-contego.log
    /var/log/kolla/trilio-datamover-api/dmapi.log

    Installing on Ansible Openstack Train

    hashtag
    Installing Trilio Datamover-API container

    The installation of the Datamover-API, short dmapi, requires to create a new container, in which all necessary packages and the Trilio dmapi code are loaded.

    hashtag
    Create the dmapi container on Controller nodes

    Create lxc container for hosting dmapi service on controller nodes with below commands.

    hashtag
    Prepare the dmapi container for installation

    Add nova user and required directory on container controller_dmapi.

    Add required packages on container controller_dmapi.

    Copy nova.conf file from nova-api container to /etc/nova directory in dmapi container. Run the below command on controller nodes:

    Create a new interface with specific ip for dmapi container.

    Edit /var/lib/lxc/controller_dmapi/config and add below section as per network bridge available on the controller node.

    Restart the container with the below commands.

    hashtag
    Install the dmapi service

    Download and run the tvault-installation script inside the container.

    The script to be executed inside dmapi container, after the following changes have been done: Comment the 2 lines below and add a line below NOVA_VERSION = 20, as nova-manage doesn't work in Ansible Openstack.

    Run the script

    hashtag
    Verify the installation and set necessary configuration

    Verify in dmapi.conf domain name for the nova service user under keystone section. Check field values for project_domain_nameand user_domain_name and update those if not in keystone section

    If SSL is enabled then add the following section in dmapi.conf:

    Verify below entries are there in keystone policy.json file

    circle-info

    file path : /var/lib/lxc/controller_keystone_container/rootfs/etc/keystone

    Once verified above checks, start the dmapi service.

    hashtag
    Install Trilio Datamover on Compute nodes

    hashtag
    Prepare the installation

    Activate the virtual environment on the compute node.

    After activating the virtual environment, find out the location of compute.filters file.

    Download the installation script.

    Modify install script to use the same location for creating trilio filters.

    Also comment the 2 lines below and add a line below NOVA_VERSION = 20, as nova-manage doesn't work in Ansible Openstack.

    hashtag
    Install the Trilio datamover service

    Run install script and if you get prompt while installing, choose the default selection.

    circle-info

    If non-default values are selected, it will overwrite the current configuration and will impact nova-compute service.

    Make sure ExecStart value look like below in /etc/systemd/system/tvault-contego.service file.

    Use below commands to restart and verify the service.

    Use below command and check if nfs/s3 storage is mounted or not.

    hashtag
    Install Horizon Plugin into Horizon service

    List running containers on controller nodes and login to horizon container using the below command.

    Install curl package on the Horizon container if not present.

    Activate virtual environment on horizon container

    Download script to install horizon plugin on horizon container and run install script

    Install script will ask for the dashboard folder, provide below path

    Verify installation using below commands

    hashtag
    Haproxy configuration on controller nodes

    Refer to the keystone haproxy settings for dmapi haproxy.

    circle-info

    Haproxy config file can be found here: /etc/haproxy/haproxy.cfg

    A sample configuration is shown below.

    Check the syntax of the file and restart the service.

    ## On Ubuntu
    lxc-create -t ubuntu -n controller_dmapi       #Create the controller_dmapi container 
    
    ## On CentOS
    lxc-create -t centos -n controller_dmapi       #Create the controller_dmapi container
    
    lxc-start -n controller_dmapi                  #Start the controller_dmapi container
    lxc-attach -n controller_dmapi      # Login to container
    mkdir /etc/nova                     # create require /etc/nova directory
    useradd nova                        # create nova user
    id nova                             # check user added on container and aslo confirm the group nova added on container
    ## On Ubuntu
    sudo apt-get install software-properties-common
    sudo add-apt-repository cloud-archive:train
    sudo apt-get install python3-dev
    sudo apt-get install curl
    sudo apt-get install python3-nova
    
    ## On CentOs
    yum install centos-release-openstack-train
    yum install python-devel
    yum install python-nova
    cp /var/lib/lxc/controller_nova_api_container/rootfs/etc/nova/nova.conf /var/lib/lxc/controller_dmapi/rootfs/etc/nova/
    lxc.net.1.type = veth
    lxc.net.1.name = ens4
    lxc.net.1.link = br-mgmt
    lxc.net.1.hwaddr = 00:16:3e:de:52:31
    lxc.net.1.flags = up
    lxc.net.1.mtu = 1500
    lxc.net.1.ipv4.address = 172.26.0.104/22
    lxc-stop -n controller_dmapi
    lxc-start -n controller_dmapi
    wget https://<ipaddress-of-triliovault-controller>/tvault-contego-install.sh --no-check-certificate
    chmod +x tvault-contego-install.sh
    # NOVA_VERSION=`nova-manage version | awk -F. &#39;{print $1}&#39;`
    # NOVA_VERSION_ROCKY=`nova-manage version`
    NOVA_VERSION = 20
    ./tvault-contego-install.sh --install
    [keystone_authtoken]
     memcached_servers =
     signing_dir =
     cafile =
     project_domain_name = Default
     project_name = service
     user_domain_name = Default
     password = ************
     username = nova
     auth_url = http://172.26.0.3:5000
     auth_type = password
     user_domain_id = default
     project_domain_id = default
     auth_uri =
     memcached_servers =
    [oslo_messaging_rabbit]
    ssl = True
    rpc_conn_pool_size = 30
    "cloud_admin": "rule:admin_required and (is_admin_project:True or domain_id:CLOUD_DOMAIN_ID or project_id:SERVICE_PROJECT_ID)"
    service tvault-datamover-api start
    service tvault-datamover-api status
    source /openstack/venvs/nova-20.0.3.dev2/bin/activate
    (nova-20.0.3.dev2) root@compute:~# find / -name compute.filters
    /openstack/venvs/nova-20.0.3.dev2/etc/nova/rootwrap.d/compute.filters
    wget https://<ipaddress-of-triliovault-controller>/tvault-contego-install.sh --no-check-certificate
    chmod +x tvault-contego-install.sh
    1797              elif [ "$value" == 2 ]; then
    1798                  NOVA_COMPUTE_FILTERS_FILE="/openstack/venvs/nova-20.0.3.dev2/etc/nova/rootwrap.d/compute.filters"
    1799                  create_filter_file /openstack/venvs/nova-20.0.3.dev2/etc/nova/rootwrap.d/trilio.filters
    # NOVA_VERSION=`nova-manage version | awk -F. &#39;{print $1}&#39;`
    # NOVA_VERSION_ROCKY=`nova-manage version`
    NOVA_VERSION = 20
    ./tvault-contego-install.sh --install
    ExecStart=/openstack/venvs/nova-20.0.3.dev2/bin/python3 /usr/bin/tvault-contego --config-file=/etc/nova/nova.conf --config-file=/etc/tvault-contego/tvault-contego.conf
    systemctl daemon-reload
    service tvault-contego start
    service tvault-contego status
    df -h
    lxc-ls
     lxc-attach -n <name of horizon container>
    ## On Ubuntu
    apt-get install curl
    
    ## On CentOS
    yum install curl
    source /openstack/venvs/horizon-20.0.3.dev2/bin/activate
    wget https://<ipaddress-of-triliovault-controller>/tvault-horizon-plugin-install.sh --no-check-certificate
    chmod +x tvault-horizon-plugin-install.sh
    ./tvault-horizon-plugin-install.sh
    /openstack/venvs/horizon-20.0.3.dev2/lib/python3.6/site-packages/
    pip list | grep tvault
    frontend datamover_service-front-1
            bind trainubuntu.triliodata.demo:8784 ssl crt /etc/ssl/private/haproxy.pem ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
            option httplog
            option forwardfor except 127.0.0.0/8
            option http-server-close
            reqadd X-Forwarded-Proto:\ https
            mode http
            default_backend datamover_service-back
    
    frontend datamover_service-front-2
            bind 172.26.0.3:8784
            option httplog
            option forwardfor except 127.0.0.0/8
            option http-server-close
            mode http
            default_backend datamover_service-back
    
    
    backend datamover_service-back
            mode http
            balance leastconn
            stick store-request src
            stick-table type ip size 256k expire 30m
            option forwardfor
            option httplog
            option httpchk HEAD / HTTP/1.0\r\nUser-agent:\ osa-haproxy-healthcheck
            server controller_dmapi  172.26.0.104:8784 check port 8784 inter 12000 rise 1 fall 1
    haproxy -c -f /etc/haproxy/haproxy.conf
    systemctl restart haproxy

    Installing on RHOSP16.1

    The Red Hat Openstack Platform Directorarrow-up-right is the supported and recommended method to deploy and maintain any RHOSP installation.

    Trilio is integrating natively into the RHOSP Director. Manual deployment methods are not supported for RHOSP.

    hashtag
    Prepare for deployment

    Depending whether the RHOSP environment is already installed or is getting installed for the first time different steps are done to be able to deploy Trilio.

    circle-exclamation

    All commands need to be run as user 'stack'

    The following command clones the Trilio configscripts github.

    If your backup target is ceph S3 with SSL and SSL certificates are self signed or authorized by private CA, then user needs to provide CA chain certificate to validate the SSL requests. For that, user needs to rename his ca chain cert file to 's3-cert.pem' and copy it into directory - 'triliovault-cfg-scripts/redhat-director-scripts/redhat-director-scripts/puppet/trilio/files'

    The following commands upload the Trilio puppet module to the overcloud registry. The actual upload happens upon the next deployment.

    hashtag
    Update overcloud roles data file to include Trilio services

    Trilio contains of multiple services. Add these services to your roles_data.yaml.

    circle-info

    In case of uncostomized roles_data.yaml can the default file be found on the undercloud at:

    /usr/share/openstack-tripleo-heat-templates/roles_data.yaml

    Add the following services to the roles_data.yaml

    hashtag
    Trilio Datamover Api Service

    This service needs to share the same role as the nova-api service. In case of the pre-defined roles will the nova-api service run on the role Controller. In case of custom defined roles, it is necessary to use the role the nova-api service is using.

    Add the following line to the identified role:

    hashtag
    Trilio Datamover Service

    This service needs to share the same role as the nova-compute service. In case of the pre-defined roles will the nova-compute service run on the role Compute. In case of custom defined roles, it is necessary to use the role the nova-compute service is using.

    Add the following line to the identified role:

    hashtag
    Prepare Trilio container images

    circle-exclamation

    All commands need to be run as user 'stack'

    Trilio containers are pushed to 'RedHat Container Registry'. Registry URL: 'registry.connect.redhat.com'. Container pull urls are given below.

    Trilio Datamover container: registry.connect.redhat.com/trilio/trilio-datamover:4.0.116-rhosp16.1 Trilio Datamover Api Container: registry.connect.redhat.com/trilio/trilio-datamover-api:4.0.116-rhosp16.1 Trilio horizon plugin: registry.connect.redhat.com/trilio/trilio-horizon-plugin:4.0.116-rhosp16.1

    There are three registry methods available in RedHat Openstack Platform.

    1. Remote Registry

    2. Local Registry

    3. Satellite Server

    hashtag
    Remote Registry

    Follow this section when 'Remote Registry' is used.

    For this method it is not necessary to pull the containers in advance. It is only necessary to populate the trilio_env.yaml file with the Trilio container URLs from Redhat registry.

    Populate the trilio_env_osp16.yaml with container urls for:

    • Trilio Datamover container

    • Trilio Datamover api container

    • Trilio Horizon Plugin

    circle-info

    trilio_env_osp16.yaml will be available in triliovault-cfg-scripts/redhat-director-scripts/

    hashtag
    Local Registry

    Follow this section when 'local registry' is used on the undercloud.

    In this case it is necessary to pull the Trilio containers to the undercloud registry. Trilio provides shell scripts which will pull the containers from 'registry.connect.redhat.com' to the undercloud and updates the trilio_env_osp16.yaml with the values for the datamover and datamover api containers.

    circle-info

    The script assumes that the undercloud container registry is running on port 8787. If the registry is running on a different port, the script needs to be adjusted manually.

    The changes can be verified using the following commands.

    hashtag
    Satellite Server

    Follow this section when a Satellite Server is used for the container registry.

    Pull the Trilio containers on the Red Hat Satellite using the given Red Hat registry URLs.

    Populate the trilio_env_osp16.yaml with container urls for:

    • Trilio Datamover container

    • Trilio Datamover api container

    hashtag
    Provide environment details in trilio-env_osp16.yaml

    Provide backup target details and other necessary details in the provided environment file. This environment file will be used in the overcloud deployment to configure Trilio components. Container image names have already been populated in the preparation of the container images. Still it is recommended to verify the container URLs.

    The following information are required additionally:

    • Network for the datamover api

    • Backup target type {nfs/s3}

    • In case of NFS

    hashtag
    Deploy overcloud with trilio environment

    Use the following heat environment file and roles data file in overcloud deploy command:

    1. trilio_env_osp16.yaml

    2. roles_data.yaml

    To include new environment files use '-e' option and for roles data file use '-r' option. An example overcloud deploy command is shown below:

    hashtag
    Verify deployment

    triangle-exclamation

    If the containers are in restarting state or not listed by the following command then your deployment is not done correctly. Please recheck if you followed the complete documentation.

    hashtag
    On Controller node

    Make sure Trilio dmapi and horizon containers are in a running state and no other Trilio container is deployed on controller nodes.

    hashtag
    On Compute node

    Make sure Trilio datamover container is in running state and no other Trilio container is deployed on compute nodes.

    hashtag
    On the node with Horizon service

    Make sure horizon container is in running state. Please note that 'Horizon' container is replaced with Trilio Horizon container. This container will have latest OpenStack horizon + Trilio's horizon plugin.

    hashtag
    Configure Trilio Appliance

    Once RHOSP16.1 Installation steps have completed successfully, follow the instructions below to now configure the Trilio Appliance.

    hashtag
    Change the nova user id on the Trilio Nodes

    In RHOSP, 'nova' user id on nova-compute docker container is set to '42436'. The 'nova' user id on the Trilio nodes need to be set the same. Do the following steps on all Trilio nodes:

    1. Download the shell script that will change the user id

    2. Assign executable permissions

    3. Execute the script

    hashtag
    Run the Trilio configurator

    Follow .

    hashtag
    Change the workloadmgr.conf to use the right mountpoint

    circle-exclamation

    It is necessary to first configure the Trilio appliance, before the steps of this section can be done.

    RHOSP16 is using a different mount point in its datamover containers than other Openstack distribution. It is necessary to adjust the mountpoint of the Trilio Nodes to match this. If the mountpoints are not getting aligned, will files created by the datamover and read by the Trilio appliance not match in their paths and backup and restore processes will fail.

    Please follow these steps to align the mountpoints:

    1. Edit /etc/workloadmgr/workloadmgr.conf file

    2. Set parameter 'vault_data_directory' to '/var/lib/nova/triliovault-mounts'

    3. create the directory for the mountpoint

    hashtag
    Troubleshooting for overcloud deployment failures

    Trilio components will be deployed using puppet scripts.

    In case of the overcloud deployment failing does the following command provide the list of errors:

    Further commands that can help identifying any errors.

    In case of the trilio datamover api container is failing to start or stuck in restart, check these logs:

    In case of Trilio datamover container failing to start or being stuck in restart, check these logs:

    hashtag
    Cinder backend is Ceph - additional steps

    hashtag
    Add Ceph details to configuration file

    If Cinder backend is Ceph it is necessary to manually add the ceph details to tvault-contego.conf on all compute nodes.

    The file can be found here: /var/lib/config-data/puppet-generated/triliodm/etc/tvault-contego/tvault-contego.conf

    Add the following information:

    circle-info

    The same block of information can be found in the nova.conf file.

    hashtag
    Restart the datamover container

    list of NFS Shares

  • NFS options

  • In case of S3

    • s3 type {amazon_s3/ceph_s3}

    • Access key

    • Secret key

    • S3 Region name

    • S3 Bucket name

    • S3 Endpoint URL

    • S3 SSL Enabled {true/false}

    • S3 SSL Cert

  • Verify that 'nova' user and group id has changed to '42436'
    assign the created directory to nova:nova
  • unmount the old mountpoint

  • Update the Trilio configurator

  • Restart the Trilio services

  • Verify the mountpoint has been configured correctly

  • this documentation to configure Trilio Appliance
    cd /home/stack
    git clone -b v4.0maintenance https://github.com/trilioData/triliovault-cfg-scripts.git
    cd triliovault-cfg-scripts/redhat-director-scripts/
    
    cp s3-cert.pem /home/stack/triliovault-cfg-scripts/redhat-director-scripts/puppet5/trilio/files/
    # ./upload_puppet_module_puppet5.sh
    Creating tarball...
    Tarball created.
    Creating heat environment file: /home/stack/.tripleo/environments/puppet-modules-url.yaml
    Uploading file to swift: /tmp/puppet-modules-8Qjya2X/puppet-modules.tar.gz
    +-----------------------+---------------------+----------------------------------+
    | object                | container           | etag                             |
    +-----------------------+---------------------+----------------------------------+
    | puppet-modules.tar.gz | overcloud-artifacts | 368951f6a4d39cfe53b5781797b133ad |
    +-----------------------+---------------------+----------------------------------+
    
    ## Make sure that
    ls -ll /home/stack/.tripleo/environments/puppet-modules-url.yaml
    'OS::TripleO::Services::TrilioDatamoverApi'
    'OS::TripleO::Services::TrilioDatamover' 
    (undercloud) [stack@undercloud redhat-director-scripts]$ grep 'Image' trilio_env_osp16.yaml
       DockerTrilioDatamoverImage: registry.connect.redhat.com/trilio/trilio-datamover:4.0.116-rhosp16.1
       DockerTrilioDmApiImage: registry.connect.redhat.com/trilio/trilio-datamover-api:4.0.116-rhosp16.1
       ContainerHorizonImage: registry.connect.redhat.com/trilio/trilio-horizon-plugin:4.0.116-rhosp16.1
    ## Command to prepare, push trilio container images to undercloud registry: 
    sudo ./podman_prepare_images.sh <UNDERCLOUD_REGISTRY_HOSTNAME> <CONTAINER_TAG> 
    
    
    ## Run following command to find 'UNDERCLOUD_REGISTRY_HOSTNAME'. Hostname is higlighted in bold format in below example.
    $ openstack tripleo container image list | grep keystone
    | docker://trilio-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-keystone:16.0-82                       |
    | docker://trilio-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-barbican-keystone-listener:16.0-84   
    
    ## 'CONTAINER_TAG' format: 4.0.116-rhosp16.1
    ## For example. If Trilio version='4.0.116',  then 'CONTAINER_TAG'=4.0.116-rhosp16.1
    
    ## Example:
    sudo ./podman_prepare_images.sh trilio-undercloud.ctlplane.localdomain 4.0.116-rhosp16.1
    (undercloud) [stack@undercloud redhat-director-scripts]$ openstack tripleo container image list | grep trilio
    | docker://undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:4.0.116-rhosp16.1                       |                        |
    | docker://undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:4.0.116-rhosp16.1                   |                   |
    | docker://undercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:4.0.116-rhosp16.1                  |
    
    -----------------------------------------------------------------------------------------------------
    
    (undercloud) [stack@undercloud redhat-director-scripts]$ grep 'Image' trilio_env_osp16.yaml
       DockerTrilioDatamoverImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:4.0.116-rhosp16.1
       DockerTrilioDmApiImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:4.0.116-rhosp16.1
       ContainerHorizonImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:4.0.116-rhosp16.1
    
    (undercloud) [stack@undercloud redhat-director-scripts]$ grep 'Image' trilio_env_osp16.yaml
       DockerTrilioDatamoverImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:4.0.116-rhosp16.1
       DockerTrilioDmApiImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:4.0.116-rhosp16.1
       ContainerHorizonImage: undercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:4.0.116-rhosp16.1
    resource_registry:
      OS::TripleO::Services::TrilioDatamover: docker/services/trilio-datamover-osp16.yaml
      OS::TripleO::Services::TrilioDatamoverApi: docker/services/trilio-datamover-api-osp16.yaml
    
    parameter_defaults:
    
       ## Enable Trilio's quota functionality on horizon
       ExtraConfig:
         horizon::customization_module: 'dashboards.overrides'
    
       ## Define network map for trilio datamover api service
       ServiceNetMap:
           TrilioDatamoverApiNetwork: internal_api
    
       ## Trilio container pull urls
       DockerTrilioDatamoverImage: devundercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:4.0.92-rhosp16
       DockerTrilioDmApiImage: devundercloud.ctlplane.localdomain:8787/trilio/trilio-datamover-api:4.0.92-rhosp16
    
       ## If you do not want Trilio's horizon plugin to replace your horizon container, just comment following line.
       ContainerHorizonImage: devundercloud.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:4.0.92-rhosp16
    
    
       ## Backup target type nfs/s3, used to store snapshots taken by triliovault
       BackupTargetType: 'nfs'
    
       ## For backup target 'nfs'
       NfsShares: '192.168.122.101:/opt/tvault'
       NfsOptions: 'nolock,soft,timeo=180,intr,lookupcache=none'
    
       ## For backup target 's3'
       ## S3 type: amazon_s3/ceph_s3
       S3Type: 'amazon_s3'
    
       ## S3 access key
       S3AccessKey: ''
      
       ## S3 secret key
       S3SecretKey: ''
    
       ## S3 region, if your s3 does not have any region, just keep the parameter as it is
       S3RegionName: ''
    
       ## S3 bucket name
       S3Bucket: ''
    
       ## S3 endpoint url, not required for Amazon S3, keep it as it is
       S3EndpointUrl: ''
    
       ## If SSL enabled on S3 url, not required for Amazon S3, just keep it as it is
       S3SslEnabled: false
    
       ## If S3 backend is not Amazon S3 and SSL is enabled on S3 endpoint URL and SSL certificates are self signed, then
       ## user need to set this parameter value to: '/etc/tvault-contego/s3-cert.pem', otherwise keep it's value as empty string.
       S3SslCert: ''
    
       ## Don't edit following parameter
       EnablePackageInstall: True
    openstack overcloud deploy --templates \
    --libvirt-type qemu \
    --ntp-server 192.168.1.34 \
    -e /home/stack/triliovault-cfg-scripts/redhat-director-scripts/trilio_env_osp16.yaml \
    -e /home/stack/templates/node-info.yaml \
    -e /home/stack/containers-prepare-parameter.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-tls.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/inject-trust-anchor.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \
    -r /usr/share/openstack-tripleo-heat-templates/roles_data.yaml
    [root@overcloud-controller-0 heat-admin]# podman ps | grep trilio
    26fcb9194566  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-datamover-api:4.0.116-rhosp16.1  kolla_start  5 days ago  Up 5 days ago  trilio_dmapi
    094971d0f5a9  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:4.0.116-rhosp16.1  kolla_start  5 days ago  Up 5 days ago  horizon
    [root@overcloud-novacompute-0 heat-admin]# podman ps | grep trilio
    b1840444cc59  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-datamover:4.0.116-rhosp16.1  kolla_start  5 days ago  Up 5 days ago  tilio_datamover
    [root@overcloud-controller-0 heat-admin]# podman ps | grep horizon
    094971d0f5a9  rhosptrainqa.ctlplane.localdomain:8787/trilio/trilio-horizon-plugin:4.0.116-rhosp16.1  kolla_start  5 days ago  Up 5 days ago  horizon
    # curl -O https://raw.githubusercontent.com/trilioData/triliovault-cfg-scripts/master/common/nova_userid.sh
    # chmod +x nova_userid.sh
    # ./nova_userid.sh
    # id nova
      uid=42436(nova) gid=42436(nova) groups=42436(nova),990(libvirt),36(kvm)
    # cat /etc/workloadmgr/workloadmgr.conf | grep vault_data_directory
    vault_data_directory = /var/lib/nova/triliovault-mounts
    vault_data_directory_old = /var/triliovault
    
    # mkdir -p /var/lib/nova/triliovault-mounts
    
    # chown nova:nova /var/lib/nova/triliovault-mounts
    
    # umount /var/triliovault-mounts
    
    # cat /home/stack/myansible/lib/python3.6/site-packages/workloadmgr/tvault_configurator/ansible-play/roles/ansible-workloadmgr/templates/workloadmgr.conf.j2 | grep vault_data_directory
    vault_data_directory = /var/lib/nova/triliovault-mounts
    
    #pcs resource restart wlm-cron 
    
    #pcs resource restart wlm-scheduler 
    
    #systemctl restart wlm-api 
    
    #systemctl restart wlm-workloads
    
    #df-h
    Filesystem             Size  Used Avail Use% Mounted on
    devtmpfs               3.8G     0  3.8G   0% /dev
    tmpfs                  3.8G   38M  3.8G   1% /dev/shm
    tmpfs                  3.8G  427M  3.4G  12% /run
    tmpfs                  3.8G     0  3.8G   0% /sys/fs/cgroup
    /dev/vda1               40G  8.8G   32G  22% /
    tmpfs                  773M     0  773M   0% /run/user/996
    tmpfs                  773M     0  773M   0% /run/user/0
    10.10.2.20:/upstream  1008G  704G  254G  74% /var/lib/nova/triliovault-mounts/MTAuMTAuMi4yMDovdXBzdHJlYW0= 
    openstack stack failures list overcloud
    heat stack-list --show-nested -f "status=FAILED"
    heat resource-list --nested-depth 5 overcloud | grep FAILED
    sudo podman logs trilio_dmapi
    tailf /var/log/containers/trilio-datamover-api/dmapi.log
    sudo podman logs trilio_datamover
    tailf /var/log/containers/trilio-datamover/tvault-contego.log
    [libvirt]
    images_rbd_ceph_conf = /etc/ceph/ceph.conf
    rbd_user = <rbd_user>
    
    [ceph] 
    keyring_ext = .<rbd_user>.keyring
    sudo podman restart trilio_datamover