1 of 1

Example runbook for Disaster Recovery using NFS

This runbook will demonstrate how to set up Disaster Recovery with Trilio for a given scenario.

The chosen scenario is following an actively used Trilio customer environment.

Scenario

There are two OpenStack clouds available "OpenStack Cloud A" and OpenStack Cloud B". "OpenStack Cloud B" is the Disaster Recovery restore point of "OpenStack Cloud A" and vice versa. Both clouds have an independent Trilio installation integrated. These Trilio installations are writing their Backups to NFS targets. "Trilio A" is writing to "NFS A1" and "Trilio B" is writing to "NFS B1". The NFS Volumes used are getting synced against another NFS Volume on the other side. "NFS A1" is syncing with "NFS B2" and "NFS B1" is syncing with "NFS A2". The syncing process is set up independently from Trilio and will always favor the newer dataset.

This scenario will cover the Disaster Recovery of a single Workload and a complete Cloud. All processes are done be the OpenStack administrator.

Prerequisites for the Disaster Recovery process

This runbook will assume that the following is true:

"OpenStack Cloud A" and "OpenStack Cloud B" both have an active Trilio installation with a valid license
"OpenStack Cloud A" and "OpenStack Cloud B" have free resources to host additional VMs
"OpenStack Cloud A" and "OpenStack Cloud B" have Tenants/Projects available that are the designated restore points for Tenant/Projects of the other side

For ease of writing will this runbook assume from here on, that "OpenStack Cloud A" is down and the Workloads are getting restored into "OpenStack Cloud B".

In the case of the usage of shared Tenant networks, beyond the floating IP, the following additional requirement is needed: All Tenant Networks, Routers, Ports, Floating IPs, and DNS Zones are created

Disaster Recovery of a single Workload

A single Workload can do a Disaster Recovery in this Scenario, while both Clouds are still active. To do so the following high-level process needs to be followed:

Copy the Workload directories to the configured NFS Volume
Make the right Mount-Paths available
Reassign the Workload
Restore the Workload

Copy the Workload directories to the configured NFS Volume

This process only shows how to get a Workload from "OpenStack Cloud A" to "OpenStack Cloud B". The vice versa process is similar.

As only a single Workload is to be recovered it is more efficient to copy the data of that single Workload over to the "NFS B1" Volume, which is used by "Trilio B".

Mount "NFS B2" Volume to a Trilio VM

It is recommended to use the Trilio VM as a connector between both NFS Volumes, as the nova user is available on the Trilio VM.

Identify the Workload on the "NFS B2" Volume

Trilio Workloads are identified by their ID und which they are stored on the Backup Target. See below example:

In the case that the Workload ID is not known can available Metadata inside the Workload directories be used to identify the correct Workload.

Copy the Workload

The identified workload needs to be copied with all subdirectories and files. Afterward, it is necessary to adjust the ownership to nova:nova with the right permissions.

Make the Mount-Paths available

Trilio backups are using qcow2 backing files, which make every incremental backup a full synthetic backup. These backing files can be made visible using the qemu-img tool.

The MTAuMTAuMi4yMDovdXBzdHJlYW0= part of the backing file path is the base64 hash value, which will be calculated upon the configuration of a Trilio installation for each provided NFS-Share.

This hash value is calculated based on the provided NFS-Share path: <NFS_IP>/<path> If even one character in the NFS-Share path is different between the provided NFS-Share paths a completely different hash value is generated.

Workloads, that have been moved between NFS-Shares, require that their incremental backups can follow the same path as on their original Source Cloud. To achieve this it is necessary to create the mount path on all compute nodes of the Target Cloud.

Afterwards a mount bind is used to make the workloads data accessible over the old and the new mount path. The following example shows the process of how to successfully identify the necessary mount points and create the mount bind.

Identify the base64 hash values

The used hash values can be calculated using the base64 tool in any Linux distribution.

Create and bind the paths

Based on the identified base64 hash values the following paths are required on each Compute node.

/var/triliovault-mounts/MTAuMTAuMi4yMDovdXBzdHJlYW1fc291cmNl

and

/var/triliovault-mounts/MTAuMjAuMy4yMjovdXBzdHJlYW1fdGFyZ2V0

In the scenario of this runbook is the workload coming from the NFS_A1 NFS-Share, which means the mount path of that NFS-Share needs to be created and bound to the Target Cloud.

To keep the desired mount past a reboot it is recommended to edit the fstab of all compute nodes accordingly.

Reassign the workload

Trilio workloads have clear ownership. When a workload is moved to a different cloud it is necessary to change the ownership. The ownership can only be changed by OpenStack administrators.

Add admin-user to required domains and projects

To fulfill the required tasks an admin role user is used. This user will be used until the workload has been restored. Therefore, it is necessary to provide this user access to the desired Target Project on the Target Cloud.

Discover orphaned Workloads from NFS-Storage of Target Cloud

Each Trilio installation maintains a database of workloads that are known to the Trilio installation. Workloads that are not maintained by a specific Trilio installation, are from the perspective of that installation, orphaned workloads. An orphaned workload is a workload accessible on the NFS-Share, that is not assigned to any existing project in the Cloud the Trilio installation is protecting.

List available projects on Target Cloud in the Target Domain

The identified orphaned workloads need to be assigned to their new projects. The following provides the list of all available projects viewable by the used admin-user in the target_domain.

List available users on the Target Cloud in the Target Project that have the right backup trustee role

To allow project owners to work with the workloads as well will they get assigned to a user with the backup trustee role that is existing in the target project.

Reassign the workload to the target project

Now that all informations have been gathered the workload can be reassigned to the target project.

Verify the workload is available at the desired target_project

After the workload has been assigned to the new project it is recommended to verify the workload is managed by the Target Trilio and is assigned to the right project and user.

Restore the workload

The reassigned workload can be restored using Horizon following the procedure described .

This runbook will continue on the CLI only path.

Prepare the selective restore by getting the snapshot information

To be able to do the necessary selective restore a few pieces of information about the snapshot to be restored are required. The following process will provide all necessary information.

List all Snapshots of the workload to restore to identify the snapshot to restore

Get Snapshot Details with network details for the desired snapshot

Get Snapshot Details with disk details for the desired Snapshot

Prepare the selective restore by creating the restore.json file

The selective restore is using a restore.json file for the CLI command. This restore.json file needs to be adjusted according to the desired restore.

Run the selective restore

To do the actual restore use the following command:

Verify the restore

To verify the success of the restore from a Trilio perspective the restore status is checked.

Clean up

After the Desaster Recovery Process has been successfully completed it is recommended to bring the TVM installation back into its original state to be ready for the next DR process.

Delete the workload

Delete the workload that got restored.

Remove the database entry

The Trilio database is following the OpenStack standard of not deleting any database entries upon deletion of the cloud object. Any Workload, Snapshot or Restore, which gets deleted, is marked as deleted only.

To allow the Trilio installation to be ready for another disaster recovery it is necessary to completely delete the entries of the Workloads, which have been restored.

Trilio does provide and maintain a script to safely delete workload entries and all connected entities from the Trilio database.

This script can be found here:

Remove the admin user from the project

After all restores for the target project have been achieved it is recommended to remove the used admin user from the project again.

Disaster Recovery of a complete cloud

This Scenario will cover the Disaster Recovery of a full cloud. It is assumed that the source cloud is down or lost completly. To do the disaster recovery the following high-level process needs to be followed:

Reconfigure the Target Trilio installation
Make the right Mount-Paths available
Reassign the Workload
Restore the Workload

Reconfigure the Target Trilio installation

Before the Desaster Recovery Process can start is it necessary to make the backups to be restored available for the Trilio installation. The following steps need to be done to completely reconfigure the Trilio installation.

During the reconfiguration process will all backups of the Target Region be on hold and it is not recommended to create new Backup Jobs until the Desaster Recovery Process has finished and the original Trilio configuration has been restored.

Add NFS B2 to the Trilio Appliance Cluster

To add the NFS-Vol2 to the Trilio Appliance cluster the Trilio can either be to use both NFS Volumes or it is possible to edit the configuration file and then restart all services. This procedure describes how to edit the conf file and restart the services. This needs to be repeated on every Trilio Appliance.

Edit the workloadmgr.conf

Look for the line defining the NFS mounts

Add NFS B2 to that as comma-seperated list. Space is not necessary, but can be set.

Write and close the workloadmgr.conf

Restart the wlm-workloads service

Add NFS B2 to the Trilio Datamovers

Trilio is integrating natively into the OpenStack deployment tools. When using the Red Hat director or JuJu charms it is recommended to adapt the environment files for these orchestrators and update the Datamovers through them.

To add the NFS B2 to the Trilio Datamovers manually the tvault-contego.conf file needs to be edited and the service restarted.

Edit the tvault-contego.conf

Look for the line defining the NFS mounts

Add NFS B2 to that as comma-seperated list. Space is not necessary, but can be set.

Write and close the tvault-contego.conf

Restart the tvault-contego service

Make the Mount-Paths available

Trilio backups are using qcow2 backing files, which make every incremental backup a full synthetic backup. These backing files can be made visible using the qemu-img tool.

The MTAuMTAuMi4yMDovdXBzdHJlYW0= part of the backing file path is the base64 hash value, which will be calculated upon the configuration of a Trilio installation for each provided NFS-Share.

Identify the base64 hash values

The used hash values can be calculated using the base64 tool in any Linux distribution.

Create and bind the paths

Based on the identified base64 hash values the following paths are required on each Compute node.

/var/triliovault-mounts/MTAuMTAuMi4yMDovdXBzdHJlYW1fc291cmNl

and

/var/triliovault-mounts/MTAuMjAuMy4yMjovdXBzdHJlYW1fdGFyZ2V0

In the scenario of this runbook is the workload coming from the NFS_A1 NFS-Share, which means the mount path of that NFS-Share needs to be created and bound to the Target Cloud.

To keep the desired mount past a reboot it is recommended to edit the fstab of all compute nodes accordingly.

Reassign the workload

Trilio workloads have clear ownership. When a workload is moved to a different cloud it is necessary to change the ownership. The ownership can only be changed by OpenStack administrators.

Add admin-user to required domains and projects

Discover orphaned Workloads from NFS-Storage of Target Cloud

List available projects on Target Cloud in the Target Domain

The identified orphaned workloads need to be assigned to their new projects. The following provides the list of all available projects viewable by the used admin-user in the target_domain.

List available users on the Target Cloud in the Target Project that have the right backup trustee role

To allow project owners to work with the workloads as well will they get assigned to a user with the backup trustee role that is existing in the target project.

Reassign the workload to the target project

Now that all informations have been gathered the workload can be reassigned to the target project.

Verify the workload is available at the desired target_project

After the workload has been assigned to the new project it is recommended to verify the workload is managed by the Target Trilio and is assigned to the right project and user.

Restore the workload

The reassigned workload can be restored using Horizon following the procedure described .

This runbook will continue on the CLI only path.

Prepare the selective restore by getting the snapshot information

To be able to do the necessary selective restore a few pieces of information about the snapshot to be restored are required. The following process will provide all necessary information.

List all Snapshots of the workload to restore to identify the snapshot to restore

Get Snapshot Details with network details for the desired snapshot

Get Snapshot Details with disk details for the desired Snapshot

Prepare the selective restore by creating the restore.json file

The selective restore is using a restore.json file for the CLI command. This restore.json file needs to be adjusted according to the desired restore.

Run the selective restore

To do the actual restore use the following command:

Verify the restore

To verify the success of the restore from a Trilio perspective the restore status is checked.

Reconfigure the Target Trilio installation back to the original one

After the Desaster Recovery Process has finished it is necessary to return the Trilio installation to its original configuration. The following steps need to be done to completely reconfigure the Trilio installation.

Delete NFS B2 to the Trilio Appliance Cluster

Edit the workloadmgr.conf

Look for the line defining the NFS mounts

Delete NFS B2 from the comma-seperated list

Write and close the workloadmgr.conf

Restart the wlm-workloads service

Delete NFS B2 to the Trilio Datamovers

To add the NFS B2 to the Trilio Datamovers manually the tvault-contego.conf file needs to be edited and the service restarted.

Edit the tvault-contego.conf

Look for the line defining the NFS mounts

Add NFS B2 to that as comma-seperated list. Space is not necessary, but can be set.

Write and close the tvault-contego.conf

Restart the tvault-contego service

Clean up

After the Desaster Recovery Process has been successfully completed and the Trilio installation reconfigured to its original state, it is recommended to do the following additional steps to be ready for the next Disaster Recovery process.

Remove the database entry

To allow the Trilio installation to be ready for another disaster recovery it is necessary to completely delete the entries of the Workloads, which have been restored.

Trilio does provide and maintain a script to safely delete workload entries and all connected entities from the Trilio database.

This script can be found here:

Remove the admin user from the project

After all restores for the target project have been achieved it is recommended to remove the used admin user from the project again.

Example runbook for Disaster Recovery using NFS

This runbook will demonstrate how to set up Disaster Recovery with Trilio for a given scenario.

The chosen scenario is following an actively used Trilio customer environment.

Scenario

This scenario will cover the Disaster Recovery of a single Workload and a complete Cloud. All processes are done be the OpenStack administrator.

Prerequisites for the Disaster Recovery process

This runbook will assume that the following is true:

"OpenStack Cloud A" and "OpenStack Cloud B" both have an active Trilio installation with a valid license
"OpenStack Cloud A" and "OpenStack Cloud B" have free resources to host additional VMs
"OpenStack Cloud A" and "OpenStack Cloud B" have Tenants/Projects available that are the designated restore points for Tenant/Projects of the other side

For ease of writing will this runbook assume from here on, that "OpenStack Cloud A" is down and the Workloads are getting restored into "OpenStack Cloud B".

Disaster Recovery of a single Workload

A single Workload can do a Disaster Recovery in this Scenario, while both Clouds are still active. To do so the following high-level process needs to be followed:

Copy the Workload directories to the configured NFS Volume
Make the right Mount-Paths available
Reassign the Workload
Restore the Workload

Copy the Workload directories to the configured NFS Volume

This process only shows how to get a Workload from "OpenStack Cloud A" to "OpenStack Cloud B". The vice versa process is similar.

As only a single Workload is to be recovered it is more efficient to copy the data of that single Workload over to the "NFS B1" Volume, which is used by "Trilio B".

Mount "NFS B2" Volume to a Trilio VM

It is recommended to use the Trilio VM as a connector between both NFS Volumes, as the nova user is available on the Trilio VM.

Identify the Workload on the "NFS B2" Volume

Trilio Workloads are identified by their ID und which they are stored on the Backup Target. See below example:

In the case that the Workload ID is not known can available Metadata inside the Workload directories be used to identify the correct Workload.

Copy the Workload

The identified workload needs to be copied with all subdirectories and files. Afterward, it is necessary to adjust the ownership to nova:nova with the right permissions.

Make the Mount-Paths available

Trilio backups are using qcow2 backing files, which make every incremental backup a full synthetic backup. These backing files can be made visible using the qemu-img tool.

The MTAuMTAuMi4yMDovdXBzdHJlYW0= part of the backing file path is the base64 hash value, which will be calculated upon the configuration of a Trilio installation for each provided NFS-Share.

Identify the base64 hash values

The used hash values can be calculated using the base64 tool in any Linux distribution.

Create and bind the paths

Based on the identified base64 hash values the following paths are required on each Compute node.

/var/triliovault-mounts/MTAuMTAuMi4yMDovdXBzdHJlYW1fc291cmNl

and

/var/triliovault-mounts/MTAuMjAuMy4yMjovdXBzdHJlYW1fdGFyZ2V0

In the scenario of this runbook is the workload coming from the NFS_A1 NFS-Share, which means the mount path of that NFS-Share needs to be created and bound to the Target Cloud.

To keep the desired mount past a reboot it is recommended to edit the fstab of all compute nodes accordingly.

Reassign the workload

Trilio workloads have clear ownership. When a workload is moved to a different cloud it is necessary to change the ownership. The ownership can only be changed by OpenStack administrators.

Add admin-user to required domains and projects

Discover orphaned Workloads from NFS-Storage of Target Cloud

List available projects on Target Cloud in the Target Domain

The identified orphaned workloads need to be assigned to their new projects. The following provides the list of all available projects viewable by the used admin-user in the target_domain.

List available users on the Target Cloud in the Target Project that have the right backup trustee role

To allow project owners to work with the workloads as well will they get assigned to a user with the backup trustee role that is existing in the target project.

Reassign the workload to the target project

Now that all informations have been gathered the workload can be reassigned to the target project.

Verify the workload is available at the desired target_project

After the workload has been assigned to the new project it is recommended to verify the workload is managed by the Target Trilio and is assigned to the right project and user.

Restore the workload

The reassigned workload can be restored using Horizon following the procedure described .

This runbook will continue on the CLI only path.

Prepare the selective restore by getting the snapshot information

To be able to do the necessary selective restore a few pieces of information about the snapshot to be restored are required. The following process will provide all necessary information.

List all Snapshots of the workload to restore to identify the snapshot to restore

Get Snapshot Details with network details for the desired snapshot

Get Snapshot Details with disk details for the desired Snapshot

Prepare the selective restore by creating the restore.json file

The selective restore is using a restore.json file for the CLI command. This restore.json file needs to be adjusted according to the desired restore.

Run the selective restore

To do the actual restore use the following command:

Verify the restore

To verify the success of the restore from a Trilio perspective the restore status is checked.

Clean up

After the Desaster Recovery Process has been successfully completed it is recommended to bring the TVM installation back into its original state to be ready for the next DR process.

Delete the workload

Delete the workload that got restored.

Remove the database entry

To allow the Trilio installation to be ready for another disaster recovery it is necessary to completely delete the entries of the Workloads, which have been restored.

Trilio does provide and maintain a script to safely delete workload entries and all connected entities from the Trilio database.

This script can be found here:

Remove the admin user from the project

After all restores for the target project have been achieved it is recommended to remove the used admin user from the project again.

Disaster Recovery of a complete cloud

Reconfigure the Target Trilio installation
Make the right Mount-Paths available
Reassign the Workload
Restore the Workload

Reconfigure the Target Trilio installation

Add NFS B2 to the Trilio Appliance Cluster

Edit the workloadmgr.conf

Look for the line defining the NFS mounts

Add NFS B2 to that as comma-seperated list. Space is not necessary, but can be set.

Write and close the workloadmgr.conf

Restart the wlm-workloads service

Add NFS B2 to the Trilio Datamovers

To add the NFS B2 to the Trilio Datamovers manually the tvault-contego.conf file needs to be edited and the service restarted.

Edit the tvault-contego.conf

Look for the line defining the NFS mounts

Add NFS B2 to that as comma-seperated list. Space is not necessary, but can be set.

Write and close the tvault-contego.conf

Restart the tvault-contego service

Make the Mount-Paths available

Trilio backups are using qcow2 backing files, which make every incremental backup a full synthetic backup. These backing files can be made visible using the qemu-img tool.

The MTAuMTAuMi4yMDovdXBzdHJlYW0= part of the backing file path is the base64 hash value, which will be calculated upon the configuration of a Trilio installation for each provided NFS-Share.

Identify the base64 hash values

The used hash values can be calculated using the base64 tool in any Linux distribution.

Create and bind the paths

Based on the identified base64 hash values the following paths are required on each Compute node.

/var/triliovault-mounts/MTAuMTAuMi4yMDovdXBzdHJlYW1fc291cmNl

and

/var/triliovault-mounts/MTAuMjAuMy4yMjovdXBzdHJlYW1fdGFyZ2V0

In the scenario of this runbook is the workload coming from the NFS_A1 NFS-Share, which means the mount path of that NFS-Share needs to be created and bound to the Target Cloud.

To keep the desired mount past a reboot it is recommended to edit the fstab of all compute nodes accordingly.

Reassign the workload

Trilio workloads have clear ownership. When a workload is moved to a different cloud it is necessary to change the ownership. The ownership can only be changed by OpenStack administrators.

Add admin-user to required domains and projects

Discover orphaned Workloads from NFS-Storage of Target Cloud

List available projects on Target Cloud in the Target Domain

The identified orphaned workloads need to be assigned to their new projects. The following provides the list of all available projects viewable by the used admin-user in the target_domain.

List available users on the Target Cloud in the Target Project that have the right backup trustee role

To allow project owners to work with the workloads as well will they get assigned to a user with the backup trustee role that is existing in the target project.

Reassign the workload to the target project

Now that all informations have been gathered the workload can be reassigned to the target project.

Verify the workload is available at the desired target_project

After the workload has been assigned to the new project it is recommended to verify the workload is managed by the Target Trilio and is assigned to the right project and user.

Restore the workload

The reassigned workload can be restored using Horizon following the procedure described .

This runbook will continue on the CLI only path.

Prepare the selective restore by getting the snapshot information

To be able to do the necessary selective restore a few pieces of information about the snapshot to be restored are required. The following process will provide all necessary information.

List all Snapshots of the workload to restore to identify the snapshot to restore

Get Snapshot Details with network details for the desired snapshot

Get Snapshot Details with disk details for the desired Snapshot

Prepare the selective restore by creating the restore.json file

The selective restore is using a restore.json file for the CLI command. This restore.json file needs to be adjusted according to the desired restore.

Run the selective restore

To do the actual restore use the following command:

Verify the restore

To verify the success of the restore from a Trilio perspective the restore status is checked.

Reconfigure the Target Trilio installation back to the original one

Delete NFS B2 to the Trilio Appliance Cluster

Edit the workloadmgr.conf

Look for the line defining the NFS mounts

Delete NFS B2 from the comma-seperated list

Write and close the workloadmgr.conf

Restart the wlm-workloads service

Delete NFS B2 to the Trilio Datamovers

To add the NFS B2 to the Trilio Datamovers manually the tvault-contego.conf file needs to be edited and the service restarted.

Edit the tvault-contego.conf

Look for the line defining the NFS mounts

Add NFS B2 to that as comma-seperated list. Space is not necessary, but can be set.

Write and close the tvault-contego.conf

Restart the tvault-contego service

Clean up

Remove the database entry

To allow the Trilio installation to be ready for another disaster recovery it is necessary to completely delete the entries of the Workloads, which have been restored.

Trilio does provide and maintain a script to safely delete workload entries and all connected entities from the Trilio database.

This script can be found here:

Remove the admin user from the project

After all restores for the target project have been achieved it is recommended to remove the used admin user from the project again.

# workloadmgr workload-show ac9cae9b-5e1b-4899-930c-6aa0600a2105 +-------------------+------------------------------------------------------------------------------------------------------+ | Property | Value | +-------------------+------------------------------------------------------------------------------------------------------+ | availability_zone | nova | | created_at | 2019-04-18T02:19:39.000000 | | description | Test Linux VMs | | error_msg | None | | id | ac9cae9b-5e1b-4899-930c-6aa0600a2105 | | instances | [{"id": "38b620f1-24ae-41d7-b0ab-85ffc2d7958b", "name": "Test-Linux-1"}, {"id": | | | "3fd869b2-16bd-4423-b389-18d19d37c8e0", "name": "Test-Linux-2"}] | | interval | None | | jobschedule | True | | name | Test Linux | | project_id | 2fc4e2180c2745629753305591aeb93b | | scheduler_trust | None | | status | available | | storage_usage | {"usage": 60555264, "full": {"usage": 44695552, "snap_count": 1}, "incremental": {"usage": 15859712, | | | "snap_count": 13}} | | updated_at | 2019-11-15T02:32:43.000000 | | user_id | 72e65c264a694272928f5d84b73fe9ce | | workload_type_id | f82ce76f-17fe-438b-aa37-7a023058e50d | +-------------------+------------------------------------------------------------------------------------------------------+

# workloadmgr snapshot-show --output networks 7e39e544-537d-4417-853d-11463e7396f9 +-------------------+--------------------------------------+ | Snapshot property | Value | +-------------------+--------------------------------------+ | description | None | | host | Upstream2 | | id | 7e39e544-537d-4417-853d-11463e7396f9 | | name | jobscheduler | | progress_percent | 100 | | restore_size | 44040192 Bytes or Approx (42.0MB) | | restores_info | | | size | 1310720 Bytes or Approx (1.2MB) | | snapshot_type | incremental | | status | available | | time_taken | 154 Seconds | | uploaded_size | 1310720 | | workload_id | ac9cae9b-5e1b-4899-930c-6aa0600a2105 | +-------------------+--------------------------------------+ +----------------+---------------------------------------------------------------------------------------------------------------------+ | Instances | Value | +----------------+---------------------------------------------------------------------------------------------------------------------+ | Status | available | | Security Group | [{u'name': u'Test', u'security_group_type': u'neutron'}, {u'name': u'default', u'security_group_type': u'neutron'}] | | Flavor | {u'ephemeral': u'0', u'vcpus': u'1', u'disk': u'1', u'ram': u'512'} | | Name | Test-Linux-1 | | ID | 38b620f1-24ae-41d7-b0ab-85ffc2d7958b | | | | | Status | available | | Security Group | [{u'name': u'Test', u'security_group_type': u'neutron'}, {u'name': u'default', u'security_group_type': u'neutron'}] | | Flavor | {u'ephemeral': u'0', u'vcpus': u'1', u'disk': u'1', u'ram': u'512'} | | Name | Test-Linux-2 | | ID | 3fd869b2-16bd-4423-b389-18d19d37c8e0 | | | | +----------------+---------------------------------------------------------------------------------------------------------------------+ +-------------+----------------------------------------------------------------------------------------------------------------------------------------------+ | Networks | Value | +-------------+----------------------------------------------------------------------------------------------------------------------------------------------+ | ip_address | 172.20.20.20 | | vm_id | 38b620f1-24ae-41d7-b0ab-85ffc2d7958b | | network | {u'subnet': {u'ip_version': 4, u'cidr': u'172.20.20.0/24', u'gateway_ip': u'172.20.20.1', u'id': u'3a756a89-d979-4cda-a7f3-dacad8594e44', u'name': u'Trilio Test'}, u'cidr': None, u'id': u'5f0e5d34-569d-42c9-97c2-df944f3924b1', u'name': u'Trilio_Test_Internal', u'network_type': u'neutron'} | | mac_address | fa:16:3e:74:58:bb | | | | | ip_address | 172.20.20.13 | | vm_id | 3fd869b2-16bd-4423-b389-18d19d37c8e0 | | network | {u'subnet': {u'ip_version': 4, u'cidr': u'172.20.20.0/24', u'gateway_ip': u'172.20.20.1', u'id': u'3a756a89-d979-4cda-a7f3-dacad8594e44', u'name': u'Trilio Test'}, u'cidr': None, u'id': u'5f0e5d34-569d-42c9-97c2-df944f3924b1', u'name': u'Trilio_Test_Internal', u'network_type': u'neutron'} | | mac_address | fa:16:3e:6b:46:ae | +-------------+----------------------------------------------------------------------------------------------------------------------------------------------+

[root@upstreamcontroller ~(keystone_admin)]# workloadmgr snapshot-show --output disks 7e39e544-537d-4417-853d-11463e7396f9 +-------------------+--------------------------------------+ | Snapshot property | Value | +-------------------+--------------------------------------+ | description | None | | host | Upstream2 | | id | 7e39e544-537d-4417-853d-11463e7396f9 | | name | jobscheduler | | progress_percent | 100 | | restore_size | 44040192 Bytes or Approx (42.0MB) | | restores_info | | | size | 1310720 Bytes or Approx (1.2MB) | | snapshot_type | incremental | | status | available | | time_taken | 154 Seconds | | uploaded_size | 1310720 | | workload_id | ac9cae9b-5e1b-4899-930c-6aa0600a2105 | +-------------------+--------------------------------------+ +----------------+---------------------------------------------------------------------------------------------------------------------+ | Instances | Value | +----------------+---------------------------------------------------------------------------------------------------------------------+ | Status | available | | Security Group | [{u'name': u'Test', u'security_group_type': u'neutron'}, {u'name': u'default', u'security_group_type': u'neutron'}] | | Flavor | {u'ephemeral': u'0', u'vcpus': u'1', u'disk': u'1', u'ram': u'512'} | | Name | Test-Linux-1 | | ID | 38b620f1-24ae-41d7-b0ab-85ffc2d7958b | | | | | Status | available | | Security Group | [{u'name': u'Test', u'security_group_type': u'neutron'}, {u'name': u'default', u'security_group_type': u'neutron'}] | | Flavor | {u'ephemeral': u'0', u'vcpus': u'1', u'disk': u'1', u'ram': u'512'} | | Name | Test-Linux-2 | | ID | 3fd869b2-16bd-4423-b389-18d19d37c8e0 | | | | +----------------+---------------------------------------------------------------------------------------------------------------------+ +-------------------+--------------------------------------------------+ | Vdisks | Value | +-------------------+--------------------------------------------------+ | volume_mountpoint | /dev/vda | | restore_size | 22020096 | | resource_id | ebc2fdd0-3c4d-4548-b92d-0e16734b5d9a | | volume_name | 0027b140-a427-46cb-9ccf-7895c7624493 | | volume_type | None | | label | None | | volume_size | 1 | | volume_id | 0027b140-a427-46cb-9ccf-7895c7624493 | | availability_zone | nova | | vm_id | 38b620f1-24ae-41d7-b0ab-85ffc2d7958b | | metadata | {u'readonly': u'False', u'attached_mode': u'rw'} | | | | | volume_mountpoint | /dev/vda | | restore_size | 22020096 | | resource_id | 8007ed89-6a86-447e-badb-e49f1e92f57a | | volume_name | 2a7f9e78-7778-4452-af5b-8e2fa43853bd | | volume_type | None | | label | None | | volume_size | 1 | | volume_id | 2a7f9e78-7778-4452-af5b-8e2fa43853bd | | availability_zone | nova | | vm_id | 3fd869b2-16bd-4423-b389-18d19d37c8e0 | | metadata | {u'readonly': u'False', u'attached_mode': u'rw'} | | | | +-------------------+--------------------------------------------------+

[root@upstreamcontroller ~(keystone_admin)]# workloadmgr restore-list --snapshot_id 5928554d-a882-4881-9a5c-90e834c071af +----------------------------+------------------+--------------------------------------+--------------------------------------+----------+-----------+ | Created At | Name | ID | Snapshot ID | Size | Status | +----------------------------+------------------+--------------------------------------+--------------------------------------+----------+-----------+ | 2019-09-24T12:44:38.000000 | OneClick Restore | 5b4216d0-4bed-460f-8501-1589e7b45e01 | 5928554d-a882-4881-9a5c-90e834c071af | 41126400 | available | +----------------------------+------------------+--------------------------------------+--------------------------------------+----------+-----------+ [root@upstreamcontroller ~(keystone_admin)]# workloadmgr restore-show 5b4216d0-4bed-460f-8501-1589e7b45e01 +------------------+------------------------------------------------------------------------------------------------------+ | Property | Value | +------------------+------------------------------------------------------------------------------------------------------+ | created_at | 2019-09-24T12:44:38.000000 | | description | - | | error_msg | None | | finished_at | 2019-09-24T12:46:07.000000 | | host | Upstream2 | | id | 5b4216d0-4bed-460f-8501-1589e7b45e01 | | instances | [{"status": "available", "id": "b8506f04-1b99-4ca8-839b-6f5d2c20d9aa", "name": "temp", "metadata": | | | {"instance_id": "c014a938-903d-43db-bfbb-ea4998ff1a0f", "production": "1", "config_drive": ""}}] | | name | OneClick Restore | | progress_msg | Restore from snapshot is complete | | progress_percent | 100 | | project_id | 8e16700ae3614da4ba80a4e57d60cdb9 | | restore_options | {"description": "-", "oneclickrestore": true, "restore_type": "oneclick", "openstack": {"instances": | | | [{"availability_zone": "US-West", "id": "c014a938-903d-43db-bfbb-ea4998ff1a0f", "name": "temp"}]}, | | | "type": "openstack", "name": "OneClick Restore"} | | restore_type | restore | | size | 41126400 | | snapshot_id | 5928554d-a882-4881-9a5c-90e834c071af | | status | available | | time_taken | 89 | | updated_at | 2019-09-24T12:44:38.000000 | | uploaded_size | 41126400 | | user_id | d5fbd79f4e834f51bfec08be6d3b2ff2 | | warning_msg | None | | workload_id | 02b1aca2-c51a-454b-8c0f-99966314165e | +------------------+------------------------------------------------------------------------------------------------------+

Example runbook for Disaster Recovery using NFS

hashtagScenario

hashtagPrerequisites for the Disaster Recovery process

hashtagDisaster Recovery of a single Workload

hashtagCopy the Workload directories to the configured NFS Volume

hashtagMount "NFS B2" Volume to a Trilio VM

hashtagIdentify the Workload on the "NFS B2" Volume

hashtagCopy the Workload

hashtagMake the Mount-Paths available

hashtagIdentify the base64 hash values

hashtagCreate and bind the paths

hashtagReassign the workload

hashtagAdd admin-user to required domains and projects

hashtagDiscover orphaned Workloads from NFS-Storage of Target Cloud

hashtagList available projects on Target Cloud in the Target Domain

hashtagList available users on the Target Cloud in the Target Project that have the right backup trustee role

hashtagReassign the workload to the target project

hashtagVerify the workload is available at the desired target_project

hashtagRestore the workload

hashtagPrepare the selective restore by getting the snapshot information

hashtagPrepare the selective restore by creating the restore.json file

hashtagRun the selective restore

hashtagVerify the restore

hashtagClean up

hashtagDelete the workload

hashtagRemove the database entry

hashtagRemove the admin user from the project

hashtagDisaster Recovery of a complete cloud

hashtagReconfigure the Target Trilio installation

hashtagAdd NFS B2 to the Trilio Appliance Cluster

hashtagAdd NFS B2 to the Trilio Datamovers

hashtagMake the Mount-Paths available

hashtagIdentify the base64 hash values

hashtagCreate and bind the paths

hashtagReassign the workload

hashtagAdd admin-user to required domains and projects

hashtagDiscover orphaned Workloads from NFS-Storage of Target Cloud

hashtagList available projects on Target Cloud in the Target Domain

hashtagList available users on the Target Cloud in the Target Project that have the right backup trustee role

hashtagReassign the workload to the target project

hashtagVerify the workload is available at the desired target_project

hashtagRestore the workload

hashtagPrepare the selective restore by getting the snapshot information

hashtagPrepare the selective restore by creating the restore.json file

hashtagRun the selective restore

hashtagVerify the restore

hashtagReconfigure the Target Trilio installation back to the original one

hashtagDelete NFS B2 to the Trilio Appliance Cluster

hashtagDelete NFS B2 to the Trilio Datamovers

hashtagClean up

hashtagRemove the database entry

hashtagRemove the admin user from the project

Example runbook for Disaster Recovery using NFS

hashtagScenario

hashtagPrerequisites for the Disaster Recovery process

hashtagDisaster Recovery of a single Workload

hashtagCopy the Workload directories to the configured NFS Volume

hashtagMount "NFS B2" Volume to a Trilio VM

hashtagIdentify the Workload on the "NFS B2" Volume

hashtagCopy the Workload

hashtagMake the Mount-Paths available

hashtagIdentify the base64 hash values

hashtagCreate and bind the paths

hashtagReassign the workload

hashtagAdd admin-user to required domains and projects

hashtagDiscover orphaned Workloads from NFS-Storage of Target Cloud

hashtagList available projects on Target Cloud in the Target Domain

hashtagList available users on the Target Cloud in the Target Project that have the right backup trustee role

hashtagReassign the workload to the target project

hashtagVerify the workload is available at the desired target_project

hashtagRestore the workload

hashtagPrepare the selective restore by getting the snapshot information

hashtagPrepare the selective restore by creating the restore.json file

hashtagRun the selective restore

hashtagVerify the restore

hashtagClean up

hashtagDelete the workload

hashtagRemove the database entry

hashtagRemove the admin user from the project

hashtagDisaster Recovery of a complete cloud

Scenario

Prerequisites for the Disaster Recovery process

Disaster Recovery of a single Workload

Copy the Workload directories to the configured NFS Volume

Mount "NFS B2" Volume to a Trilio VM

Identify the Workload on the "NFS B2" Volume

Copy the Workload

Make the Mount-Paths available

Identify the base64 hash values

Create and bind the paths

Reassign the workload

Add admin-user to required domains and projects

Discover orphaned Workloads from NFS-Storage of Target Cloud

List available projects on Target Cloud in the Target Domain

List available users on the Target Cloud in the Target Project that have the right backup trustee role

Reassign the workload to the target project

Verify the workload is available at the desired target_project

Restore the workload

Prepare the selective restore by getting the snapshot information

Prepare the selective restore by creating the restore.json file

Run the selective restore

Verify the restore

Clean up

Delete the workload

Remove the database entry

Remove the admin user from the project

Disaster Recovery of a complete cloud

Reconfigure the Target Trilio installation

Add NFS B2 to the Trilio Appliance Cluster

Add NFS B2 to the Trilio Datamovers

Make the Mount-Paths available

Identify the base64 hash values

Create and bind the paths

Reassign the workload

Add admin-user to required domains and projects

Discover orphaned Workloads from NFS-Storage of Target Cloud

List available projects on Target Cloud in the Target Domain

List available users on the Target Cloud in the Target Project that have the right backup trustee role

Reassign the workload to the target project

Verify the workload is available at the desired target_project

Restore the workload

Prepare the selective restore by getting the snapshot information

Prepare the selective restore by creating the restore.json file

Run the selective restore

Verify the restore

Reconfigure the Target Trilio installation back to the original one

Delete NFS B2 to the Trilio Appliance Cluster

Delete NFS B2 to the Trilio Datamovers

Clean up

Remove the database entry

Remove the admin user from the project

Scenario

Prerequisites for the Disaster Recovery process

Disaster Recovery of a single Workload

Copy the Workload directories to the configured NFS Volume

Mount "NFS B2" Volume to a Trilio VM

Identify the Workload on the "NFS B2" Volume

Copy the Workload

Make the Mount-Paths available

Identify the base64 hash values

Create and bind the paths

Reassign the workload

Add admin-user to required domains and projects

Discover orphaned Workloads from NFS-Storage of Target Cloud

List available projects on Target Cloud in the Target Domain

List available users on the Target Cloud in the Target Project that have the right backup trustee role

Reassign the workload to the target project

Verify the workload is available at the desired target_project

Restore the workload

Prepare the selective restore by getting the snapshot information

Prepare the selective restore by creating the restore.json file

Run the selective restore

Verify the restore

Clean up

Delete the workload

Remove the database entry

Remove the admin user from the project

Disaster Recovery of a complete cloud

Reconfigure the Target Trilio installation

Add NFS B2 to the Trilio Appliance Cluster