Open Stack Cloud DR Po C NEC India

Open. Stack Cloud DR Po. C NEC India © NEC Corporation 2017

Contents 1. Background I. III. 2. 3. I. III. IV. V. 4. • 2 Requirement for DR Solution in Cloud Different type of DR solutions Selection Criteria for DR solutions Environment Setup Results Summary Test Scenarios Functional test outcome Performance test outcome Issues identified Appendix Architecture of Trilio © NEC Corporation 2017

Background The following material highlights the Disaster Recovery activity done for Backup/Restore based Recovery Solution, using the Trilio® V 2. 4 DR Solution. This activity has been carried out to: q. Find requirement and current market trends for Cloud systems q. Examine possible solutions to these requirements, understand the solutions and how it can generate business, and create Po. C based on this understanding. q. Promote talent who can do proposals, introduce the business requirements and do operation of the system. 3 © NEC Corporation 2017

Requirement for DR Solution in Cloud q With migration of application to cloud systems, cloud systems have become very important. Minor failure in the systems or human errors can cause major business impact ( like [1], [2]) q Disaster recovery solution provide business continuity in case of complete failover of primary site. q In case of a disaster situation, primary site workload would be switched to Replica site and would be made as active site so that businesses are not affected. q This solution focus on switching to Replica site with defined RTO/RPO, so in case such a disaster occurs, the customer is notified of the maximum time in which his system will come back online. [1]: https: //www. theregister. co. uk/2017/10/03/faulty_fire_systems_take_down_azure_across_northern_europe/ [2]: https: //www. usatoday. com/story/tech/news/2017/02/28/amazons-cloud-service-goes-down-sites-scramble/98530914/ 4 © NEC Corporation 2017

Requirement for DR Solution in Cloud q. Market Trend for DR solution: Although the concept of DR in cloud is still nascent, a lot of SMBs are beginning to opt for DR to guard their business applications It is an attractive alternative for companies that are strapped for IT resources and who have a secondary infrastructure which is not effectively utilized. Having DR sites in cloud reduces Data Centre costs/space, IT resources, IT Infra leading to significant cost reduction. Each leading Cloud Service Provider ( AWS, Azure, Openstack ) now provide DR solutions via from their own end or via third-party software. 5 © NEC Corporation 2017

Different type of DR Solutions q Multiple options for DR mechanism to fit into Customer requirements: Option 1: Use Backup Images to recover at DR cloud Option 2: Use storage replication to replicate data continuously to recover the secondary site Option 3: Combination of the above two solutions leading to a hybrid solution. 6 © NEC Corporation 2017

Selection Criteria for DR solutions Feature Criteria System Recovery Post Restoration, we should have the same system as was existent before the disaster Low Recovery Point Objective The recovery point objective defines the last backed up copy. RPO depends on the method by which data is being copied from Master to Replica ( e. g. Volume Backup, Block Level Synchronization etc) Low Recovery Time Objective defines how fast can the system be recovered. That depends on how the data has been managed at the Replica site and how quickly can it deploy the necessary applications. Supported Storages Different variations of storages supported. VM Granularity allows prioritizing VM backup by allowing backing up specific VMs Recovery Points Number of recovery points supported Hypervisor Compatibility Type of Hypervisors it can support backup Support for Incremental updates Whether the backup images can be taken incrementally or not Maximum number of VM supported Maximum number of VM which can be backed up Multi-site support for DR Whether multiple sites can be backed up to a single replica cloud Scaling Whether the DR solution works with scaling the Cloud Up/Down Openstack version support Whether the DR solution is limited to specific Openstack Versions 7 © NEC Corporation 2017

Environment Setup 8 © NEC Corporation 2017

Environment Setup q. As mentioned earlier, the following material tracks the Proof Of Concept work done for DR Solution using Option 1 (Restoration using Backup Images). q. For creating the following Proof of Concept, we have used Trilio® DR Solution v 2. 4 q. For the Proof of Concept, we have tested Trilio on § Single-Site system: For functional testing § Multi-Site system: For Scalability and Performance testing 9 © NEC Corporation 2017

Trilio Architecture http: //www. trilio. io Trilio has 3 major components: • Trilio API : This component exists on the Controller and exposes the API for user control • Trilio Datamover : This component exists on all the Compute nodes. This component is responsible for taking the snapshots of the volumes and copying the data from Compute nodes to the Trilio. Vault • Trilio. Vault : This component exists as the VM and does most of the control and processing. • http: //www. trilio. 10 © NEC Corporation 2017 Operator Send Backup/Restore command Trilio API Initiate Backup/Restore Trilio. Vault (VM) Execute operations for VM and Volume Backup/Restore Trilio DM

Trilio DR Solution : High Level proposed design(Single Site) Configure Trilio via GUI/CLI Operator Execute Openstack CLI /control via dashboard 11 © NEC Corporation 2017

Trilio DR Solution: Single Site Trilio Services NTP Local Storage Trilio Node NFS Mount NFS FS Mounted NFS Node Rsync to 2 nd Site NTP Local Storage Trilio API Rabbit. MQ Neutron Linux Bridge Network Interfaces 12 Keystone Glance Controller Node mysql Horizon Nova-api, novacond, nova-sched © NEC Corporation 2017 Cinder LVM HDD Trilio DM Compute Node Cinder Volume Neutron Linux. Brg Agent Network Interfaces Novacompute Legend: Mgmt/Ext network: Data network: Control network: Trilio Network

Single site Trilio deployment model: Hardware Spec. Total Nodes: 8 ( Replica and Primary have same configuration ) 13 CPU Memory(GB) HDD(GB) OS Controller 4 6 10 Ubuntu 14. 04 Compute 4 4 20 --same-- NFS 4 2 10 --same-- Trilio 4 16 20 --same-- © NEC Corporation 2017

Trilio DR Solution: Multi-Node (Primary-Site) Layout for Multi-Node of Primary Site ( Same on secondary ) Legend: Ø As a VM: Ø Baremetal: Trilio DM Rabbit. MQ Keystone Trilio API Controller Node Neutron Linux Bridge Cinder Horizon Nova-api, novacond, nova-sched eth 1 Linux Bridge Neutron Rabbit. MQ Trilio API 14 eth 1 Keystone Controller Node Horizon Nova-api, novacond, nova-sched © NEC Corporation 2017 mysql NFS FS Node NTP eth 1 Cinder Volume Compute Node Neutron LB Agent Novacompute eth 1 LVM HDD Local Storage Glance mysq l Cinder LVM HDD eth 1 Trilio Services NTP Trilio Node Trilio DMCompute Node ( On separate Cinder machine) Compute Node Volume Nova. Neutron LB compute Agent Data , Trilio, Control and External N/W

Multiple site Trilio deployment model: Hardware Spec. Total Nodes: 14 ( 7 on each site ) CPU Memory(GB) Network Bandwidth HDD(GB) OS Controller-1 1 8 10 Gbps 20 Ubuntu 14. 04 Controller-2 1 8 20 --same-- NFS 4 5 10 --same-- Trilio 4 18 25 --same-- Compute-1 4 32 30 --same-- Compute-2 4 32 10 Gbps 100 --same-- Compute-3 4 5 10 Gbps 25 --same-- Note : Storage is on the Controller node 15 © NEC Corporation 2017

Results 16 © NEC Corporation 2017

DR Test Scenarios 17 © NEC Corporation 2017

Summary of test outcomes DISASTER RECOVERY PERFORMANCE SCALABILITY GOOD AVERAGE GOOD Test Execution Summary: Type of Testing # of tests executed # of incomplete tests due to Environment limitations # of tests passed # of tests failed due to Trilio issue # of tests conditionally passed Functional 27 0 23 2 2 System 2 4 0 0 6 Reason for environmental limitations: q System: Due to lack of space, we couldn’t recover the Trilio backup job for 12 VM/15 VMs during system testing. 18 © NEC Corporation 2017

System Test Outcomes 16 14 12 10 8 6 4 2 0 Backup Size(GB) 2 GB 4 GB 6 GB Volume Backup Time (Minutes) Backup time ( min) Backup size (GB) Backup Size(GB) 30 25 20 Backup Time (Minutes) 15 10 5 0 2 GB Volume 4 GB Volume 6 GB Volume The following tests were conducted keeping the number of VMs constant and varying the size of volumes. • Backup time seemed to tend toward a more constant value with increase in the size of volumes. • Backup size seemed to be increasing with increase in size of volume and deduplicated data. 19 © NEC Corporation 2017

System Test Outcomes 70. 00 • 60. 00 50. 00 • 40. 00 RPO (minutes) 30. 00 RTO (minutes) Size of Backup(GB) 20. 00 10. 00 4 5 8 10 12 Number of Volumes 20 © NEC Corporation 2017 15 • As expected, RPO , RTO and Backup size increases with increase in VM and Volume size. VM and volume size do not directly impact the data backup size if the duplicated data in the volume is high. RTO could not be calculated effectively due to failure to restore the systems at higher configurations( restoring larger number of VMs )

Openstack Issues identified 1. https: //bugs. launchpad. net/cinder/+bug/1724430 2. Trilio: After restoration fails, Neutron ports , Glance images and Cinder volumes may continue to exist in the system. On Restarting the restore, we observed that new ports were created, causing us to reach the quota limit at an accelerated pace. In case restore fails the created ports, images and volumes can be deleted, just like the VMs which were recovered, are deleted if restore fails. 3. In Mitaka , for Glance v 1 configuration files in Controller with Keystone defined, config files can use “default” as the default-project-id. However, when accessing the keystone authentication from a different node, the exact ID needs to be specified in the “default-project-id”. 4. Ubuntu 14. 04 has a problem where in static IP addresses can be overridden with DHCP provided IP addresses if the dhclient is running : https: //ubuntuforums. org/showthread. php? t=1391829. Solved by shutting down dhclient 5. Rabbit. MQ reported Channel. Error which caused a lot of agents going out-of-sync in Mitaka. 21 © NEC Corporation 2017

Trilio DR Solution : Issues identified and dependencies Following issues/dependencies were identified during initial investigation: 1. Dependency on Nova User/Group ID: As per the deployment guide shared by Trilio, Nova User/Group ID should be 162. While deploying Openstack, we observed that the User ID of Nova may not always be 162, but we need to change the User ID of Nova after it is deployed, which may seem like a restriction. 2. Dependency on Ubuntu 14. 04: Trilio requires Controller/Compute nodes to be on Ubuntu 14. 04, while Openstack in the current version favors 16. 04. It would also be beneficial to test Trilio with Cent. OS/Red. Hat. 3. NFS Permission : Seems like without ``no-rootsqaush`` option, and mode 755 on the NFS mounted folder, Trilio cannot write data on the mount point. 4. Observation: When Controller was working at 100% capacity ( w. r. t. CPU utilization ) , the Trilio Backup job was stuck, which was expected. The job didn’t recover after Controller came back to normal processing, though. We think this 5. Backup size of Incremental images: Incremental and Full backup images seem to be of the same size. This may put a restriction on the overall disk size. If Incremental Images are smaller than Full Images, then it helps us maintaining more backup generations. 22 © NEC Corporation 2017

Conclusion ▌ There are some issues with the User Interface and the error messages , which have been reported to Trilio , for improvement in future if not done in the current version of v 2. 4. ▌ Trilio uses Mitaka for v 2. 4. Due to Openstack Mitaka being EOL, we recommend Trilio to provide support with a newer version of Openstack. ▌ We verified that Trilio uses Glance v 1. As Glance v 1 has been deprecated, we recommend Trilio to provide support with Glance v 2. ▌ Trilio uses Ubuntu 14. 04 is an older version of Ubuntu which is not supported by recent releases of Openstack. We recommend Trilio to provide support with a newer version of Openstack. ▌ Overall, we recommend using Trilio , but with some of the issues and changes mentioned earlier. 23 © NEC Corporation 2017

24 © NEC Corporation 2017