Cloud Penetration and Load Testing Best Practices Avantika

Cloud Penetration and Load Testing Best Practices Avantika Chadha Consultant, Deloitte Consulting India Pvt. Ltd. 1

ABSTRACT Cloud computing is the term everyone is talking about these days. One may be using it, even if we don’t realize it. If we use an online service to send email, edit documents, watch movies or TV, listen to music, play games or store pictures and other files, it is likely that cloud computing is making it all possible behind the scenes. Here a few of the things that can be done with the cloud: • Create new apps and services • Store, back up and recover data • Host websites and blogs • Stream audio and video • Deliver software on demand • Analyze data for patterns and make predictions The impact of the cloud on testing practices has grown with the cloud’s growing presence in the IT space. Testing practices are now dealing with several aspects of the cloud simultaneously. The strength of cloud computing's security barrier for user-protection and corporate compliance is crucial, especially if the company will store sensitive information in the system. This paper describes the best practices for cloud penetration and load testing. 2

INTRODUCTION: CLOUD COMPUTING Cloud computing is a form of Internet-based computing that provides shared computer processing resources and data to computers and other devices on demand. It is a model for enabling ubiquitous access to a shared pool of configurable computing resources (e. g. , computer networks, servers, storage, applications and services), which can be rapidly provisioned with minimal management effort. Whether you are running applications that share photos to millions of mobile users or you’re supporting the critical operations of your business, a cloud services platform provides rapid access to flexible and low cost IT resources. With cloud computing, you don’t need to make large upfront investments in hardware and spend a lot of time on the heavy lifting of managing that hardware. Instead, you can provision exactly the right type and size of computing resources you need to power your newest bright idea or operate your IT department. You can access as many resources as you need, almost instantly, and only pay for what you use. 3

CLOUD TESTING Cloud testing is the process of testing the performance, scalability and reliability of Web applications in a cloud computing environment. The focus areas for cloud testing basically are performance and security testing. For performance testing the critical factors which should be addressed • Addressing the loads that will be applied by other clients/customers of the cloud provider. • Addressing the loads that will be applied against the internet providers. To build a successful software both performance and security should be rendered to the cloud. 4

CLOUD PENETRATION TESTING Cloud Penetration Testing is a method of actively checking and examining the Cloud system by simulating the attack from the malicious code. Cloud computing is the shared responsibility with Cloud provider and client who earn the service from the provider. Regular Security monitoring should be implemented to monitoring the presents of threats, Risks, and Vulnerabilities. LA contract decides what kind pen testing should be allowed and how often it can be done. Important Cloud Computing Penetration Testing Checklist: 1. Check the Service level Agreement and make sure that proper policy has been covered between Cloud service provider (CSP) and Client. 2. To maintaining the Governance & Compliance, check the proper responsibility between Cloud service provider and subscriber. 3. Check the service level agreement Document and track the record of CSP determine role and responsibility to maintain the cloud resources. 5

4. Check the computer and Internet usage policy and make sure it has been implemented with proper policy. 5. Check for all unused ports and protocols and make sure unused services should be blocked. 6. Check the data which is stored in cloud servers is encrypted by Default. 7. Check for Two Factor Authentication being used and validate the OTP to ensure the network security. 8. Check the SSL certificates for cloud services in the URL and make sure certificates purchased from repudiated Certificate Authority (COMODO, Entrust, Geoturst, Symantec, Thawte etc. ) 9. Check the Component of access point, data center, devices, using appropriate security Control. 10. Check the proper input validation for Cloud applications to avoid the web application Attacks such as XSS, CSRF, SQLi etc. 6

CLOUD LOAD TESTING As cloud computing continues to mature, one is hard pressed to identify a class of enterprise software that is not delivered and consumed as a service. Performance and load-based application testing, important parts of ALM, can be counted among these cloud offerings. Moving these functions to the cloud offers typical cloud benefits, most notably lowered capital and operational costs, and support for distributed development teams. But cloud-based testing also changes the way the tests themselves are performed. One of the biggest challenges in application lifecycle management (ALM), according to Lanowitz, is performance. "Performance will make or break whether or not someone is going to use your app. If you think about the type of apps you use -- enterprise or personal apps -- performance is the determining factor, so make sure that performance is there and that you're able to test appropriately for performance. " Moving performance and load based application testing to the cloud, brings the cloud benefits to the Software testing by lowering capital and operational costs, and also support for distributed development & testing teams. It also allows you to simulate load tests constituting millions of concurrent users coming from multiple geographical locations. 7

MAIN REASONS TO CONSIDER CLOUD MODEL Performance testing demands extensive capital investments in hardware. Cloud is the good choice for organizations that do not want to have a full dedicated investment in testing infrastructure, as it fulfills all test environment needs and requirements. If Cloud computing is utilized effectively, it can help in achieving enough test coverage while minimizing the investment and configuration costs for any company. The usage of right environment for testing will also help in controlling the costs by enabling defect detection earlier in the life cycle. According to various research reports, 30 percent of the bugs in production are due to the incorrect test environment configurations. The study also indicates that the efforts involved in fixing these defect leakages are huge. With the Cloud-based rental and pay-as-you-go model, the testing team is provided with a test lab for the required timeframe. Organizations can shorten the provisioning time as cloud enables provisioning of test servers on an on-demand basis. Cloud model amplifies elasticity in the application platform. This implies that the actual resources used by the application may grow or shrink based on the application load. 8

Cloud environments could be shared with the development team for debugging purposes. With the Cloud, testing team can say -- We have tested this software in a real environment in the Cloud. Here is the defect and here is a link to the environment that was used for testing. Now developers can access that URL, to see the defects and fix them. (2) IDC reports that enterprise cloud application revenues reached $22. 9 B in 2011 and is projected to reach $87. 3 B by 2018. Cloud-based business application services will grow from $13. 4 billion in 2011 to $62. 2 billion in 2018. So it is very clear that the cloud model will supplement mainframe and client/server installations in the years to come. 9

BEST PRACTICES PENETRATION TESTING The following best practices can help maximize the advantages—and minimize the challenges— of penetration testing with the cloud. CLOUD CONSIDERATIONS Testing on behalf of a service provider or a tenant? Tenants lease infrastructure from a service provider based on one of three models: Infrastructure as a Service, Platform as a Service, or Software as a Service. In an Iaa. S deployment, the virtual machine and everything on it is under control of the tenant, giving leeway to attack everything up to the hypervisor. In Paa. S, the provider sets up everything needed for a given application, up to and including pre-installed databases, so there we have fewer options—really only the application and interface. Testing Paa. S deployments can negatively affect other tenants. Saa. S is a turnkey solution offering very few opportunities for testing, but still check out the application interface and API key management. If testing on behalf of a provider, every piece of the infrastructure is in play, so be thorough. If testing a service provider for Iaa. S or Paa. S, should inquire about their existing security process 10

INTERACTION WITH CLOUD SERVICE PROVIDERS Is permission required from the cloud provider? Pen-tests can violate terms of service, not to mention laws. Make sure one doesn’t end up with client’s service shutdown and data lost because the provider was unaware of the penetration test and treated you the same as any other attacker. Ask the cloud provider if they have a policy in place regarding security testing, and provide them with the expected time and date of the test and which virtual machines planned to attack. They may request that IP address is provided of the testing source, as well as the planned IPs and virtual machines planned to test. Gather information directly from the provider to save time. DIVING DEEP INTO THE CLOUD Testers should take care to only infiltrate their own instances, IPs, and ports. In Iaa. S or Paa. S the shared resources with other tenants and could impact their performance, which could violate Terms of Service. When notifying the service provider about pen testing, ask what might be off limits due to multiple tenants. Choose a vulnerability scanners and back them up with manual validation. Check common web exploits and other application weaknesses. The target will likely depend on the instance being audited: Microsoft Exchange server, database, backup server, storage, and/or application host. Attack the instance both from within the internal network and outside of it (employee vs. outsider hack). 11

LOAD TESTING The following best practices can help maximize the advantages—and minimize the challenges— of load testing with the cloud. TWO-STAGE PROCESS A two-stage process for load testing enables to employ cloud testing in the situations for which it is most effective and appropriate. In the first stage of the process, conduct internal tests with a medium load to quickly identify and resolve preliminary performance issues. Then, increase the load incrementally, proceed to the second stage, cloud-based load testing, for large scale tests that validate the entire delivery chain of the application. This approach enables teams to isolate problems. The source of any performance issue identified in the first stage is clearly within the firewall (because no other systems are involved in the test). It enables earlier testing. With the two-stage process, one don't have to wait for the application to be deployed and accessible from the Internet to test it. This lowers costs. Cloud testing is based on a pay-per-use model. When you can test internally on hardware you already have, you can reduce the amount of testing that you need to perform from the cloud and cut costs. 12

DATA SECURITY Encrypt the communication between your controller and load generators. This helps secure data sent to the load generators during the test (including account information) as well as the data that is retrieved (including error messages). Lastly, ensure that the load generators are secured with their own firewalls to protect them from outside threats. To ensure complete data security, we can add OTP check as a check. USING DIFFERENT CLOUD PROVIDERS There are several advantages of using multiple cloud providers. First, it helps to test from more geographical regions, which provides more realistic results that capture the effects of various third-party servers and content delivery networks. Second, it is more scalable. For exceptionally large scale tests, one can engage multiple providers simultaneously to bypass limitations that a single provider may place on bandwidth or the number of machines in use. Third, it enables to detect potential network issues at the cloud provider level. Load testing solutions that are locked into a single provider limit the test engineer’s ability to conduct realistic, reliable, large-scale tests. 13

TUNE LOAD GENERATORS To ensure that the load generator machines in the cloud are capable of generating large loads, one must properly tune the system to support the creation of a high number of sockets and threads per process. Additionally, allocate an appropriate heap size for Java based load generators. MONITOR THE SERVERS Once a performance bottleneck is identified, required is the information to track down its root cause. This information should be gathered during the test by monitoring each component of the infrastructure including application servers, database servers, and so on. Specifically, one want to monitor both the system—including the operating system, disks, and network—and the server software—including connection pools, threads, cache hits, and indexes. Linking all the information gathered during the tests with the tests themselves is much easier when the monitoring is integrated with the load testing tool. This enables to correlate the response times and errors generated by load testing with the monitored data to track down the cause of problems quickly. 14

CASE STUDIES There is also a need for best-in-class testing platforms and customizable testing frameworks, and a changed management mindset. • For instance, a leading US Financial Organization moved from legacy system to clouds but were skeptical about the security. In an unprecedented move at this organization, testers in their roles performed a rigorous pen. Test on the system and filtered potential security risks in business governance, and thus the QA function laid the foundation for a successful migration. And also saved the huge hardware cost the company had to bear due to legacy systems. (3). 15

• In another interesting case, at a gaming company, cloud testing practices helped improve streaming service that uncoupled gaming from the customary hardware. The cloud was the best solution to this problem. As a result of hosting services to the cloud it ensured an average of sub-35 milliseconds of latency. QA interventions enriched the cloud and with rigorous load testing ensured sub-35 milliseconds of latency with proper insights. Results: 16

• The Indian railway system, physically distributed and managed at a regional level, the information system of Indian Railway is a highly complex distributed mission critical system, integrated in networks. There are 120, 000 tickets generated every day in Indian Railway, each is 3 KB in size. The tracking period is normally 45 days, and the ticket needs to be stored online for one year. Therefore, 160 GB data is generated per day and the amount of data in ready-to-be-searched state is over 1 Terabytes. In addition, 4 -6 replicas of the data are required, must be considered. A cloud bed in this system can offer key advantages including Efficiency, Scalability, Reliability, and Flexibility. With complete set of Security Management: Security is critical to a cloud environment, especially for mission critical railway operations. Different requirements exist at several levels in the cloud and Load Testing the scalability and efficiency could increase multifold times. 17

THE GROWING CLOUD TECHNOLOGY (5) 18

LIMITATIONS OF PENETRATION AND LOAD TESTING IN THE CLOUD • Constructing Environment: For On Demand Testing For an on demand testing service, what are the steps to be taken to create a testing environment which is systematic or automatic? Even though the present technologies in cloud support instinctive delivery of needed computing resources (5) 19

for every Software as a Service (Saa. S) or for an application in the cloud, for setting up the needed test domain in a cloud there are no carrying solutions to assist engineers using a beneficial way. • The most important aspect of the limitations of cloud service penetration testing is where the responsibilities of the system start and end. This means it is important whether the target system is running within an Iaa. S (Infrastructure as a Service), Paa. S (Platform as a Service) or Saa. S (Software as a Service) configuration. Iaa. S will allow for much more intrusive and broad testing than Saa. S, because of the difference in the level of responsibilities and possibly the risk to multi-tenant shared systems. For example, in an Iaa. S configuration, the Operating System of a target server is managed by and dedicated to the customer. In a Saa. S configuration, this is not the case. A penetration test could affect the target and possibly cause an outage for other customers using that same shared system. In the worst case, there could even be an unintended information leak from another cloud customer. Determination of the responsibilities helps to set the scope of the test. (5) 20

• Integration Testing: Even though we saw various research articles describing software integration testing problems and schemes, not plentiful research outcomes are been put in in the real engineering system. The main causes is the present software and mechanisms are created without permitting technology and solution to help and assist organized software integration. In a cloud organization, engineers needed to be deal with combination of various Saa. S and applications inside and outside clouds in a black-box sight depending on the offered APIs and connectivity protocols. • Regression Testing: The regression testing problems and challenges caused by software modifications and bug-fixing must be addressed the by on-demand software validation in clouds. Yet, maximum current exploration in software regression testing grants most consideration to re-test a particular software version in a preconfigured test environment. The multi-tenancy characteristic of clouds might origin the trouble to put on the present explorated work in cloud testing, particularly for on-demand software regression testing service each and every time software is modified. In further, we also lack of dynamic software validation methods and (5) solutions to address the dynamic features of Saa. S and Clouds. 21

CONCLUSION Penetration and Load testing in the cloud isn’t much different from traditional environments, but it does require additional planning and communication. Most of the risks associated with the cloud come from non-functional requirements not being met or often not even being articulated and therefore not being tested or supported. To avoid such risks a proper plan for load and security testing should be in place. Try to gather as much information as possible ahead of time, and work with the smallest amount of contacts at the service provider as possible, unless organizing a security test to see how their detection team reacts to penetration testing. This helps ensure a more legitimate, normal environment. If one is unable to test any portion of the cloud, make sure to get configuration files, security audit policies, and previous penetration testing results from the provider. Read the Service Level Agreement (SLA) thoroughly and make sure it is covered in the case of a security breach. Security in the cloud is still a combination of careful planning and constant vigilance. 22

REFERENCES & APPENDIX Cited References: (1) http: //www. 360 logica. com/blog/how-cloud-testing-tools-prepare-to-cut-down-timeand-cost-to-develop-an-application/ (2) http: //www. agileload. com (3) http: //www. flbpo. co/the-cloud/ (4) https: //www. intel. in/ (5) http: //www. information-age. com (6) https: //blogs. cisco. com Other References: • https: //aws. amazon. com/ • https: //www. cloudindustryforum. org • https: //www. guru 99. com • Metasploit: The Penetration Tester’s Guide 1 st Edition by David Kennedy • The Art of Application Performance Testing by Ian Molyneaux 23

Author Biography Avantika Chadha Avantika has around 5 years of experience working with cross functional team on large scale business projects for the Public Sector, Telecom clients. Has over 5 years of experience in Functional testing and Automation testing of custom applications. Proficient in Requirement gathering, Test plans, Test design, Test execution and preparing Test summary reports. Has worked on various tools like SOAP UI, Parasoft SOATest. Jenkins, RFT, Selenium. 24

Thank You!!! 25