AI and Software Testing Vikram Raghuwanshi Senior Consultant

AI and Software Testing Vikram Raghuwanshi Senior Consultant Capgemini Technology Services

ABSTRACT This presentation gives an Introduction to Artificial Intelligence; and how AI touches our lives on day to day basis. It talks about the usage of AI in QA. It explains that how to deploy AI Systems for QA, and what benefits can be expected out of this. This will help Test Managers to decide if using AI techniques for software testing; will be helpful to their organization 2

What is AI When Machines and devices are capable of taking their own independent decision, we say that these machines and devices exhibit Artificial Intelligence. Figure: An AI Robot 3

Some applications of AI Virtual Assistants Driverless cars Driverless Trains Space Exploration Security Robotic Surgeons Advanced algorithms used in Stock markets o Cyborg technology o o o o Figure: Google Driverless car 4

How AI based systems work AI based systems employ heuristic algorithms to generate analytical models based on historical Test data. It also generates Charts and Mind Maps. Based on these models, AI system takes decision in day to day Test activities. For example, it analysis defects already logged, prepares a model, and then selects tests more likely to find defects in subsequent releases of the Application Under Test. 5

Heuristic algorithms used in AI The term heuristic is used for algorithms which find solutions among all possible ones, but they do not guarantee that the best will be found Some heuristic algorithms used in AI systems are: § Fuzzy systems § Neural networks § Evolutionary computation

Models Used by AI Systems Some models generally used by AI Systems are: Ø Naive Bayes Is it Sunny? Yes Wear white clothes No Is it raining Ø ARIMA Ø Decision Tree Yes Wear Rain Coat Figure: Decision Tree No Wear normal clothes 7

Prerequisites on using AI based system Following are the Prerequisites: Ø Data Cleansing & Standardization Ø Test plan Ø Test Cases Ø Defects Ø Requirements Traceability Matrix Ø Any other related artifacts Ø Relevant access to data source Ø AI Engine needs access rights to various systems. For example, for ALM it needs query editor access, for TFS it needs developer access 8

Integrating AI Based System Quite a Few vendors in open market offer AI solutions for Testing. Chose a System and do a POC. The system will have an API to communicate with your existing applications. Ask your development team / Vendor to do the integration. It will involve integration of various tools, as depicted in the next slide 9

Integrating AI Based System CI Tools Test Management Tools Build Tools AI Based Tool Reporting Tools Dashboard Database Systems Test Automation Tools 10

Integrating AI Based System AI tool will need a lot of historical data, so that it may generate Analytics Based on that Data AI Tool Test cases Defects Downti me Logs Code Quality Report RTM 11

Applications of AI in Software Testing Test Plan Support Test Suite Analysis RTM analysis Identification of Test Cases Continuous Integration An AI system can be built nowadays, to support software testing. It can have the features given alongside. It can have Chat Bots, which will chat with customers and users, collect feedback to help improve the software, and will also collect information about problems faced by the end users. The AI system will than log defects based on chat inputs Chat Bots Defect analysis Exhaustive Reporting Preserving Of test Artifacts Machine Learning 12

Analytics Generated with AI Tool AI tool gets various data feeds; it then gives critical predictive analysis driven at real time to the testing teams across project’s lifecycle assisting them to keep adjusting to various moving parameters influencing outcomes of the project. Analytics generated: Ø Assets analytics: Captures various metrics around analytics and provide recommendations on improving the leverage from individual assets. Analyses assets usage trend, execution duration, attached criticality through requirements, defects mapping etc. Ø Staff analytics: Assists understanding the distribution of capabilities, specific skills among the complete team (including client’s own staff and vendor resources), track real time progress being made on workforce transformation initiatives, predict resource needs for ongoing and upcoming programs upfront. 13

Analytics Generated with AI Tool Ø Customer analytics: Customer feedbacks from multiple social channels providing critical feedback for applications designs, indicating usage patterns for specific apps allowing testing teams to plan better around digital testing coverage. Ø Delivery analytics: Presents various analytics for testing delivery and trends to all possible levels and help testing teams to make decisive calls on testing plans in real time Ø Rightsizing solutions: Combination of various analytical utilities guiding testing organization to size their testing appropriately real time Ø Predictive analytics: Provides predictive views for all key dimensions associated with success of any project. 14

Benefits of Using AI in QA Ø Early completion of Software Testing Life Cycle Ø Better coverage – Test suggested by the system shall ensure optimal code coverage Ø Will help in getting Quality software with less time and effort Ø Will help in creation of Reusable prediction models Ø Unattended execution – In case of failures during Test Execution, AI system shall execute recovery routines and achieve unaided execution Ø Will help in integrating Source Control Systems, Build Servers, environment subsystems, Test management software and Cognitive Systems. Execution & Reporting Selecting Tests Environment provisioning 15

Example of an actual AI enabled System for QA Capgemini’s Smart QA is an analytics-driven platform that helps banks, financial institutions, and insurers build an end-to-end ecosystem, which includes testing assets, a test environment, test data and performance data It has An analytics engine, which includes cognitive capabilities powered by smart techniques and insights which test architects can use to prioritize what needs to be tested, optimize testing efforts, and identify areas of improvement. Smart QA also knows how much testing is required and which assets should be used. 16

Smart QA ‘s Sample Dashboard Testing summary view is given below. Defect status with respect to Project, Status, Severity, and Priority is tracked in various sections of this dashboard. 17

Smart QA ‘s Sample Dashboard Defect & Execution Metrics view is given below. It represents the metrics related to Testing efficiency. 18

Conclusion and Recommendation AI is next level of automation, lot of case studies already exist on the benefits of automation and how it saved time and efforts for enterprises, mostly in Regression suite execution. Case studies related to AI testing are not easily available in public domain yet. Vendors offering AI solutions, for example IBM, can definitely help in this regard. We should appreciate the benefits AI is giving us and we should try to contribute in whatever way we can. Given the benefits, we should move towards using AI into Software Testing Life Cycle.

References Ø Ø General search on AI on Google en. wikipedia. org https: //en. wikipedia. org/wiki/Evolutionary_computation https: //www. ibm. com/watson/ 20

Author Biography Vikram is having 12+ years of rich IT experience. He has worked on support, S/W Development, S/W testing and Automated testing. He has keen interest in current & emerging technologies. His educational qualifications include BSc. ( PCM ) & MCA 21

22