Paul Gerrard paulgerrardconsulting com paulgerrard How to Test

Paul Gerrard paul@gerrardconsulting. com @paul_gerrard How to Test the Internet of Everyting

Background • The Io. E will dominate our industry for 10 to 20 years • Right now, it is very confusing • Standards emerging, but current applications are bleeding edge, speculative, exploratory (often simple) • Security and privacy risks dominate today • But functional and integration complexity challenges must also be overcome • There also broader societal risks • Government, companies and individuals need to pay attention. Intelligent Definition and Assurance Slide 2

Testing the Internet of Everything An article and webinar series 1. "Internet of Everything – What is it and how will it affect you? " 2. "Internet of Everything – Architecture and Risks" 3. "Internet of Everything – Test Strategy" All available at: http: //gerrardconsulting. com/? q=node/652 Intelligent Definition and Assurance Slide 3

What is the Io. E? Devices, makers, corporates, smart cities, drones, 3 -D printed cars

Some cheap devices Slide 5

My Robot (and a better one) https: //www. youtube. com/watch? v=f 0 Clu. Qiw. LRg Slide 6

Slide 7

Smart City World Congress Slide 8

Slide 9

Test this? Slide 10

Architecture(s)

A Layered Model of the Io. E • No standard for networking cheap devices • No standard/conventional architecture for Io. E • But. . . – some patterns are emerging and the architecture can be split into logical layers – Standards are emerging (slowly) – The 7 layer model describes the features of elements in an Io. T architecture. Intelligent Definition and Assurance Slide 12

Seven layer architecture Local, metro network internet connected Layers are often combined into one object/system Not Internet connected Local Network Mobile Devices Internet connected Data-Centre Based Not every system has seven separate layers. Slide 13

Drivers

Introduction • Io. E test strategy – for testing everything? – Can only paint a picture – Dimensions, influences, and opportunities – Suggest what testers need to think about • The challenge we face is somewhat different to what we have tested before. Intelligent Definition and Assurance Slide 15

What's in Scope? • The scope of Io. E is growing and includes: – Vast numbers of static devices, mostly sensors – Mobile devices: cars, buses, trains, planes, ships, satellites. . . • It will include: – Objects in homes, cities, work, hospitals, social venues – Wearable, human-embedded devices, phones and tablets • Io. E integrates with 100 s of services, payment and legacy systems • Eventually, Io. E will be called the Internet • The test strategy for the Io. E is the test strategy for everything • Once more with feeling. . . Intelligent Definition and Assurance Slide 16

The test strategy for the Io. E is the test strategy for everything

The Biggest Thing in 30 Years? • Continuous delivery and Dev. Ops bring factory automation processes into software • After Waterfall and Agile, CD and Dev. Ops become 'the third way' • The Io. E will affect everyone on the planet. • From 50 -700 bn devices over the next 5 to 20 years; no one knows • We are on a journey that could increase the scale of the internet by one hundred times • How will we test it? Intelligent Definition and Assurance Slide 18

The Testing Journey (from the 1980 s to the present) • Screen based applications running on dumb terminals – 1980 s • GUI applications and Client/Server – 1990 s • The emergence of the Internet - 1995 • 'Mobile Revolution' - today • Internet of Everything - emerging – Smart phones, buildings, cars, ships, planes and cities – Big. Data, pervasive, wearable computing – Massive scale • Io. E brings new levels of complexity and scale • Non-functional risks are well-understood – what's new? • Functional testing and simulation at very large scale. Intelligent Definition and Assurance Slide 19

If CMOs hold the budget… • There is a shift in power in the mobile space • If marketers hold the money for building Io. E: – IT must meet short and long term business goals – Software teams must deliver, experiment and learn • Market testing and experimentation to drive rapid evolution • Dev. Ops, Continuous Delivery, high levels of automation and simulation • Test analytics* blended with production analytics. * Thinking Big: Introducing Test Analytics, http: //gerrardconsulting. com/? q=node/630 Intelligent Definition and Assurance Slide 20

Connecting the Unconnected • 'Internet. org by Facebook' wants to provide cheaper access to data (perhaps 1/100 th of the current price) • Developing economies will probably lead the way in implementing the Io. E with mobile internet • But what networking technology/protocols will be used for local connectivity? "No technology dominates yet. We are waiting for technologies to improve in range and performance. The standard technology for connecting the Io. T hasn't been invented yet. " • We are still at the start of the Io. E journey Intelligent Definition and Assurance Slide 21

The Scope of Io. E Testing

Functional testing • Hardware-Level functionality – Most of this testing will be performed by manufacturers • Object and Server level functionality – We'll have to test simple web-apps to systems with many complex sub-systems • Integration and Interaction at all levels – Companies, testers will have to use judgement on deciding what to test and how much • Collaboration confusion – Use expectations and user testing of collaborating systems will be difficult. Intelligent Definition and Assurance Slide 23

Testing @ Scale • Scalability – We'll have to do some large-scale load, performance and stress testing • Wearables and embedded – We will need to simulate and test thousands of devices carried by or embedded in people • Everything connected – We'll need to do location-based tests at very large scale. Intelligent Definition and Assurance Slide 24

Networks • Moving networks – We'll have to test security and resilience of our systems against alien or rogue networks • Network security risks at multiple levels – Networks will need to be hardened and tested at all levels, not just 'perimeter' • Device registration, provisioning, failure and security – Power-down, power-up and automated authentication, configuration and registration processes will need to be tested • Mobile objects – Power, interference, network strength, roaming and jamming issues will all need testing. Intelligent Definition and Assurance Slide 25

Big Data • Logistics – A substantial data storage service will have to be tested (performance, backup/recovery etc. ) • Analysis and visualisation – These outputs and processes will need to be tested • Personal and corporate privacy – Testing will need to embrace the ethical use of data and it's protection from misuse. Intelligent Definition and Assurance Slide 26

The Scope of Io. E Testing • Not all Io. E systems will be huge or complex • But we face a wider range of concerns than ever before: – Most bring a new technology and risk profile – The range of testing issues we need to consider – The scale of the testing required has increased • We'll need new approaches to tackle the risks of failure. Intelligent Definition and Assurance Slide 27

Test Strategy for the Io. E

Modelling, Testing and Test Data • Tools to model the Io. E are only just emerging • We'll need to use Big Data test techniques • 'Pattern-based test design' – Generate tests from patterns to simulate scenarios – 'Heuristic oracles' will derive from conventions or common-sense – Personas as patterns of user or system behaviour • Developers will need to make systems testable through High Volume Automated Testing. Intelligent Definition and Assurance Slide 29

Test Environments; Testing in the Field • Test environments will vary widely – A test lab for a home could be set up in any office – An urban environmental management system will be harder to create • Scale of field testing will vary widely, but will become a common (and possibly mandatory) testing stage • Barcelona has set up an urban-scale testing facility – Barcelona Urban Lab is positioned as a field test environment to prove the viability of innovative systems • Cities around the world will offer similar services. Intelligent Definition and Assurance Slide 30

Slide 31

Tool Support • Automated support will be essential and probably dominate testing • Most components tested through APIs, web or other services • New protocols emerging; custom drivers may be required • The challenge: "How do we design the thousands or millions of tests to feed the tools? " Intelligent Definition and Assurance Slide 32

Test Analytics • Dev. Ops processes and production analytics monitor system vital signs (like sensors) – These analytics are essentially the same • Production systems are platforms for experimentation – The code that serves marketers can also serve developers and testers • Analytics code is your sensor network • Dev. Ops processes are 'things', too. Intelligent Definition and Assurance Slide 33

Performance Testing and Test Data • Local network protocols – IP based (XMPP, MQTT); few tools, but cloud services emerging – Low Power/Lossy Network (Zig. Bee, 6 Low. PAN, DASH 7/RFID, Bluetooth and NFC); only need simple drivers to 'chirp' • Coherent data may be a problem, e. g. location data – Cars must follow the road network – People can only walk through certain places in a shopping mall – So you will have to capture the GPS data using a tool perhaps • To create unique paths, you could edit together segments of intersecting paths. Intelligent Definition and Assurance Slide 34

The Future for Testing and Testers We need to think a little differently A New Model for Testing

Manual v Automated Testing? • Automation will not make testing easy; it will make testing possible • More and more, Dev. Ops processes are a source of data for analysis just like production • You might not need to be a programmer but you may need to: – Write your own utilities, download, configure and adapt open source tools or create automated tests – Have informed conversations with developers • Should testers learn to code? Probably Intelligent Definition and Assurance Slide 36

New Model Testing 29 page paper: http: //dev. sp. qa/download/new. Model Slide 37

Shift-Left • Every company seems to be 'shifting left' – Developers responsible for testing – no testers – Testers embedded in development teams – Testers morphing into business or systems analysts • Testing thinking is being moved left and earlier in the process • The New Model identifies these thinking activities • Testers need to learn how to 'let go' of testing, so view this as an opportunity, rather than a threat. Intelligent Definition and Assurance Slide 38

Close • The strategy for testing the Io. E is the strategy for testing everything • Some of the tools we need don't yet exist – Test modelling, test pattern/data generators and automated oracles – Simulation/high volume automation/visualisation tools • Testing in the lab and experimentation in the field become one discipline informed by Test Analytics • New test approaches will be required • The New Model aims to trigger some new thinking in this area. Intelligent Definition and Assurance Slide 39

Slide 40

Slide 41

Slide 42

Paul Gerrard paul@gerrardconsulting. com @paul_gerrard How to Test the Internet of Everything