Dev For Ops Lesson 4 Automated testing Unit

Dev For Ops – Lesson 4 Automated testing: Unit, Integration & End-To-End Cornelis de Mooij & David Vossen Get the latest version of this presentation at www. stepupsessions. com

Before we start: It is assumed for this lesson that you have created the Notes. API, a simple Spring Boot API, as described in lessons 2 & 3. If you haven’t done so, many of the steps described here will still work for other projects with some alterations, but if you prefer to have a matching setup, make sure you follow lessons 2 & 3 first. For a copy of the lesson 2 & 3 presentations, contact Cornelis. de. Mooij@ing. com or David. Vossen@ing. com.

Before we start: The code that is implemented during this lesson, including the implementation of the exercises, can be found at https: //github. com/cornelisdemooij/Notes. API 2 Note that the page linked above will show the latest version of the code, which will be a spoiler in case you want to do the exercises yourself. If you want to see the code for a specific step, you can find a list of the previous versions at https: //github. com/cornelisdemooij/Notes. API 2/commits

Course Overview • Previously: • Lesson 1: Setup Dev Environment & Introduction to Merak • Lesson 2: API Structure: Model, Repository, Service & Endpoint • Lesson 2. 5: Introduction to Git: Init, Add, Commit & Push • Lesson 3: Connecting APIs & Databases with Hibernate & SQL • Today’s Lesson: • Lesson 4: Automated testing: Unit, Integration & End-To-End • Later Lessons: • Lesson 5: Version Control With Git: Advanced Skills

Overview • What is software testing and why should we do it? • Types of software tests • Adding tests to Notes. API • Unit Tests • Integration Tests • End-To-End Tests • Testing advice • • Which tests should I start with? How do I test an existing and complex codebase? Common testing mistakes and how to avoid them Different types of tests for different types of codebases • Further reading

Overview Any Questions?

What is software testing and why should we do it? • Software testing is any process that verifies the functionality or evaluates the performance of a software application. • Many potential benefits: • • Save time by checking correctness before building more Know earlier that multiple parts of software can work together Being sure that the entire system works before deploying to production Zero effort required to check old functionality still works Prevents downtime of your application for your customers Learn how your code performs under rare & extreme conditions Identifying mistakes in your logic Not getting punished for not being in compliance with ING’s rules

Software testing at ING • Software tests are usually required for software projects at ING. • Recommended “test coverage” is usually >70%. • Tests coverage is the percentage of your code that is run during testing. • Different versions: by lines, logical branches, functions or statements • High test coverage =/= good tests! Easy to create bad tests that wouldn’t detect problems while still running every line of code. • Tests for your individual application should be run in build pipeline: • Test fails? Pipeline fails. No deployment. • High-traffic applications require additional scalability testing: • Should be run on real servers, usually at night on Acceptance with Gatling. • Important applications and hardware can have stricter requirements.

Types of software tests • There are many types of software tests: • System Tests • Unit Tests • Integration Tests • Scalability Testing • E 2 E (End to End) Tests • Stress Tests • Regression Tests • Smoke Tests • User Acceptance Tests • Generated Tests • Alpha & Beta Tests • Mutation Testing � Caution: these terms are sometimes misused! �

Types of software tests • There are many types of software tests: • Unit Tests Today’s focus • System Tests • Integration Tests • Scalability Testing • E 2 E (End to End) Tests • Stress Tests • Regression Tests • Smoke Tests • User Acceptance Tests • Generated Tests • Alpha & Beta Tests • Mutation Testing

Types of software tests – Unit Tests • A unit test is a fast, simple test for a small section of code. • Should test one specific aspect of the program. • Should stay within 1 thread. • Should not use external components, such as databases or other services. • Duration should be measured in seconds, otherwise you would not run them often. @Test public void sort. Should. Work() { List unsorted = {1, 2, 0}; List answer = {0, 1, 2}; List sorted = unsorted. sort(); assert. Equals(sorted, answer, "Sorting failed!"); }

Types of software tests – Integration Tests • Tests interaction between different parts of the application. • Using real parts is often not a good idea: import some. kind. of. Mock; public class Mock. Test @Mock My. DB db. Mock; @Rule public Mock. Rule mock. Rule = Mock. JUnit. rule(); • If test fails, which part is malfunctioning? • Can’t test until all parts are implemented. • Test will be slow if its dependencies are slow. @Test public void test. Query() { Test. Class t = new Test. Class(db. Mock); • All 3 issues can be prevented with mocks: • • A mock is a fake implementation of another class. You can define responses for each call to a mock. Various mocking frameworks exist, e. g. Mockito. Often allows verification of number of calls { boolean check = t. query("* from t"); assert. True(check); verify(db. Mock). query("* from t"); } }

Types of software tests E 2 E (End-to-End) Tests • Tests the functionality of the entire application, locally. • Can use multiple threads, external parts like databases and other services. • Can take minutes, but faster is still better. • If real systems are too slow or unreliable to test regularly, external services can be mocked an embedded database can be used. • System Tests: similar, tests integration of entire application but no connections to external parts. These are mocked instead if needed.

Types of software tests – Regression Tests • After implementing new functionality and its tests, the tests should be kept around and run regularly, to prevent “regression”. • Regression = opposite of progression, so moving backwards • Every type of test should become a regression test, if still relevant. • Regression tests should be maintained: • Fix code or tests as code changes, instead of disabling inconvenient tests. • If regression tests become too slow, try to improve them, with more mocks, embedded databases, combining similar tests, etc. • Readme files & documentation should cover what types of test exist and how they can be run. Build pipelines should run tests automatically too.

Types of software tests User Acceptance Tests (UATs) • More commonly known as Alpha & Beta Tests. • Alpha = Internal, so employees and maybe family or friends • Beta = External, members of the public are either invited or can sign up. • So terms like “Public Alpha” or “Private Beta” are nonsense. • UATs at ING are usually Alpha Tests on Acceptance environment. • Should be as close to production set-up as possible. • Approval of Alpha Tests can be integrated in deployment pipeline. • At ING, product owners are responsible for alpha testing on Acceptance before giving approval in i. Validate, which should be checked by pipeline.

Types of software tests – Stress Tests • Special tests, designed to investigate what happens when the application is overloaded: • • Does the application crash, shut down gracefully, or slow down? Is data corrupted? Can this be repaired or rolled-back? Can the application recover or restart after overloading? Early warning signs, like longer response times, or is failure abrupt? • Various types of overload, alone or in combination: • Large amount of normal traffic (sudden popularity) • DDo. S: sudden large amount of malformed, incomplete or crafted requests • Congestion of limited resources: processing, database access, storage, etc.

Types of software tests – Smoke Tests • Similar to stress tests: special tests designed to investigate what happens when application (or hardware) runs for a long time. • Biggest concern usually memory leak, slowly filling up RAM. • Other potential problems include: • Running out of disk space • Running out of connections in a connection pool by not closing when done • Various overflow (or underflow) errors because of algorithmic mistakes: stack, counters, buffers, etc. • Cause might not be in your own code, but in a dependency, so code analysis can’t guarantee you are safe.

Types of software tests – Generated Tests • Large number of (unit) tests are created automatically to test same code for many different input scenarios. • Usually overkill for business applications, more relevant for libraries containing sorting algorithms, data structures etc. • Set of generated tests usually based on an “invariant”: a property of an algorithm, system or structure that is always true. • For example, after sorting a list: • All its items should be in order. • The sorted list should contain the same number of items as the original list. • Each item present in the original list should be present in the sorted list.

Types of software tests – Mutation Testing • Relatively new and not common, but powerful: mutation testing can reveal subtle mistakes in your code *and* in your tests. • Introduces small “mutations” in your code, then runs your tests. • Mutation can be changing + to -, > to >=, && to ||, etc. • If your code and your tests are good, these changes should make the tests fail. • If the tests do pass for some mutations, it’s a sign your code and/or your tests have subtle errors. • Mutation testing framework for Java: https: //pitest. org/

Types of software tests – Quiz What type of test is relevant in each scenario? • A 1000 customers are invited to try a new version of your software: • A User Acceptance Test (UAT), specifically a closed Beta test • You push a small change; the build pipeline runs all the old tests: • Regression tests, which can include unit tests, integration tests, etc. • The test automatically makes a 100 random arrays to test sorting: • Generated test, which checks invariants instead of hardcoding results. • Your system is left on and used for 4 weeks, without rebooting: • A smoke test, to check for resource leaks and subtle algorithm errors. • A test tries to get item 4 from a size 2 array and expects an error: • A unit test. Testing frameworks can often check for errors that should occur.

Adding tests to Notes. API

Unit Tests Testing the business logic inside Note. Service.

Unit Tests (1/6) • Add a resources folder under the test folder. • Create an application. properties file inside it with this content: spring. h 2. console. enabled=true spring. datasource. url=jdbc: h 2: mem: testdb spring. data. jpa. repositories. bootstrap-mode=default spring. jpa. database_platform=org. hibernate. dialect. H 2 Dialect spring. jpa. hibernate. ddl-auto=create-drop Our API needs a DB connection, but we don’t want to mess up our actual DB, so we set up an H 2 embedded database. This is also faster. • Open Notes. Api. Application. Tests (under test/java/com. *. Notes. API) • Click green triangle next to class Notes. Api. Application. Tests { • Your first unit test, the built-in context. Loads test, should now run successfully.

Unit Tests (2/6) Our application is quite simple, not much to unit test. Main business logic is in Note. Service, in save and update. By. Id. Need Note. Repository to test these, blurring unit & integration tests. Instead, we’ll modify these methods to separate business logic from persistence. Just as an example, probably better to do integration tests. • Replace save method with this (we’ll unit test save. Helper): • • public Note save(Note note) { note = save. Helper(note); return note. Repository. save(note); } public Note save. Helper(Note note) { note. creation = Timestamp. from(Instant. now()); note. modified = Timestamp. from(Instant. now()); return note; } No more business logic inside save method, instead it calls the helper… …to where the business logic has been moved.

Unit Tests (3/6) • Replace update. By. Id method with (we’ll unit test update. By. Id. Helper): public Optional<Note> update. By. Id(Long id, Note new. Note) throws Exception { Optional<Note> optional. Old. Note = find. By. Id(id); if (optional. Old. Note. is. Present()) { Note old. Note = optional. Old. Note. get(); Again, no more business logic in this new. Note = update. By. Id. Helper(old. Note, new. Note); method, instead it calls the helper… return Optional. of(note. Repository. save(new. Note)); } else { throw new Exception("Error: tried to update a note that does not exist. "); } } public Note update. By. Id. Helper(Note old. Note, Note new. Note) { …to where the business logic has new. Note. id = old. Note. id; new. Note. creation = old. Note. creation; been moved. new. Note. modified = Timestamp. from(Instant. now()); return new. Note; }

Unit Tests (4/6) • Add unit package in test/java/com. *. Notes. API. • Make Java class Note. Service. Unit. Test in unit with this code: package com. yourname. Notes. API. unit; import import static org. assertj. core. api. Assertions. assert. That; com. yourname. Notes. API. model. Note; com. yourname. Notes. API. services. Note. Service; org. junit. jupiter. api. Test; org. springframework. beans. factory. annotation. Autowired; org. springframework. boot. test. context. Spring. Boot. Test; java. sql. Timestamp; java. time. Instant; Make sure to update ”yourname” for the package and the imports. We’ll use Junit as our testing framework, Assert. J to help with checking test results, and Mockito (later) to set up mock objects. @Spring. Boot. Test public class Note. Service. Unit. Test { @Autowired private Note. Service note. Service; } @Test public void context. Loads() throws Exception { assert. That(note. Service). is. Not. Null(); } First test here is checking that the Note. Service gets autowired successfully.

Unit Tests (5/6) • Add method to Note. Service. Unit. Test: @Test public void save. Helper. Test() { // Given: Note note = new Note(); note. title = "Test title"; note. body = "Test body"; note. id = 100; Timestamp now = Timestamp. from(Instant. now()); // When: Note new. Note = note. Service. save. Helper(note); } Common way to divide test in sections: • Given: input and preparation • When: call to code to test • Then: check result from call Alternatively: Arrange, Act, Assert // Then: assert. That(new. Note. title). is. Equal. To(note. title); assert. That(new. Note. body). is. Equal. To(note. body); assert. That(new. Note. id). is. Equal. To(note. id); assert. That(new. Note. creation). is. Close. To(now, 1000); assert. That(new. Note. modified). is. Close. To(now, 1000); Code takes some time to run, so time won’t be exact match.

Unit Tests (6/6) • Add another method to Note. Service. Unit. Test: @Test public void update. By. IDHelper. Test() { // Given: Note old. Note = new Note(); old. Note. title = "Test title"; old. Note. body = "Test body"; Pretend this note is an hour old: old. Note. id = 101; old. Note. creation = Timestamp. from(Instant. now(). minus. Seconds(3600)); old. Note. modified = old. Note. creation; Note new. Note = new Note(); new. Note. title = "New title"; new. Note. body = "New body"; Contents must be different, to verify that the note is really updated. Timestamp now = Timestamp. from(Instant. now()); // When: Note new. Note = note. Service. update. By. Id. Helper(old. Note, new. Note); } // Then: assert. That(new. Note. title). is. Equal. To(new. Note. title); assert. That(new. Note. body). is. Equal. To(new. Note. body); assert. That(new. Note. id). is. Equal. To(old. Note. id); assert. That(new. Note. creation). is. Close. To(old. Note. creation, 1000); assert. That(new. Note. modified). is. Close. To(now, 1000); Again, no exact time match.

Integration Tests Testing the interaction of Note. Service with Note. Repository when saving or updating by id.

Integration Tests (1/4) • Add integration package in test/java/com. *. Notes. API. • Make Java class Note. Service. Integration. Test in integration with this code: package com. yourname. Notes. API. integration; import import import com. yourname. Notes. API. model. Note; com. yourname. Notes. API. repositories. Note. Repository; com. yourname. Notes. API. services. Note. Service; org. junit. jupiter. api. Test; org. mockito. Mockito; org. springframework. beans. factory. annotation. Autowired; org. springframework. boot. test. context. Spring. Boot. Test; org. springframework. boot. test. mockito. Mock. Bean; java. util. Optional; static org. assertj. core. api. Assertions. *; Again, make sure to update ”yourname” for the package and the imports. Note the new Mockito import. The Mock. Bean import is needed to make Spring autowire our mock. @Spring. Boot. Test public class Note. Service. Integration. Test { @Autowired private Note. Service note. Service; } @Mock. Bean private Note. Repository note. Repository; This Mock. Bean will pretend to be a Note. Repository. No real Note. Repository instance is made.

Integration Tests (2/4) • Add method to Note. Service. Integration. Test: @Test public void save. Test() { We need to tell our mock what to return when the save // Set up mock response: method is called, which should be mock. Note for any input. Note mock. Note = new Note(); mock. Note. title = "Mock note title"; Mockito. when(note. Repository. save(Mockito. any(Note. class))). then. Return(mock. Note); // Given: Note note. To. Save = new Note(); note. To. Save. title = "Doesn't matter"; // When: Note saved. Note = note. Service. save(note. To. Save); } Content of note. To. Save doesn’t matter because note. Service. save() returns the note that Note. Repository. save() returns. // Then: assert. That(saved. Note. title). is. Equal. To(mock. Note. title); Mockito. verify(note. Repository, Mockito. times(1)). save(Mockito. any(Note. class)); Mockito can verify that a mock’s methods were called an expected number of times.

Integration Tests (3/4) • Add another method to Note. Service. Integration. Test: @Test public void update. By. Id. That. Exists. Test() { // Set up mock response: Note mock. Note 1 = new Note(); Accept any id: mock. Note 1. title = "Mock note 1 title"; Mockito. when(note. Repository. find. By. Id(Mockito. any. Long())). then. Return(Optional. of(mock. Note 1)); Note mock. Note 2 = new Note(); Note the different title, so we can tell them apart later. mock. Note 2. title = "Mock note 2 title"; Mockito. when(note. Repository. save(Mockito. any(Note. class))). then. Return(mock. Note 2); // Given: Note note. To. Update = new Note(); note. To. Update. title = "Doesn't matter"; // When: Optional<Note> optional. Saved. Note = null; try { optional. Saved. Note = note. Service. update. By. Id((long) 999, note. To. Update); } catch(Exception e) { fail("No exception should be thrown by note. Service. update. By. Id in update. By. Id. That. Exists. Test"); } We explicitly fail this test here, because this code shouldn’t be reached if update. By. Id works. } // Then: assert. That(optional. Saved. Note). is. Not. Null(); assert. That(optional. Saved. Note. is. Present()). is. True(); Note saved. Note = optional. Saved. Note. get(); assert. That(saved. Note. title). is. Equal. To(mock. Note 2. title); Mockito. verify(note. Repository, Mockito. times(1)). find. By. Id(Mockito. any. Long()); Mockito. verify(note. Repository, Mockito. times(1)). save(Mockito. any(Note. class)); In the happy flow, both mocked methods should be called once.

Integration Tests (4/4) • Add last method to Note. Service. Integration. Test: @Test public void update. By. Id. That. Does. Not. Exist. Test() { // Set up mock response: Mockito. when(note. Repository. find. By. Id(Mockito. any. Long())). then. Return(Optional. empty()); Note that we don’t stub the save method, as we don’t need it. // Given: Note note. To. Update = new Note(); note. To. Update. title = "Doesn't matter"; // When & Then: Limitations of Given, When & Then: sometimes it blurs together. assert. That. Thrown. By(() -> { note. Service. update. By. Id((long) 999, note. To. Update); // When. }). is. Instance. Of(Exception. class). has. Message("Error: tried to update a note that does not exist. "); } Mockito. verify(note. Repository, Mockito. times(1)). find. By. Id(Mockito. any. Long()); Mockito. verify(note. Repository, Mockito. times(0)). save(Mockito. any(Note. class)); We can still verify the save method, even without stubbing it.

A Note On Mocks & Other Test Doubles • There are 5 types of Test Doubles (see Gerard Meszaros’ article) • Often all called Mock, but each has own name and characteristics: • Dummy Object: • Dummy version of a required input that doesn’t matter for a particular test. • Useful when constructing real input takes more code and/or is slow. • Test Stub: • Holds predefined data, uses it to answer calls during tests. • Used when we don’t want to use real objects, e. g. if they have unwanted side effects. • Mock Object: • Like Test Stub, but registers received calls. Allows us to verify expected calls happened. • Test Spy: • Allows verification like Mock Object but has real implementation underneath. • Fake Object: • Have working implementations, not same as production. Usually takes shortcut.

End-to-End Tests Testing the entire application, from Note. Endpoint to the (embedded) database.

End-To-End Tests (1/4) • Add E 2 E package in test/java/com. *. Notes. API. • Make Java class Note. Endpoint. E 2 ETest in E 2 E with this code: package com. yourname. Notes. API. E 2 E; import com. yourname. Notes. API. endpoints. Note. Endpoint; import org. springframework. beans. factory. annotation. Autowired; import org. springframework. boot. test. context. Spring. Boot. Test; Again, update ”yourname” for the package and imports. @Spring. Boot. Test public class Note. Endpoint. E 2 ETest { @Autowired private Note. Endpoint note. Endpoint; } • Notice that there are no @Mock. Bean objects: • In this test, all the real implementations will be used. • The only Test Double is our H 2 embedded database, a Fake Object.

End-To-End Tests (2/4) • In Note. Endpoint. java, change return Response. accepted(true). build(); to return Response. accepted(result). build(); . • Add this method to Note. Endpoint. E 2 ETest: @Test public void post. Note. Success. Test() { // Given: Note note. To. Post = new Note(); note. To. Post. title = "Test title"; note. To. Post. body = "Test body"; Timestamp now = Timestamp. from(Instant. now()); // When: Response response. Posted. Note = note. Endpoint. post. Note(note. To. Post); Note posted. Note = (Note)response. Posted. Note. get. Entity(); } // Then: assert. That(posted. Note. title). is. Equal. To(note. To. Post. title); assert. That(posted. Note. body). is. Equal. To(note. To. Post. body); assert. That(posted. Note. id). is. Not. Null(); assert. That(posted. Note. creation). is. Close. To(now, 1000); assert. That(posted. Note. modified). is. Close. To(now, 1000); We need to unwrap the Response object return by our endpoint.

End-To-End Tests (3/4) • Add another method to Note. Endpoint. E 2 ETest: @Test public void post. Note. Failure. Test() { // Given: Note note. To. Post = new Note(); note. To. Post. title = "Test title that is too big: . . . . . . . " + ". . . . . . . . . "; note. To. Post. body = "Test body"; The title (and the body too, kind of a design flaw) has a character limit of 255 characters, that we’ll try to exceed to trigger an error: // When: Response response. Posted. Note = note. Endpoint. post. Note(note. To. Post); } // Then: assert. That(response. Posted. Note. get. Status()). is. Equal. To(500); • This test fails, revealing an error in our implementation. • Well, just goes to show, how important testing can be. • We’ll fix our Note. Endpoint post. Note method on the next slide.

End-To-End Tests (4/4) • In Note. Endpoint, change the implementation of post. Note: try { Note result = note. Service. save(note); if (result != null) { return Response. accepted(result). build(); } else { return Response. server. Error(). build(); } } catch(Exception e) { return Response. server. Error(). build(); } • Good practice to not reveal the error contents in API response. • Can be helpful temporarily while debugging though. • Exercise: Add end-to-end tests for the other methods, fixing them if needed.

Testing advice

How do I test an existing and complex codebase? • Start with writing/improving end-to-end or system tests. • Then, integration tests, starting with output (endpoints) • Mock the rest of the system as much as possible. • Move further step by step, to services, then repositories • Unit tests only needed for complex logic; benefit from learning here • See also further reading: “Working Effectively With Legacy Code” • Migrating a complex codebase? • Read: “Monolith to Microservices” by Sam Newman

Common testing mistakes & avoiding them • Don’t test your mocks (surprisingly common): • No point in testing that the mock gave the answer that you told it to give. • Testing for coverage instead of correctness: • High test coverage =/= good tests! • Test for common mistakes: Off-by-one errors, Boolean logic mistakes, forgetting parentheses, edge cases, corner cases etc. • Test handling of wrong input: results in expected error? Or crash? • Mutation testing can help but takes effort. • Poor man’s mutation testing: after writing test, try to make it fail yourself. • Not documenting your tests: • Add comments with your reasoning: why does this exist? How does it work? • Add readme file with instructions for running your tests. • Not maintaining your tests: clean up, speed up & review • Commenting tests when they start failing; fix them or it will hurt in the long run.

Different types of tests for different types of codebases Unit Tests: Integration Tests: E 2 E (End to End) Tests: Regression Tests: User Acceptance Tests: System Tests: Scalability Testing: Stress Tests: functionality • Smoke Tests: • Generated Tests: • Mutation Testing: • • Algorithms, data structures, complex business logic APIs, complex frontends/apps, embedded software Any software, especially complex software Any software User facing software: frontend, apps, games Like E 2 E, but when deployment is expensive/slow (Potentially) high-traffic software like common APIs Public facing software especially if key Hardware, games, embedded software, servers Algorithms, data structures Complex business logic

Further Reading

Further Reading Chapter 4 covers testing and its benefits for refactoring: making sure that the application keeps working as it should while re-organizing and optimizing it. Refactoring – Improving the Design of Existing Code by Martin Fowler.

Further Reading Chapter 9 covers Test Driven Development (TDD) and how to keep your tests “clean”: Tests should be fast, independent, repeatable, self-validating and timely. Clean Code by Robert C. Martin, aka “Uncle Bob”.

Further Reading Covers tools and techniques for changing and improving big, old, complex code, which starts with adding tests. Working Effectively With Legacy Code by Michael C. Feathers. Disclaimer: haven’t personally read this yet but have heard it be recommended multiple times for working with legacy code.

Automated testing: Unit, Integration & End-To-End Any Questions?

Next Lesson • Dev Lessons For Ops • Lesson 5: Version Control With Git: Advanced Skills • For questions & comments, please contacts us at Cornelis. de. Mooij@ing. com & David. Vossen@ing. com • Get the latest version of this presentation at www. stepupsessions. com