Testing Microsoft Office David Hansen TWC Test Manager

Testing – Microsoft Office David Hansen TWC Test Manager 14

§ 13 years (+2 internships) at Microsoft § Started in Outlook (email, attachments, setup, automation) § Now shared Office TWC team (security, performance, reliability, feedback) § Education § BS in Mechanical Engineering from BYU in 1992 § MBA from University of Washington in 2006 § Personal § Grew up in Reno, NV. § Married for 15 years to Wendy § 3 children (Hayley 13, Dillon 10, Mali 8) § Contact info: david. hansen@microsoft. com

§ Testing’s role in shipping commercial software § The kinds of challenges testers face § What makes a great tester Not (intentionally) § Recruiting for Microsoft

Software Development Engineer Program Manager Software Development Engineer in Test

PM Design the feature Dev (SDE) Implement the feature Test (SDET) Typical Office/Microsoft PM/Dev/Test team ratio is 0. 5 : 1 More precise definitions at http: //www. microsoft. com/college Validate the feature

Office Senior VP Shared Dev VP Shared PM VP Shared Test VP Ben Me Shared DM Shared GPM Shared TM Etc… Excel VP Dev Manager Word VP Etc… Test Manager Group Program Manager Test lead Feature Tester Test lead

§ Micro-version of product lifecycle § Designed for quick spin of information between triad § Goal is ‘dogfoodable’ feature quality before checking into ‘main’ *Feature Development Planning Implementation Verification Start Feature Crew Deliver Feature Code Perform Code review Deliver Feature Specification Deliver Test Automation Run check-in tests Test code on a private release build Deliver Dev Design Doc Deliver Test Design Spec Check code into main branch Deliver Detailed Feature Schedule Spec Inspection Check-in Test Feature Crew Signoff Build Verification Test

Office TWC Challenges

§ Data has significant impact on performance results § Scale and performance testing about pushing limits § How do you get all that data in there?

§ Automation? § Disadvantages Homogenous data sets. Lists all have similar fields § Stress and functionality test cramming data § Memory leak could bust you § Change in functionality/paths break test § § Generating lots of data takes a long time. Time is your most valuable resource as a tester.

§ Attach existing database? § Disadvantages Toughest problem in developing a server product: upgrade § Product changes constantly break solution § End up writing lots of build x to build y migration code instead of new features §

§ Tester owns the solution § Drive any product changes with Dev/PM § Author tools as necessary § Validate process § Generate performance results that are consistent, reliable, and accurate § Then you can start entering bugs…

Office TWC Challenges

§ Targeted attacks are more popular (going after assets instead of fame) § Office installed on a vast number of PCs (especially government/business) § Fuzzing (http: //en. wikipedia. org/wiki/Fuzzing) has risen as popular tool for finding buffer overruns

§ Smart vs. Dumb fuzzing § Competing against lots of motivated people with more time. How do you scale? Distribute the problem § Automate to deal with data explosion § § Move the battlefront Soldiers inside the walls § Sanitize (Wringer or MOICE) § Source analysis (catch it in the code) §

§ Aware and constantly learning about changing attack § § vectors Build infrastructure and tools to allow scaling out Validate data coming from the tools Provide ideas on moving the battlefront Validate the features Try to break out of the box § Verify box doesn’t break workflow, functionality, performance, etc… §

Office TWC Challenges

§ Feature only works with Office in a bad state How do you reproduce various states § Are you giving someone a broken fire extinguisher? § § Validity of the data Can developers fix the bug? § Minimal impact on user experience. They’re already mad. § Categorize crashes for decision making § No Personally identifiable information §

§ Write test plan § Automation § Validate data for developers and managers § Scale of the service § Tools § Simulate crash and doc recovery § Scale test the service § Service health monitoring § Process § Set goals for the organization § Drive investigation and action on the data § Help other testers across office use the data

§ Fixing bugs creates bugs § Beta feedback: Verbose – Lots of data § Squeaky wheel effect with customers. Not democratic. § § How can you tell if you’re making it better or worse?

Win. XP SP 1 example:

§ Share. Point survey and update feature § Offline store to local store migration § Null in filename § Office Min/Max functionality § Assembly code for exploit

§ § § § § Passion for software and technology Curious about how things work Raw smarts and quick learners CS background and coding skill Test mentality (malicious mindset) Communication & collaboration skills Deal with ambiguity & drive for results Customer focus Detail oriented and persistence Broad and deep technical knowledge

§ § § § § Passion for software and technology Curious about how things work Raw smarts and quick learners CS background and coding skill Test mentality (malicious mindset) Communication & collaboration skills Deal with ambiguity & drive for results Customer focus Detail oriented and persistence Broad and deep technical knowledge

A genuine passion for the work we do. From the day I started in Testing I loved it. I didn’t just like it, I loved it. I love finding bugs. I love learning about how things are implemented and using that knowledge to break the software. I love coding tools that help make it easier to find bugs faster and more efficiently (w. Eatmem, Win. Where). I knew only Pascal, Fortran and a little Basic when I came to MS. I immediately set out teaching myself to program in C and write Windows apps after that. I loved learning to code and I applied what I learned to writing simple games and tools (Pegged, Dr. Black. Jack, Win. Where, w. Eatmem, etc). I loved all of this. I had an innate genuine passion for testing and coding and applying it to making software better.