Best Practices For Testing Windows Drivers Dieter Achtelstetter

Best Practices For Testing Windows Drivers Dieter Achtelstetter Software Design Engineer Windows Device Experience Group Microsoft Corporation

About This Presentation Fulfill a request from OEM and IHV For best practices on testing Suggestion for Ways to reduce support costs for you and your customers Increasing your customer satisfaction Finding more bugs earlier in the process, where they cost a lot less to fix This advice is based on the experience of many device testers at Microsoft While this session attempts to cover most drivers/devices, some points may not be applicable to your driver/device

Agenda Overview Tools Test configurations Testing specific driver aspects Types of testing Resources Call to action

Overview Of Topics Static tools PREfast SDV Driver aspects Memory PO (Power Management) Penetration/Security I/O Cancel Testing types Stress tests Manual testing Run-time tools Driver Verifier Pn. P Setup Concurrency Long haul testing Scenario testing Test configurations System Settings Test strategies Test early Plan for testing Resources DTM kits Test Framework

Static Analysis Tools that run against the source code “PREfast for Drivers” “Static Driver Verifier” Find bugs fast Quick and inexpensive ways to find bugs Must pass these tests for driver signatures or logo Run early/run often Use these tests early in your development cycle Resolve issues found by the tools Run the tests for every new driver drop For details At Win. HEC: “Static Analysis and Verification of Drivers” Tools and documentation: WDK

Driver Verifies different aspects of drivers at runtime Built into Windows Type verifier from the Run menu Valuable in your regular testing Use standard settings Does Special pool, IRP verification, IRQL checking, etc. Same settings are used for driver signature or logo testing Only enable settings for drivers under test To ensure proper testing of your driver Get high test coverage By running with a full test pass You must exercise the code; otherwise, it cannot verify the driver For details Tools and documentation: WDK Many new features for Windows Vista

Test Configuration Vary your test environment Don’t just test your device in a static environment – change things around System configuration System environments Device settings Make these part of your regular test pass

System Configuration Different platforms: x 86, x 64, Itanium System with S 1 and S 3 support (for PO testing) Ensure system has WDDM video drivers So you get hybrid sleep (S 3 with Hiberfile) Multiprocessor/multi-core machines Only way to truly test concurrency Hyperthreading doesn't count for this testing Different chipsets Checked build Skews timing Has additional assertions See “Using the Checked Build” in the WDK

System Environments Low memory See “Boot Parameters to Manipulate Memory” in the WDK Large memory This is important for both for X 86 and 64 bit See “Will I need special hardware for memory top-down testing? ” in the WDK High CPU utilization From something other than testing the device Stress testing does this Skews timing Can uncover race conditions in your driver

Device Settings Test your device with all of its settings Not just the default settings Only you know what these settings do If your device uses hardware resources (IRQ, I/O Port. . . ), change them WDK tool “Pn. P Driver Test” does rebalancing Manually in Device Manager

Device Configuration Test several of the same device in a system Topologies Behind a PCI-PCI bridge Connected to a large tree of devices For USB or 1394 device, for example …

Testing Specific Driver Aspects of driver testing that may not usually be your focus But tend to have high failure counts Driver aspects Memory PO Penetration/Security I/O Cancel Pn. P Setup Concurrency

Memory Still a problem area with drivers Corruption – testing methods Driver Verifier with special pool setting Static analysis tools Code review Stress testing Leaks – testing methods Code review Manual testing Use “Pool. Mon” In the WDK to monitor pool allocations Generally, do repetitive operations, and look for continued pool usage increases

Plug And Play – Pn. P WDK tests “Disable/Enable with IO” “Pn. P Driver test” Long haul testing Set up system that tests Pn. P for several days Test the driver/device in docking stations Test undock/dock Manual testing Physically unplug and replug The device The bus/media: USB cable, net cables. . . Use your device Then disable/enable it

Power Management Windows Vista = more aggressive PO management Not just for laptops anymore WDK tests “Sleep Stress with IO” – cycles through sleep states and does device I/O before and afterward “Pwrtest” – Logs power events Long haul testing Set up system that tests PO for several days Manual testing Use the device, and then Put the system to sleep Hibernate the system

Pn. P And Power Management Test both together Don’t just test each in isolation This is a real-world scenario For example, often when you close the lid on a laptop, you also unplug devices WDK tests Common Scenario Stress with IO Manual testing Combine existing automated tests Or do these manually

Fault Injection Test driver code paths that are hard to hit under normal use Error handling Corner cases Hard to test You may need to add test code to the driver during testing Write a test filter driver Driver Verifier has a Fault Injection setting For USB, try Device Simulation Framework

Concurrency Do different things to the driver at the same time Analysis of our tests show they do not give very good coverage in this area System configuration Use multiprocessor machines Testing Use your device in multiple processes Use your device and Disable it Shut down the system Put the system to sleep Change the configuration Get data from it (WMI/IOCTL) …

Penetration/Security Be resilient against mischief Don’t assume that only your user-mode code will be calling your device Code review the IOCTL and WMI code path Who should have access to your devices: Test for it See “Creating Reliable Kernel-Mode Drivers” in WDK Errors in buffered I/O, etc. Test and code review for these pitfalls Device path exerciser in WDK test – limited but useful Run with Driver Verifier on target driver enabled For bugs found with this tool, do a code review Look for similar bugs in other code paths Where bug was found, look for other types of bugs Don’t expect this tool to find other bugs in this area for you Use static analysis and verification tools

I/O Cancellation Still a problem area with drivers Hard to test I/O must be currently active in the driver Testing Use your device, and then kill the process using it Doesn’t guarantee you tested I/O cancellation Set a break point or add tracing on cancellation code Make sure it gets hit during your testing May need to add test code to your driver Experiment

Setup Packages Test for all scenarios OEM preinstall During OS installation (for boot drivers) After OS installation OS Migration Install your driver on Windows XP, then upgrade to Windows Vista and ensure drivers and applets work Same for Windows Vista to Windows Vista upgrades For example, Home Basic to Home Premium SKUs Test with multiple packages Create 2 packages with 2 different versions of your driver Install version 1 Upgrade to version 2 Rollback to version 1 Un-install Tools: Chk. Inf

Testing Types Testing types Stress tests Scenario testing

Stress Testing Combines tests to simulate a heavy load Tests your driver under different system conditions Can find bugs that can be missed in feature testing Stress tests WDK has a stress test Allows you to add your device-specific test Build your own Testing – time and configuration Run for at least 24 hours at a time Run on different system configuration Run with different device settings

Scenario Testing Test your devices the way customers will use them Don’t just rely on synthetic tests/automation Focus on corner cases scenarios Mainstream scenarios tend to not be a problem Customers always find new ways to use your devices

Resources Driver Test Manager kits – DTM Test frameworks Windows Device Testing Framework – WDTF Device Simulation Framework – DSF Driver Frameworks Kernel-mode Driver Framework – KMDF User-mode Driver Framework – UMDF Windows Error Reporting

DTM Test Kits Run early When software, firmware, and hardware changes are still possible Use these tools on a daily basis in your testing Keep in mind that driving issues to a resolution takes time: Plan for 2 -4 weeks Many useful DTM tests for your regular testing Use DTM automation features in your lab Add your own tests At Win. HEC: WDK and DTM presentations “How to Use the WDK to Develop, Sign and Test Drivers” “Using the WDK for Windows Logo and Signature Testing” Sign up for the WDK Beta: www. microsoft. com/whdc/driver/wdk/betawdk. mspx See WHDC Web site: www. microsoft. comwhdc “Index of Windows Driver Kit Tools” in the WDK

WDTF – Windows Device Testing Framework for building driver/device tests Some DTM tests are built using WDTF Pn. P and Power Management tests Write your own tests For details At Win. HEC: “Using the Windows Device Testing Framework” Tools and documentation: WDK

DSF – Device Simulation Test Framework Simulate hardware through software Currently only for USB For details At Win. HEC: “Using the Device Simulation Framework for Software Simulation of USB Devices” session WDK DSF USB Loop Back Device Simulation

Windows Driver Frameworks (WDF) Generally, testing is same as for other drivers Some features that might help Features In Flight recorder Integrated Verifier Normal Verifier Skew execution Object live-time tracking Flags leaks on unload Fault injections Via registry key settings Debugger extension UMDF X App Verifier X KMDF X X Driver Verifier X X X

Windows Error Reporting Database of Windows crashes Improve experience/reduce costs Improve your customers’ experience Reduce your support costs – and your partners’ Engage through Winqual Sign up to see crash data for your drivers Provide solutions through Windows Update Check data regularly For just-released products, check daily For products on the market for a while, check monthly For details Winqual Web site: https: //winqual. microsoft. com

Call To Action Incorporate the testing points from this presentation into your regular test pass Test early Start in the design and prototyping phase Take advantage of the resources listed here Engage with Windows Error Reporting Engage with driver testers at Microsoft Drvtest @ microsoft. com

Additional Resources WDK: http: //www. microsoft. com/whdc/wdk Testing tools on WHDC: http: //www. microsoft. com/whdc/Dev. Tools/tools/ Driver aspects PO: http: //www. microsoft. com/whdc/system/sysperf/resumeperf. mspx Memory: http: //www. microsoft. com/whdc/driver/security /mem-alloc_tst. mspx Setup: “Using Driver Install Frameworks (DIFx)” – WDK Microsoft Symbol Server All OS symbols for all releases and patches: http: //www. microsoft. com/whdc/devtools/debugging/ Send email: Windows Driver testing alias Drvtest @ microsoft. com

Questions Tomorrow’s “Ask the expert” session

© 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U. S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.