High Volume Automated Testing for Software Components
Presented by: Harry Robinson and Doug Szabo, Microsoft
View the slide deck: HVTA 2013-11-13
Note from Harry: During the presentation, we showed sequences that expose bugs in sort routines. For those who would like to try their luck, here is the URL that hosts the Sorting Demo: https://www.brian-borowski.com/Software/Sorting. The algorithm we showed is called Shearsort. To get people started, the sequence “8 7 6 5 4 3 2 1 0” succeeds; the sequence “8 1 6 3 4 5 2 7 0” fails.
“Bugs are more insidious than ever we expect them to be.” – Boris Beizer
Would you expect to find bugs in an award-winning library of sorting routines written by professional coders and featured in the 2006 O’Reilly book, Windows Developer Power Tools?
Or, to phrase it differently, which of the following inputs will expose a bug in this well-regarded sorting library?
A. 0 1 2 3 4 5 6 7 8
B. 1 0 3 2 5 6 4 7 8
C. 8 1 6 3 4 5 2 7 0
D. 1 0 3 2 5 4 7 6 8
E. 8 7 6 5 4 3 2 1 0
The bug-exposing input turns out to be input C.
Would you have chosen that sequence for your unit testing? Probably not.
Let a relentless tester and an enlightened developer show you how simple, high-volume automation found insidious bugs that eluded a bevy of well-crafted unit tests. See the results, ask questions, get answers, and find out whether this technique should be part of your toolkit.
About our speakers:
Harry Robinson has been working on and thinking about software testing for a long time, pioneering advanced test generation approaches at Bell Labs, HP, Google and Microsoft over the past 20 years. He currently focuses on test techniques that combine human and machine intelligence. He is Principal SDET for Microsoft’s Windows Embedded team.
Doug Szabo has been developing and breaking software for 20 years across a range of applications from geodesy to 3-D hyperbolic graphs to automated mapping and facilities management systems. His 3-D work provided the visualization interface for Test Model Toolkit, Microsoft’s first model-based testing tool. Doug is a big fan of using programmatic test generation to get machines to do the heavy lifting in test.