I’m currently involved with a WPF project as a tester.  My charge is to automate the System-Level testing of this project for my client.  Our belife is that automated testing allows us to find “popup bugs” before they become a problem.  This has worked well, in combination with the team doing a degree of dog-fooding.  They catch the esoteric things, I find the simple things no one looks for.  Also, I’m currently working with a lot of them as we build manual test scripts for our verification phase.  This is a lot more focused act that, once automated as we move forward with version two, will allow us to have a better understanding of the “shipability” of our software.  We’re regulated and we want to automate most of the acceptance testing, at least in theory.

I’ve been using TestComplete for this.  T.C. is good stuff, but there are some things about it I don’t like.  It’s meant for recording and playback of tests.  But, to automate in a regulated situation, I don’t want to see javascript (T.C.’s CSharp script is a horrible trap, don’t go there) like the following

var p1;
var w1;
p1 = Sys.Process('MyProcess');
w1 = p1.Window('HWind: MainWindow').MainWindow.Base.WPFObject('Grid', '', 1).ScrollView;
w1.x = 0;
w1.y = 0;
w1.WPFObject('RowItem', '', 2).StackPannel.WPFObject('Button', 'Cancel', 1).Click(3,15);

What the hell is my intent here?  Also, I’m crawling the visual tree, which may be brittle.  The renaming of internal variables, like “StackPannel” to something more revealing like “VisibleFlarns,” I’d have to go edit this test.  And a buch of other tests that traverse that point in the tree.  We kind of saw this coming and have rewritten our tests to a degree.

var process = Sys.Process('MyProcess');
VisibleFlarn(process, 2).WPFObject('Button', 'Cancel', 1).Click();

This is slightly more clear.  The main benefit is that we’re hiding the traversal of the tree to the visible flarns.  That means we only have one place to hunt down our changes to the tree.  But it doesn’t take full advantage of our environment, JavaScript.  Foreign objects, like the process, windows, and buttons can’t be extended with helper methods.  So, I’ve started the next round of testing with a bunch of decorators and strict reuse.

var process = new MyProcess();
process.GetVisibleFlarn(2).Cancel();

That’s starting to get to good.  The last piece is that regulatory oversight loves positive passing of tests, not just error logs.  T.C. doesn’t have that.  T.C. doesn’t even really have asserts.  You can only error, or log.  In fact, I’ve yet to find a test framework that does that out of the box.   Even those that assert, don’t log positive asserts.  Why is this important?  I want to be able to say Req-123 was tested 7 times, and passed each one.  That’s what the oversight wants.  They fear, “Well, it’s didn’t show up in the error logs, so I’m assuming it worked.”  So, I wrote some asserts that take explicit requirements and “Why” I’m asking about that.

Now, on to prong two, of my assault.

I recently picked up “Scripted GUI Testing with Ruby” by Ian Dees as I visited my local book store. I like the book. It covers using Win32API to grab window handles and throw around events. Oh, it’s also got some crap about Jruby and an apple-talk bridge if you want to go that route. Basically, it covers testing on all the platforms that aren’t X11 or .NET. That doesn’t detract from the book’s handling of GUI testing strategy, but I want to fill in that gap.

The CLR has a pretty good testing story, for WPF at least, in the White testing framework from ThoughtWorks.  It uses Automation Ids to identitfy and find all of your controls.  So, the pieces I’m looking at are, “What are Automation Ids,” and “What are their best practices?”  Couple that with a running example of using IronRuby to get my hands on the application and bang it around. The big deal is our current project doesn’t use Automation Ids (accessibiltiy isn’t a huge driving need), so I need to show some benefit and low risk.

What am I going to have to do, for this experiement?  I’m, currently, only interested in rSpec and don’t need cucumber.  I’ll start by using White directly, but I’m guessing I’ll be wrapping it some. I’m also interested in alternatives to AutomationIds.  But I’ll need to wrap the standard asserts with something that positively checks asserts against requirement numbers.

Will the regulators go for this?  I don’t know.  When working in a regulated environment, you can be innovative with how you handle testing as long as the auditors are comfortable with your ability to be repeatable and cover what they find important.

Should be fun.