Browsing the blog archives for the year 2009.

Automatic, for the People

work safe

Confession, this is the album where I stopped listening to REM. Dunno why, because Monster was one of the best albums I never bought.

Let us review our strategy, especially referencing our usage of White. I find it interesting that Test Complete makes it so hard to “do the right thing” as propounded by both Scripting GUI Tests in Ruby and by White. And just what is that?

The responsibility of the test script is to

  • perform a series of actions and verifications.
  • allow someone to read the script and understand what the hell is going on.

The first is a gimme. The second is interesting. When I have my developer hat on, I believe code is read more often then it is written. Wearing my manual tester hat, I believe the scripts should be clear and show intent. Tricky scripts are trouble. They are also a pain to maintain. So, if automated testing the meeting of these two worlds, the clarity and intent of the scripts must be sacrosanct. The libraries the scripts use, on the other hand, just need to be treated like code and kept maintainable.

What do I want my test scripts to look like? There are two options, DSL, or clearly abstracted test code.

First, lets look at DSLs, Baby! WAHOOO! Tests that double as manual tests!

open sekrit with New Document
insert image "picture.png" into sheet
first image item should match stored image "stored_0001.png"

I could do that. I could use some functional programming language. I could also use something like cucumber and code behind my feature files. But what’s the point? All I need is clarity, so the auditors / business people understand. I LIKE dsls, but this buys us nothing right now.

Option two uses Objects to clarify the application.

sek = Sekrit.open( :New_Document )
doc = sek.document

doc.add_image("picure.png")
Assert.equal doc.images[0].as_bitmap, Stored.images["0001.png"]

This is at the proper level of abstraction for a slightly motivated business user to understand. What they don’t care about at this level is the control hierarchy and control manipulation. Unless the requirements say, “via a popup menu” or some such.

If you’re going to be technical, this is a facade. It’s a wrapper to the complex underbelly of the white framework. It allows for a higher level use of the collected objects and re-use as we don’t have to repeat the code for complex actions and interactions. It also decouples our tests from changes in our application. I’ve had to chase that down in our current automation set, and I hate it.

How do we build up this abstraction? To be compatible with White’s recommended practices, I’m going to call this facade a “Screen”. We’re going to write a test for an application for which we have no source. Download blu and warm up your tweetin’ fingers. We’re going to log in!

# our screens
class Blu
  def initialize(application, window)
    @app = application
    @win = window
  end
  def login_screen
    return LoginScreen.new(@app, @win)
  end
end
class LoginScreen
  def initialize(application, window)
    @app = application
    @win = window
  end
  def login_user( username, password, remember = false)
    @win.get_textbox("UsernameTextBox").enter username
    @win.get_textbox("PasswordTextBox").enter password
    rem = @win.get_checkbox("Rememberage")
    if rem.checked != remember
      rem.enter " "
    end
    @win.get_button("LoginButton").click
  end
end

You noticed that the BluScreen has a LoginScreen? Our application has regions and concepts to which we wish to refer. For example, we may want to represent a ribbon control, a toolbar, different types of documents, or even the clipboard.

How do we use these screens? Let’s flesh out this little story in Dan North “Story” format.

 As I malicious user, so I can't defame Brian's good twitter name,
   Given blu is open
   And no one is logged in
   When I attempt to log in using a bad password
   Then I should see a warning that my login information was incorrect
app = Application.attach("blu")
win = app.get_window("blu")

blu = Blu.new(app, win)
login = blu.login_screen
login.login_user( "Myotherpants", "NoWai!" )

# A test would have assertion here...

I’m holding out on you. The truth of the matter is, I started with the test steps first, then made screens that abstracted those steps. But there’s magic in the screens, they have magic strings passed around like @win.get_checkbox("Rememberage"). Where did that come from? I got it from UISpy. I’ve read about Woodstock, Snoop, and a few other tools, but UISpy is the only tool I used for this post. It is in the Windows SDK. UISpy shows a tree of controls. When you click on one, it hilights it with a red box. It also displays information such as the automation Id, those magic strings we have scattered in our code. Before my two picture tutorial on UISpy, let me note, those strings are what’s likely to break your script in the future as your application grows.

UI Spy looking at a textbox

UI Spy looking at a textbox

Blu when UISpy is indicating a textfiled

Blu when UISpy is indicating a textfiled

what else is there?

These aren’t tests, they are automation. I’ll eventually wrap the automation with RSpec and get some real tests. I’ll get to Cucumber. I’ll publish examples. I promise.

But next, I want to address three types of custom controls:
User controls, Inhereited and Templated Controls, and Framework Elements. Another problem is finding un-named controls. I’ll see you next week.

One last bit, here’s what the white ruby library is starting to look like.

white_loc = (File.dirname(__FILE__) + "\\White_Bin\\")
$LOAD_PATH.push(white_loc)
require "White.Core.dll"

Application = Core::Application
Button = Core::UIItems::Button
CheckBox = Core::UIItems::CheckBox
TextBox = Core::UIItems::TextBox
Window = Core::UIItems::WindowItems::Window
SearchCriteria = Core::UIItems::Finders::SearchCriteria

class Window
  def get_button(*args)
    self.method(:get).of(Button).call(*args)
  end
  def get_button_labeled(label)
    self.get_button(SearchCriteria.by_text(label))
  end
  def get_checkbox(*args)
    self.method(:get).of(CheckBox).call(*args)
  end
  def get_textbox(*args)
    self.method(:get).of(TextBox).call(*args)
  end
end

Generic White Rubies are Open

work safe

I’ve been continuing my exploration of IronRuby and White for gui testing. I expected to deal with Automation IDs first, but I’ll get back to that. For now I’m tying IronRuby to White, and handling generics.

My install of white is not in the GAC, so I’ll be requireing DLLs from a directory. DLLs are looked for in ruby’s $LOAD_PATH, just like ruby libraries. It’s an array. You’ll notice that White suggests also including an NUnit dll, but I’ve not needed that with RSpec. And, I want to shorten the namespace stuff. So let’s start with white.rb and we’ll add to it later.

white_loc = "C:\\Projects\\Libs\\White_Bin_0.18"
$LOAD_PATH.push(white_loc)
require "White.Core.dll"
Application = Core::Application

So, next we just start using the application. I want to launch my application, conviniently called “sekrit.” I need to find the window titled “Sekrit (version 0.5)”. Then I need to call the C# method win.Get<Button>("Open") but that uses generics that ruby doesn’t have. Using what I learned from a Stack Overflow post how to invoke an overloaded generic method in ironruby, we can work around that in IronRuby 0.5.

require 'white'
app = Application.launch("c:\\projects\\sekrit\\build\\sekrit.exe")
win = app.get_window("Sekrit (version 0.5)")
win.focus
win.method(:get).of(Core::UIItems::Button).call("Open")

Oh, lawdy! Did you see that last line? It ain’t pretty. This is something we’ll be doing a lot of, so let’s wrap that up. I need to add, essentially, a non-generic extention method to wrap the generic call. But, do it with Ruby to extend a .NET class. Back to white.rb for some open class magic.

class Core::UIItems::WindowItems::Window
  def get_button(*args)
    self.method(:get).of(Core::UIItems::Button).call(*args)
  end
end

Now that cleans up our earlier program.

require 'white'
app = Application.launch("c:\\projects\\sekrit\\build\\sekrit.exe")
win = app.get_window("Sekrit (version 0.5)")
win.focus
win.get_button("Open")

Okay, one more problem. I talked about RSpec, but that’s because I hacked my install. There’s a known bug in IronRuby 0.5’s implementation of File.expand_path(file_name) that uses the .NET System::IO::Path methods that throws execptions when you try to expand a path with “:8” on the end to help track line numbers. I don’t know yet if it’s been fixed. If not, that may be another post.

So, what’s left? RSpec doesn’t run out of the box. I need to wrap some stuff up so I can say app.open_file("some_file.ext") and other higher level “user action” concepts (those shouldn’t be in white.rb, but sekrit.rb). I need to handle finding controls quickly and easily. I’ll take a look at that this weekend and see what I can find. Especially, since I’ll be hanging with a WPF MVP both this evening at IndyALT.NET and on Saturday.

Thoughts on GUI Testing

work safe

I’m currently involved with a WPF project as a tester. My charge is to automate the System-Level testing of this project for my client. Our belife is that automated testing allows us to find “popup bugs” before they become a problem. This has worked well, in combination with the team doing a degree of dog-fooding. They catch the esoteric things, I find the simple things no one looks for. Also, I’m currently working with a lot of them as we build manual test scripts for our verification phase. This is a lot more focused act that, once automated as we move forward with version two, will allow us to have a better understanding of the “shipability” of our software. We’re regulated and we want to automate most of the acceptance testing, at least in theory.

I’ve been using TestComplete for this. T.C. is good stuff, but there are some things about it I don’t like. It’s meant for recording and playback of tests. But, to automate in a regulated situation, I don’t want to see javascript (T.C.’s CSharp script is a horrible trap, don’t go there) like the following

var p1;
var w1;
p1 = Sys.Process('MyProcess');
w1 = p1.Window('HWind: MainWindow').MainWindow.Base.WPFObject('Grid', '', 1).ScrollView;
w1.x = 0;
w1.y = 0;
w1.WPFObject('RowItem', '', 2).StackPannel.WPFObject('Button', 'Cancel', 1).Click(3,15);

What the hell is my intent here? Also, I’m crawling the visual tree, which may be brittle. The renaming of internal variables, like “StackPannel” to something more revealing like “VisibleFlarns,” I’d have to go edit this test. And a buch of other tests that traverse that point in the tree. We kind of saw this coming and have rewritten our tests to a degree.

var process = Sys.Process('MyProcess');
VisibleFlarn(process, 2).WPFObject('Button', 'Cancel', 1).Click();

This is slightly more clear. The main benefit is that we’re hiding the traversal of the tree to the visible flarns. That means we only have one place to hunt down our changes to the tree. But it doesn’t take full advantage of our environment, JavaScript. Foreign objects, like the process, windows, and buttons can’t be extended with helper methods. So, I’ve started the next round of testing with a bunch of decorators and strict reuse.

var process = new MyProcess();
process.GetVisibleFlarn(2).Cancel();

That’s starting to get to good. The last piece is that regulatory oversight loves positive passing of tests, not just error logs. T.C. doesn’t have that. T.C. doesn’t even really have asserts. You can only error, or log. In fact, I’ve yet to find a test framework that does that out of the box. Even those that assert, don’t log positive asserts. Why is this important? I want to be able to say Req-123 was tested 7 times, and passed each one. That’s what the oversight wants. They fear, “Well, it’s didn’t show up in the error logs, so I’m assuming it worked.” So, I wrote some asserts that take explicit requirements and “Why” I’m asking about that.

Now, on to prong two, of my assault.

I recently picked up “Scripted GUI Testing with Ruby” by Ian Dees as I visited my local book store. I like the book. It covers using Win32API to grab window handles and throw around events. Oh, it’s also got some crap about Jruby and an apple-talk bridge if you want to go that route. Basically, it covers testing on all the platforms that aren’t X11 or .NET. That doesn’t detract from the book’s handling of GUI testing strategy, but I want to fill in that gap.

The CLR has a pretty good testing story, for WPF at least, in the White testing framework from ThoughtWorks. It uses Automation Ids to identitfy and find all of your controls. So, the pieces I’m looking at are, “What are Automation Ids,” and “What are their best practices?” Couple that with a running example of using IronRuby to get my hands on the application and bang it around. The big deal is our current project doesn’t use Automation Ids (accessibiltiy isn’t a huge driving need), so I need to show some benefit and low risk.

What am I going to have to do, for this experiement? I’m, currently, only interested in rSpec and don’t need cucumber. I’ll start by using White directly, but I’m guessing I’ll be wrapping it some. I’m also interested in alternatives to AutomationIds. But I’ll need to wrap the standard asserts with something that positively checks asserts against requirement numbers.

Will the regulators go for this? I don’t know. When working in a regulated environment, you can be innovative with how you handle testing as long as the auditors are comfortable with your ability to be repeatable and cover what they find important.

Should be fun.

Irony and Circles

work safe

When I started my DSL talk for los officiea, I was looking around for neat DSL things to add when I found Irony. I found it at a Lang.NET talk. Then I wrote a small DSL to define some geometric circles. So, here’s a quick rundown of how I made it happen.

Irony is used to define a parser. Inside the parser’s constructor, we build the BNF tree that describes the grammar. When that’s done, Irony will spit out an Abstract Syntax Tree (AST) that we can use for “real work”. Let’s take a look at some of the grammar, before we see the Parser code.

program <- shape +

shape <- circle | polygon | rectangle

circle <- circle point radius number

point <- [ number , number ]

What this looks like is the program ‘circle [100, 150] radius 35’. It gives me circle with a center and a radius. What’s that look like in C#? Or at least the circle bit?

public class ShapesParser : GrammarParser {

  public ShapesParser() {
     // Terminal statements
     var numbers = new NumberTerminal("number");
     // Non-Terminal
     var circle = new NonTerminal("circle", CircleBuilder);
     var point = new NonTerminal("point", PointBuilder);

     // Rules
     circle.Rule = "circle" + point + "radius" + number;
     point.Rule = "[" + number + "," + number + "]";
     this.Root = Circle
  }
}

You’ll notice the definitions for the non-Terminals contain delegates. These are methods in the ShapesGrammar that accept an NodeArgs and breaks out the interesting stuff and creates the custom node for the AST. This means we aren’t beholden to the default way ASTs fall out of the grammar, but inform it with some real smarts and make it fit our mindset.

But it seems that every time I play with Irony, I run into something. I started playing with FSharp, but the hyperstrong typing is far too picky. This wouldn’t be a problem if I was confident in FSharp and Irony, but I have to do a lot of casting involving an API I’m just learning. In addition, there’s a disjunction between functions in fsharp and delegates. I’ll have to figure that out.

Yesterday, I started working on Irony again, using VS10 because I can. Then it turns out nUnit is having problems loading .NET 4 dlls, because I didn’t build it. Enter yack shaving as I’m having issues building Nant.

None of these are Irony’s problem. It’s a great library, and as soon as I get a full demo working, I’ll be posting more.

Slides for my DSL Talk

Uncategorized

I just started a talk about DSLs. I’m posting the slides for consumption.

Slides

Slides w/ Notes

I just had to play

work safe

I’ve been working on a presentation and some other stuff all month, but I had to work on a T-Shirt idea for IndyALT.NET. It’s been nagging at me for a while. And it was fun! It has been far too long since I pushed pixels!

Red-Green-Refactor

Lightning Talks, a lesson learned

Uncategorized

Last week, IndyAlt.NET had a series of lightning talks that was a lot of fun. We where covering tools we’ve used. It was kind of a show and tell night.

Turns out that giving lightning talks are a specialized skill. I learned some things and I want to share.

There are two types of things you can talk about, small things and large things. This seems obvious, but you need to know which one you’re covering.

Small things: You may run under. This is okay. State the thesis quickly and up front. “Why do I care”. It can be cool, address a problem, or just be “pink”[0]. You can cover some neat use cases in that time.

Large things: It’s fine to not hit detail. Think of this as an overview of your thing. Give me a hook so I want to look into it more. I want to see why I want to look into this tool. Think of this as a 5 minute infomercial for the “Rolly”, not a lifestyle comercial for a cadillac. If it’s open source, I should want to commit to it. If it’s a product, I want to use it. If its a cool technique, I want to do it.

[0] My wife eleveates “Being Pink” to core functionality. When she got her pink Razor, that was its defining and most important feature. She was shocked to find out she likes her iPhone more, but she had to get a pink case for it. It was a moral imparative.

Parameterized Modules in Erlang

work safe

There is an erlang syntax that is beloved by OO converts to the language, the parametrized module. Here is why.

> Obj = param_example:new("Tony").
> Obj:name().
"Tony"

This Syntax makes an OO guy feel cozy. And taking a look at the implementation of our module, we see the the implied variable is defined on the module definition.

-module(param_example, [Name]).
-export([get_name/0, set_name/1]).

get_name() -> Name.

There is a caveat. If a module is parametrized, all functions are parametrized. You cannot use non-parametrized functions to wrap your parametrized function, like this.

> Obj = param_example:new("Tony").
> ListOfNames = param_example:new_from_list(["Tom", "Dick", "Harry"]).

In fact, there’s some magic, functions actually have an arity of one more than you think. Actually, an extra parameter for each module parameter. In addition, these are constant, like all erlang variables. Since you don’t have access to a recursive call stack, you can’t change them.

However you can use a separate module to front- end the parametrized calls. And “mutating” calls can return new instances with new parameters.

So when Is it a good thing and when do you not want to use it? I like the idea of using parametrized devices to wrap a bunch of data I’m going to be passing around but still has some functionality that needs to be called.

The scenario I didn’t find it useful for is having several processes that I can call without having a reference. More concretely, I need to have rooms addressable via names of some other soft reference. And their variables need to be modifiable. I abused named gen_server services for this.

-module(named_example).
-behaviour(gen_server).
-export([start_link/2, get_description/1, handle_call/3,init/1]).
-record(room, {name, description}).

server_name(Name) -> list_to_atom( lists:concat([ ?MODULE, "_", Name])).

start_link(Name, Description) ->
  gen_server:start_link({local, server_name(Name)}, ?MODULE, #room{name=Name, description=Description}, []).

get_description(Name) ->
  get_description:call(server_name(Name), description.

handle_call(description, _From, State) ->
  {reply, State#room.description, State}.

So, there are two ways of dealing with a collection of processes that are cleaner (don’t use the raw message send) than just passing around a Pid. They do different things, but might could be used in combination. I’m just not sure how, yet.

Oh, hey. If you have a better solution, please post a comment.