Saving the world, one test case at a time


Links
Blog Categories

Sign In



Blogs I read

Predicting bugs in code using source control information

Posted on Sunday, 09 July 2006
Google have been publishing some of their internal tech talks for a while now, and I recently came across this little gem; "Predicting bugs in code changes using SCM information ". The concept proposed by this video is that you can use the combination of your source control check-in logs and defect tracking history to identify where the areas of lowest quality code are and focus your testing efforts on them.

In Code Complete, Steve McConnell espouses some statistics that: "80% of errors are found in 20% of a project’s code.", and "50% of errors are found in 10% of a project’s code.".

This leads me to a logical conclusion that with a little bit of thought and planning with VSTS, you could identify the 20% of code that has the most defects logged against it and focus your testing efforts there. However like everything, once you start measuring you begin squeezing the balloon, and effecting the system that you are measuring.

Disclaimer  |  Comments [0]  |  Trackback

The great comment debate: Adobemedia v Microsoft Expression

Posted on Sunday, 09 July 2006
Frank Arrigo blogged last week about a debate that has broken out within the comment of one of his recent posts.

I didn't think that it was appropriate to jump in mid sentence in someone else's "conversation", however I did want to highlight and discuss one important point that was mentioned by whoever goes by the name of "I'd rather not tell..."

The main oversight on Microsoft's part in my opinion is that they haven't released a Mac version of their products.

This is a very important point. Designers use Apple's products because Apple embraces and represents what designers are passionate about ... great design. If Microsoft wants these products to succeed in the design market, they really need to produce Mac OS X native versions. This will not be a simple task however as the Expression suite are all written in managed code, on top of WinFX. As much as I would like it I doubt that Microsoft are about to port the whole of .net Framework 3.0 anytime soon.

Disclaimer  |  Comments [0]  |  Trackback

How did I miss this? Windows Media components for Quicktime

Posted on Wednesday, 07 June 2006
Way back in January, Microsoft released, (via Flip4Mac), components to allow playback of windows media files in the native OS X quicktime player. The components are free for playback, but you need to pay to encode.


Disclaimer  |  Comments [0]  |  Trackback

The simple things in life ...

Posted on Monday, 05 June 2006
Way back in June 2004, I blogged about how a developer should be aware of their ability and actively push to increase their knowledge. Well working back with some junior developers again has prompted me to blog about the simple, basic things that all good developers just seem to know, and most developers starting out either don't know or don't care about. I am going to assume that most just don't know, and I am hoping help rectify that situation. With a my list of the top 10 things (in no particular order) that all junior developers need to know.

1. Code layout matters.
The layout of code is a significant factor in the maintainability of your code. Good layout with correct indenting and an appropriate amount of whitespace makes a huge difference in the amount of time it takes to analyse and understand the intent of code. For example, what the heck does this code do ?
public static bool Ckpal ( string str ){
bool found = true;
char[] chars = str.ToCharArray();

int i = 0;
int j = (chars.Length -1);
while ( (i != j) && (i != j-1)) {
if (chars[i] != chars[j])


found = false;

i++;j--;
if (chars[i] != chars[j])
found = false; }
return found;}
No idea? I wrote this (then modified it for this example), and even I have trouble understanding it. Layout should clearly show the control flow of the code. Here is a modified version that is formatted with improved layout.
public static bool Ckpal ( string str )
{
	bool found = true;
	char[] chars = str.ToCharArray();
	int i = 0;
	int j = (chars.Length -1);
	while ( (i != j) && (i != j-1))
	{
		if (chars[i] != chars[j])
		{
			found = false;
		}
		i++;j--;

		if (chars[i] != chars[j])
		{
			found = false;
		}
	}
	return found;
}

That is a lot better, but far from understandable, which conveniently leads me to point 2, naming. For more details on code layout, refer to: Code Complete, Chapter 31: layout and Style.

2. Use good naming rules for variables, methods and classes.

The goal with the names of your variables is that the intent should be clear and your code should be self documenting. The subject of naming and style can quickly head in to a religious war, about abbreviations, Hungarian notation, and Pascal case vs. Camel case. I am not going to go there, other than to highlight that with the code completion of modern editors, there is really no reason to use abbreviations. Another key point is that the intent of your variables and constants should be clear from their name, and that you should have an agreed standard that you use when more than one developer is working on the same code. To illustrate this, I have altered the code in the previous example by using good names.

public static bool CheckForPalindrome ( string StringToCheck )
{
	bool stringIsAPalindrome = true;
	char[] characterArray = StringToCheck.ToCharArray();
	int i = 0;
	int j = (characterArray.Length -1);
	while ( (i != j) && (i != j-1))
	{
		if (characterArray[i] != characterArray[j])
		{
			stringIsAPalindrome = false;
		}
		i++;j--;

		if (characterArray[i] != characterArray[j])
		{
			stringIsAPalindrome = false;
		}
	}
	return stringIsAPalindrome;
}

The intent if this code should be a lot clearer now, but it is still pretty complex, so let's add some comments.
// This method checks if a string is a palindrome
public static bool CheckForPalindrome ( string StringToCheck )
{
	bool stringIsAPalindrome = true;
	
	// Convert the input string into an array of characters.
	char[] characterArray = StringToCheck.ToCharArray();
	
	// Traverse through the array from each end simultaneously.
	// Comparing the characters as we go.
	int i = 0;
	int j = (characterArray.Length -1);
	while ( (i != j) && (i != j-1))
	{
		if (characterArray[i] != characterArray[j])
		{
			stringIsAPalindrome = false;
		}
		i++;j--;

		if (characterArray[i] != characterArray[j])
		{
			stringIsAPalindrome = false;
		}
	}
	return stringIsAPalindrome;
}
For more details on good names, refer to Code Complete, Chapter 11: The power of variable names.

3. If you can't build it, you can't ship it.
If you can't compile, deploy test your entire application at least once a day, you are kidding yourself if you think you will ever ship it, without major pain. Your application should be ready to release to your customers every day, the only difference from one day to the next should be the number of bugs, and the features that have been completed. Any application that spends every days on end in "pieces on the floor", will most likely never see the light of day. For more details refer to: Dynamics of Software Development, Rule # 32, If you build it, it will ship.

4. Be aware of the impacts of change.
The second that a public method gets saved into a shared code base, there is a chance that someone else may use that code. At one point I was managing a team, working on three sites with a shared codebase. Some of the most painful moments were when a developer in Melbourne, changed one of the methods in the common library without any concern for what the code in the other two development sites was doing. When class is built with public methods, you should be very careful before you just go and change a method by adding, removing or changing parameters. Ideally, for all your public methods you should use ndoc to produce MSDN style documentation, including examples of how the code should be used.

5. Don't manage change with optional parameters.
I have recently come to the conclusion that I really don't like Visual Basic 6. Ironically it is the things that you can do in VB that you can't do in C# that I don't like. The main culprit is optional parameters. Optional parameters make it too easy to slap some code together and just add another parameter on your function when you need to extend it. This however is fraught with danger as it can introduce some strange run time errors when you are calling the same function in different ways from different locations in your code base. The other thing that optional parameters let's you do is extend a function by adding just one more parameter for that special case. Then another when the second special case, comes along. Without optional parameters, you will have to think and consider options like re-factoring, adding another overload or adding a facade to your existing function.

6. Design before you code.
Building a house is easy, if you have never done it, all you need to do is grab some bricks, and start laying them on a slab, until you are done. That's how it's done isn't it? Um, no. There is one great example of a house that was built with no plans. The Winchester mystery house. It has 160 rooms, had 7 stories, has 467 doors, (some open to blank walls and steep drops), and took 38 years to build.

No sane person would build a house this way, yet some developers do exactly that every day when they write code. A significant number of developers don't take the time to learn about design patterns, UML and other design techniques, such a Test Driven Development. For .net developers, you should be a regular visitor to the Microsoft patterns and practices site.


7. Understand application support and software testing.
When I look back at the 15 years I have spent working in IT, I feel that the time that I spent working in various support and testing roles, has helped make me a significantly better developer than I would otherwise have been. If you ever find yourself asking to install development tools into a testing environment so you can diagnose a problem, that is a sure sign that you need to improve the supportability of you application. There is nothing like being on the front line supporting buggy software to appreciate the difference that good logging, error handling and supporting documentation make. When I switch gears in to developer mode, I probably spend 30% of my time writing defensive code that is designed to protect my app from the tricks that I would throw at it if I was testing. I usually then unit tests and throw it at the code anyway. The types of things I am talking about here are passing empty strings, negative numbers, boundary tests etc.

8. Learn how to estimate.
Ok here is an exercise. Estimate the number of seconds that it will take to read point 9 in this post. Record your estimate. Ok read point 9 below at your normal reading speed, timing how long you take. Don't worry I'll be waiting here for you when you get back. Ok, how did your estimate compare to your actual. When I did this test I thought it would take 20 seconds, but it took one 1 min 15 secs. Ok, now estimate how long point 10 will take to read, record it and measure the time it takes. See you soon. I estimated a minute and it took me 48 seconds. Where am I going with this. If you take the time to estimate your tasks before you do them, record it and then record the result, you will start to learn how long it really takes you to do things. You will then be able to provide estimates that will be some what realistic instead of just guess-timates.

9. Clarify requirements with assumptions.
Assume there are two developers, Bill and Ben. Both of them get a dodgy requirement: "Implement spell checking." Bill throws up his hands and fires off an email to the customer. "I don't have enough information to implement this requirement, please provide more details." Ben on the other hand, has a think about how he would expect it to work, documents his assumptions and asks the customer to clarify them. "I assume that you want spell checking to work as follows: The F7 key will open the spell check window. All misspelt words will have a red line underneath them. I am also going to assume that we can use the third party Superspell component from Superspell software, and that we don't need to support non-english languages." Can you clarify these assumptions for me please.

The difference in these two approaches is subtle, but they will yield a huge difference in response. If the customer does not know what they want, Bill may not get a response for weeks as they go off and research what they want. Ben may get a response much sooner, saying no they don't want red lines, but they do want to support non-english languages, but not until release 2. Ben has also already started to manage expectations and help define the product. Bill on the other hand, if he repeats this approach, may be viewed as being a hindrance to delivering the product, even if he is only trying to get the information he needs to do his job properly.

10. It's only a job.
Most people in IT can tell you of at least one project where they had to work very long hours, weekends and late nights. They can tell you that it was just expected of them, and at the end of it all they got was some overtime pay and a thank you. I hear stores like this time and time again. My advice to young players, is that you really need to balance your work life and your home life. You don't want to look back when you are older and wish you hadn't spent 15 hours a day, 7 days a week at work for the last three years and burnt yourself out. Don't misunderstand me that you shouldn't work hard, however, you need to work hard and smart.

We can waste a lot of time without even realising it at work, maybe surfing the web, maybe using instant messenger for a 15 minute conversation when we could have spent 2 minutes on the phone instead. Look at ways of making your work day 8 hours a day at work at 100%, instead of spending 12 hours a day at 66%. Developer productivity tools like CodeRush can make a huge difference for this type of thing. If your employer won't buy tools like CodeRush, then buy it yourself. Once you are producing more in less time, you should be able to spend less time at work for the same result. If your employer still expects you to still spend 12 hours a day, every day of the year, you should consider changing jobs.

Disclaimer  |  Comments [0]  |  Trackback

Building a GUI test automation framework: Mapping the GUI

Posted on Sunday, 21 May 2006
This post is the second in a series of posts, building a GUI test automation framework. In the previous post, I outlined that maintainability was one of our key design goals. There are two key techniques I use to develop maintainable tests.
1. Reducing the number of lines of code in a test to the absolute minimum.
2. Mapping the application's GUI so when a control changes, you only need to change it's definition in one place.

For this post I will base my example on the new Login control that ships with ASP.net 2.0.



Record and playback is evil
Recorded tests shouldn't be used in any serious test automation approach. To show why, I will record entering a username, password and pressing the login button in an unnamed commercial tool.

' Begin Record on Monday, 21 May 2006 at 6:45:39
' PM by BruceM

    ' Attach to Untitled Page - Microsoft
    ' Internet Explorer Window
    Window("Untitled Page - Microsoft Internet Explorer Window").Attach

    ' Attach to Caption='Untitled Page'
    HTMLBrowser("Caption='Untitled Page'").Attach
        HTMLEditBox("Name=Login1$UserName").Click 47, 11
        HTMLEditBox("Name=Login1$UserName").SetText "username"
        HTMLEditBox("Name=Login1$Password").SetText "password"
        HTMLButton("Name=Login1$LoginButton").Click

' End Record on Monday, 21 May 2006 at 6:45:51 PM
' by BruceM

First things first, the code is just plain ugly. Indenting and spacing is all over the place, the comments are bad and it follows no known standard that I know of.

Now let's say that we have this code at the start of every test, and we have 400 tests in our test suite. Our lead UI developer has decided to replace the login control with their own and he renames the login controls as follows:
"UserName" now becomes "txtUsername"
"Password" becomes "txtPassword"
"LoginButton" now becomes "btnLogin"
To change the control names in all our tests, we would need to change 1600 lines of code, and that is painful, not to mention a complete waste of time that could be spent elsewhere.

Enter the GUI mapping file
To avoid this situation, the fix is technically quite simple. If we define variables for all the controls in a single file, which all the tests refer to, we only need to update the mapping file in a single location instead of each and every test. Mercury's WinRunner uses this approach for it's GUI map files, which contain all the definitions in a single file.

If aren't using WinRunner, (I'm not), and your tool does not have a GUI map, you can simply extend your tool and "roll your own". I have used various solutions to build GUI maps from XML files and the MSXML DOM, to a class that is full of public constants similar to the following.
	Public Const LOGIN_PAGE as string = "Caption='Untitled Page'"
	Public Const LOGIN_USERNAME_TEXTBOX as string = "Name=Login1$txtUserName"
	Public Const LOGIN_PASSWORD_TEXTBOX as string = "Name=Login1$txtPasword"
	Public Const LOGIN_BUTTON as string = "Name=Login1$LoginButton"
This method is nice that with a good tool, you should now have code completion happening in your editor. You also don't have a runtime performance issue looking up controls in an XML file which you get with the DOM method.

Our updated code is now as follows:
' Begin Record on Monday, 21 May 2006 at 6:45:39
' PM by BruceM

    ' Attach to Untitled Page - Microsoft
    ' Internet Explorer Window
    Window("Untitled Page - Microsoft Internet Explorer Window").Attach

    ' Attach to Caption='Untitled Page'
    HTMLBrowser(LOGIN_PAGE).Attach
        HTMLEditBox(LOGIN_USERNAME_TEXTBOX).Click 47, 11
        HTMLEditBox(LOGIN_USERNAME_TEXTBOX).SetText "username"
        HTMLEditBox(LOGIN_PASSWORD_TEXTBOX).SetText "password"
        HTMLButton(LOGIN_BUTTON).Click

' End Record on Monday, 21 May 2006 at 6:45:51 PM
' by BruceM


Common GUI map gotcha's
Moving all your controls to a GUI map solves one problem but creates another. With all the controls now being defined by your test developers, you can have different developers define the same control using different standards. e.g.:
	Public Const LOGIN_PASSWORD_TEXTBOX  as string = "Name=Login1$txtPasword"
	Public Const TEXTBOX_PASSWORD  as string = "Name=Login1$txtPasword"
I recommend that you should develop a standard, and stick to it. The goal is that if anyone adds a new control, it will have the same variable name regardless of who defined it. A standard may be PAGENAME_CONTROLNAME_CONTROL_TYPE, or tlaPagenameControlName. (Where tla is the three letter acronym for the control type, e.g. txt for a textbox.)

If you can just use the same standard the developers are using for the application itself.

At this point you might be asking, if the test code produced is completely un-maintainable, why do the tools provide that feature. The answer is that tool vendors are in the business of selling tools, not selling maintainable automated tests. Providing record and playback in an automated test tool allows testers who can't code to quickly produce an automated test, by simply ignoring the maintainable test goal. Personally I only use record and playback to know what command I should be using to interact with a particular control, before rewriting that command myself.

In my next post in this series, I will cover using functions to reduce the number of lines of code in each test.

Disclaimer  |  Comments [2]  |  Trackback