Tejas Software Consulting Newsletter

v1 #6, August 2001

Rumors of the demise of asqnet.org's email forwarding were not exaggerated - faught@asqnet.org is truly no longer valid. If you haven't wiped it out of your records yet, please note that faught@tejasconsulting.com is now the way to reach me.
 

Contents

• CITE Test Harness Open Sourced - A proprietary tool suite is freed
• Tejas Newswire - News bits on the recent past and future events
• Feature Article - Tools, Tools, Everywhere, but How Do I Choose?

 

CITE Test Harness Open Sourced

CITE was supported on three Unix variants - HP-UX, ConvexOS, and SPP-UX. While CITE includes a suite of X Windows test tools called Xspecter, I believe these tools only support analog GUI testing, so practically speaking, CITE is a command-line oriented test manager. An early version of CITE is described in the paper "An Integrated General Purpose Automated Test Environment" by Peter Vogel, published in the proceedings of the ACM-ISSTA '93 conference.

Whether CITE becomes something that's useful and available on a broader range of platforms will depend on how much interest is generated in the open source community. I expect that it will need some work in order to productize it and get it working outside its previous environment. Contact Eric if you're interested in working with the CITE source code.
 

Tejas Newswire

My talk on "Event-Driven Scripting" last month went well. Putting perl code on a slide isn't easy, but it actually got the point across. You'll find the slides at  http://tejasconsulting.com/papers/event-driven/Event-Driven.htm.

My talk at the upcoming Software Test Automation Conference in Boston, August 29-30, is titled "Scripts on My Tool Belt". Let me know if you're planning to be at the conference - it's always good to match email addresses to faces.

Watch upcoming issues of STQE magazine for a book review from me, and also an article on scripting.

Business is going well for me, and I expect to be fully booked for several months. But there always must be that next project, so please let me know if you'd like to discuss how I could help with your company's software quality. Email has been piling up more than usual lately, so don't hesitate to call me at 817-294-3998 if I don't respond promptly to email.
 

Feature Article
Tools, Tools, Everywhere, but How Do I Choose?

Jenting Huang asked a question on the StickyMinds Software Testing message board recently: "With so many available testing tools in the market, how do you decide which one to choose? Are there any ratings/market research papers available on the Internet so we can focus on a few within our pricing range and required functionality?" This question is indicative of many similar questions that have been popping up lately. Others want to know where they can find a comparative analysis of some of the popular tools.

I can understand the frustration. There are 181 tools on the StickyMinds Tools Guide, and it continues to grow. In the Test Execution category alone, there are 67 tools!  Competition among tool vendors is wonderful, but at some point you have to narrow down the list and choose a tool.

There is plenty of good information out there about the process of selecting a tool. See Elisabeth Hendrickson's "Evaluating Tools" article, for example. What I'll focus on here is how to find the detailed type of information that Jenting was asking for. For market research, one option is the "Automated Software Quality Tools Forecast and Analysis, 2001-2005" report, available from IDC for US$2,500.00. You can explore the topics at the bottom of that page for other similar reports. I don't know whether market research reports like this are good investments for a tool evaluation, since I haven't used them before. Looking at the table of contents for some of them, it looks like really interesting information about tool vendors, but I can't tell whether they include enough technical information to help with a tool evaluation.

Regarding tool prices, we do ask vendors to include their pricing in the StickyMinds Tools Guide. However, most vendors choose not to do this. You'll have a hard time finding prices listed anywhere else as well. I believe there are a number of reasons for this. Prices tend to change, and seeing one price on a list somewhere and getting a higher quote from the vendor would be most upsetting. Also, pricing can be complex. To compare prices, you have to consider the licensing scheme and how licenses are bundled. This is difficult to represent in a simple matrix. So unfortunately, this will continue to be one of those things that you'll probably have to contact each vendor for in most cases.

There are a few places you can go to get independent tool reviews that will help in your search. I have a list of them on the "Other Sources of Tool Information" page at testingfaqs.org. The most well-known report is the Ovum "Software Testing Tools" evaluation service, which offers online access to reviews for test tools from nine different vendors, with quarterly updates. The reports are comprehensive, including prices and valuable information about the vendor that you might not be able to find anywhere else. The sample that I viewed was presented on a somewhat cumbersome string of Web pages. The price of the service gives some people sticker shock, but if you're considering tools from the vendors they cover, it's likely a good investment.

I've also started a bibliography of tool reviews on the "Other Sources of Tool Information" page. I'm looking for help from people who have seen tool reviews published in magazines and elsewhere, so I can list all of the tool reviews published in recent years. So please contact me if you see a tool review that hasn't been added yet. There are twenty-three reviews on the list so far, some on-line, and some available by ordering magazine back issues.

Another source of reviews is the book Quality Web Systems, due out in August 2001 and now available for ordering. Lead author Elfriede Dustin tells me that the book includes a tool evaluation matrix for tools from Mercury Interactive, Rational, Empirix, Compuware, and Segue in an appendix of approximately thirty pages.

Of course, you should also ask the vendors for information. They may be able to point you to independent reviews or their own competitive analysis. You will have to apply your own judgment, since vendors are unlikely to make you aware of reviews that they consider overly negative, and their competitive analysis will be heavily biased in favor of their tool. Nonetheless, such information provided by tool vendors is useful when it gives you detailed technical information and gives you questions to ask the other vendors.

For more informal information about tools, check online forums like the StickyMinds Message Boards. One forum with areas specifically set up for certain tools is QAforums.com. Check the Cigital Labs SRM Hotlist for newsgroups and mailing lists that can also be helpful. A few tools have mailing lists dedicated to discussions about the tool. But don't consider a query on an online forum to be a scientific survey--for any given tool, you'll likely find people who love the tool and also people who think it's worthless; the number of responses on each side won't necessarily reflect the market acceptance of the tool.

So your quest for help in evaluating tools may not be easy, and it may not be free, but there are some resources out there that you should take advantage of before making that important decision about purchasing a test tool. You're more likely to find information about the more popular tools, which makes it harder for the smaller vendors to get your attention. If you have unique needs that aren't met by the big vendors, you'll have a harder time finding tool reviews that are relevant, but the good news is that your decision will be easier once you've narrowed down the list to those that do meet your needs. Let's just hope that at least one good option is left on your list after the dust settles!