Usability testing at conferences
We reserved a conference-floor meeting room (which was included in the cost of our company’s booth) for the actual testing. We rented two monitors and had one power strip. Internet connectivity was not necessary for our testing, but we could have used wireless (for free) if we needed to. We shipped two servers and two desktops, with mice and keyboards, and an 8-port switch. Each of the desktops had Morae Recorder installed as well as an in-house Adobe AIR (formerly Apollo) application I wrote to allow users to pick descriptive words they would associate with the products after the test (thanks to Microsoft for the idea). We had everything up and running within an hour of carrying the boxes to the testing room. This setup gave us the capability to test two participants at once which increased our efficiency and allowed us to conduct more tests during open hours.
The tests were structured to take no more than 30 minutes, and would consist of several tasks with participant-tester interaction, followed by a SUS satisfaction survey, and the Apollo/AIR reaction-card application. In past testing sessions I would give participants tasks and then silently observe with very little interaction, but this time I decided to make the task-portion more interactive and I’m much happier with the results. At the risk of disturbing their natural problem-solving flow, the feedback we recorded provided us with excellent insight into positives and negatives with the designs: what was working and what wasn’t.
When it came time to recruit volunteers, I asked all of our booth presenters to add a slide to their presentations about the usability testing we were doing at the conference. I also offered anyone who would participate a $50 American Express gift card. I wondered whether it was too much, but surprisingly, it was still challenging to get people to sign up. Ultimately we filled every available slot we had, but not without some work (I recruited 3 people while riding the buses back and forth to the convention center).
All in all, it was a great success and provided us with valuable feedback and insight that we will take back to R&D and consider over weeks to come. Several small changes will be made resulting in a much better product prior to release. And despite all the positives I’ve discussed, there were some drawbacks to testing at a conference. Namely the noise, even inside the meeting rooms, as well as the general excitement level (it’s not exactly a “relaxed” setting). But at least for us, the positives far outweigh the drawbacks. And the final price? Here it is (rounded up)..