Nielsen The Use and Misuse of Focus Groups
Summary: Focus groups can be a powerful tool in system development, but they should not be the only source of information about user behavior. In interactive systems development, the proper role of focus groups is not to assess interaction styles or design usability, but to discover what users want from the system.
Focus groups are a somewhat informal technique that can help you assess user needs and feelings both before interface design and long after implementation. In a focus group, you bring together 6–9 users to discuss issues and concerns about the features of a user interface. The group typically lasts about 2 hours and is run by a moderator who maintains the group's focus.
Focus groups often bring out users' spontaneous reactions and ideas and let you observe some group dynamics and organizational issues. You can also ask people to discuss how they perform activities that span many days or weeks: something that is expensive to observe directly. However, they can only assess what customers say they do and not the way customers actually operate the product. Since there are often major differences between what people say and what they do, direct observation of one user at a time always needs to be done to supplement focus groups.
Narrow View
Although focus groups can be a powerful tool in system development, you shouldn't use them as your only source of usability data. People with an advertising or marketing background often rely solely on focus groups to expose products to users. Thus, because advertising and marketing people frequently contribute to website development, focus groups are often used to evaluate web projects. Unfortunately, focus groups are a rather poor method for evaluating interface usability. It is thus dangerous to rely on them as your only method in a web design project. Traditional market research targets products for which usability is a minor concern. When judging, for example, which proposals a politician should support, how sweet a chocolate bar should be, or whether to show a new Mercedes braking in snow or in rain, you need expose a group of consumers only to different versions of the proposal, candy, or commercial, ask them which they prefer, and listen to their reasons as to why they prefer one or the other.
Software products, websites, and other interactive systems also need to be liked by customers, but no amount of subjective preference will make a product viable if users can't use it. To assess whether users can operate an interactive system, the only proper methodology is to watch one user at a time use the system. Because focus groups are groups, individuals rarely get the chance to explore the system on their own; instead, the moderator usually provides a product demo as the basis for discussion. Watching a demo is fundamentally different from actually using the product: There is never a question as to what to do next, and you don't have to ponder the meaning of numerous screen options.
Consider, for example, the problem of windowing versus scrolling as methods for changing the information visible on the screen. The windowing principle says that to see the information in the beginning of a file, the user moves the window to the top of the file. Scrolling, on the contrary, says that to see the beginning of the file, you scroll down the screen until the desired content becomes visible. In other words, the command to get to the top of the file should be called UP (or shown as an upward-pointing arrow) if windowing is preferred, whereas the same command should be called DOWN if scrolling is preferred.
When they actually carry out the task, most users perform better in the windowing model (which is therefore used in most current GUI standards). But if you give a demo of moving text files to people new to computers, many of them will say that the scrolling model characterizes what they are seeing (since they see the text move down to get to the beginning). If GUIs had been designed by focus groups, we would have ended up with a suboptimal command.
Benefits
In interactive systems development, the proper role of focus groups is not to assess interaction styles or design usability, but to discover what users want from the system. For example, in developing Sun's new online documentation system, we ran a focus group with system administrators to discover:
- their thoughts and preferences on issues such as distributing and replicating huge documentation files across multiple servers
- whether or not they needed faster access to local copies of the documentation on specific client machines
These questions would never emerge in a usability test (although we did run usability studies to see if administrators could operate the system). We could have investigated the needs of system administrators in other ways — including field trips to customer locations — but it was more efficient to have a focus group discuss the problems in a single session.
Getting Focused
For participants, the focus-group session should feel free-flowing and relatively unstructured, but in reality, the moderator must follow a preplanned script of specific issues and set goals for the type of information to be gathered. During the group session, the moderator has the difficult job of keeping the discussion on track without inhibiting the flow of ideas and comments. The moderator also must ensure that all group members contribute to the discussion and must avoid letting one participant's opinions dominate. After the session, data analysis can be as simple as having the moderator write a short report summing up the prevailing mood in the group, illustrated with a few colorful quotes. You can also do more detailed analyses, but the unstructured nature of the groups make this difficult and time-consuming.
Focus groups require several representative users. Because you need a flowing discussion and various perspectives, the initial focus group should have at least 6 users. Typically, you should run more than one focus group, because the outcome of any single session may not be representative and discussions can get sidetracked.
Other Issues
As with any method based on asking users what they want — instead of measuring or observing how they actually use things — focus groups can produce inaccurate data because users may think they want one thing when they need another. You can minimize this problem by exposing users to the most concrete examples of the technology being discussed as possible.
For example, Irene Greif ran focus groups to assess a version management facility for Lotus 1-2-3. The new features were presented to the focus group as a way to let multiple users compare alternative views of a spreadsheet across computer networks. Initially, group members were skeptical about these ideas and expressed distrust in networks and nervousness about what other people would do to their spreadsheets. After seeing a prototype and scenarios of version management in use, participants moved from skepticism to enthusiasm.
A cheap way to approximate a focus group is to rely on email, websites, or online communities. For example, Yia Yang started a project on undo facilities by posting on the British academic network, asking users which undo facilities they used and how they liked them. Posting questions to a group with an interest in the issues can generate considerable discussion. A disadvantage is that online discussions are difficult (or impossible) to keep confidential unless they take place on an intranet, behind a firewall.
Another disadvantage to this approach is bias. Internet users tend to be people with above-average interest in computers, and participants in online discussion groups tend to have above-average involvement in the group's topic.
Although online discussions are unlikely to reflect the average user's concerns, they can be a good way of getting in touch with power users. These users have needs that will sometimes surface later for the average user. Thus, addressing the power users' needs may be a way of getting a head start on future usability work.
References
Focus Groups: A Practical Guide for Applied Research (5th edition), by Richard A. Krueger and Mary Anne Casey.
- USA, Canada, and most countries outside Europe:
- Buy from Amazon.com
- Europe:
- Buy from Amazon.co.uk