Customer Survey Results
In September 2005 we contracted with Nielsen Media Research to conducted an extensive survey in order to learn more about our opportunity, characterize our target audience, and test specific design decisions. Using web-ads placed on sites oriented towards MMO and PC gamers, Nielsen selected over 600 active gamers who expressed interest in both Star Trek and MMOs. They then used additional filter questions sorted participants into three groups of about 200 each:
- MMO gamers: Those who play MMOs frequently with various levels of Star Trek enthusiasm.
- Star Trek enthusiasts: Those who expressed a high level of interest in Star Trek and had watched several series and many episodes.
- Gamers: Those with at least a minimal interest in Star Trek and MMOs who don't fall into the MMO or Star Trek enthusiast categories.
The purpose of selecting people who fit into one of these three groups was to ensure that we were getting feedback from people who might be interested in Star Trek Online and to understand our opportunity within various target audiences. We also wanted to ensure that we were hearing from people beyond those that identify themselves as Star Trek enthusiasts since we assume their opinion is under represented within the existing Star Trek Online community forums.
We asked this sample of our potential audience over fifty questions that covered everything from general game playing habits to demographic data, with a focus on MMO preferences and feature guidance. Below I'll cover some of the highlights from the results.
We were pleased to learn that MMO gamers with low-to-moderate levels of enthusiasm in Star Trek, express a strong interest in Star Trek Online. This tells us that, in general, players believe Star Trek will make for an interesting MMO experience even if they are not hard core fans of the franchise. They were followed closely by Star Trek Enthusiasts. This is very good news because it shows that if we do our job well, Star Trek Online has the opportunity to capture a significant segment of the existing MMO audience and has strong potential to bring in new MMO players from the vast, world-wide Star Trek Fan community. As expected, general PC gamers came in third place at 30% expressing moderate interest in a Star Trek MMO. Which given the size of that audience, again represents a significant opportunity. I should add that these numbers capture initial interest levels that were assessed early in the questionnaire before any details about the STO's feature set were revealed. Overall, we are very pleased with the level of interest expressed by all three groups and especially pleased that the idea of a Star Trek MMO scored so well within the existing MMO player community.
The idea of Factions for STO has long been a controversial issue within the team; Starfleet alone makes for a full development plate. On the other hand, we all recognize the value that factions bring to an MMO. So while we haven't ruled out the possibility of factions at release, at this time, they remain off the table. This decision has been in place for quite a while and hasn't moved. However, we were still curious to measure the perceived value of factions with this survey. We knew that given a choice, most players would choose factions. But we wanted to know if the lack of factions would turn people off. So we tested the faction concept against the non-faction concept by randomly alternating the two game concepts in the survey.
Boldly go where no one has gone before!
Join Starfleet and explore the greatest wonders from our star system and beyond. Find incredible discoveries in some realms - and dangerous threats in others! Team up with thousands of players as you graduate from Starfleet Academy, master your skills aboard a starship, explore strange new worlds, and seek out new life and new civilizations.
Experience Star Trek as never before - as a Starfleet officer, a Romulan, or a Klingon!
The time has come to choose sides: Defend the Federation as a Starfleet recruit, outwit your enemies as a cunning Romulan officer, or pursue honor in battle as a merciless Klingon warrior. Can these ancestral enemies unite to combat a re-emerging ancient threat, or will the galaxy plummet into chaos? Join with thousands of players to command starships in epic space combat, explore strange new worlds, and determine the fate of the galaxy!
On average, the introduction of factions lead to an 8% increase in purchase intent for MMO gamers, and 6% for Star Trek fans. That the difference was so marginal, was a surprise to the team. One explanation for surprising result is that a Star Trek MMO is appealing regardless of factions. The other is that factions are just not valued, which I find hard to believe. IMHO, factions would have a greater effect than is indicated in the survey, but these numbers certainly don't support an argument that we should find a way to make factions happen even if it means sacrificing other goals.
Another surprise was that the Borg is the most appealing opposing player faction. We knew the Borg would be a popular enemy, but we didn't expect that actually playing as a Borg drone would be so appealing. Though it was a surprise, I think we can understand why they would be a popular player faction. In fact, these results would appear to contradict the overall conclusion that faction value is marginal. One explanation is in the wording of the question. We asked people to select their favorite opposing faction. I can only guess that this wording had the effect of swaying people away from the Federation. Still, this point clearly deserves more research.
We took the opportunity to test interest in some high-level features to gauge both value and relative importance. We had a series of questions that asked players to choose their favorite characteristics or features of a Star Trek MMO game. Most of the features we asked about are planned in some form or another, faction related questions being the most obvious exception.
The first question asked responders to pick their five favorite high-level characteristics of Star Trek Online. We asked this question after presenting several one-sheet descriptions of STO that each took a different approach to describing the game. As you can see, starship related activities dominate the top of the list. We were surprised to see ground combat rank so low. At first we thought this might be because it is a feature that people take for granted, but its low rank was confirmed by the follow-up question that asked people to select their least favorites with the same list. Again ground combat showed up as a marginal feature. The away mission is a core component of Star Trek so it is a bit of a mystery why it ranked so low. My thought is that the answer lies in the how the feature is expressed. Perhaps if we had said "Away Missions", the feature would have ranked higher. Of course, this theory illustrates one of the challenges we face in making a Star Trek MMO: combat is a basic activity that is central to a fun MMO. Yet combat is on the periphery of most people's conception of Star Trek. This of course has been a challenge faced by all Star Trek games. Even with an MMO, a genre that is ultimately more about community than combat, we will always struggle to balance the needs of the interactive medium with the ethics of Star Trek.
The results' second general feature question also supports the notion that the space experience is our priority. This doesn't mean that we'll drop ground combat. In fact, even though ground combat doesn't get a high ranking on its own, we feel that giving players a chance to engage in combat as an individual on the ground and to combat in space with starships is a compelling feature. We think that having both modes of combat, without making two different games, will go a long ways towards breaking up the typical monotony of MMO combat. However, these results make it clear that above all else, making a brilliant space experience is critical to Star Trek Online's success.
We also asked potential buyers for their feelings about the solo vs. crew starship experience, and about combat preference for action vs. RPG style combat. As you can see from the results, we are on the right track to offer a starship experience that allows people to choose whether to captain a crew, crew for a captain, or go it alone, controlling the entire ship. When it comes to combat style, we were reasonably confident that we'd made the right decision to move away from FPS style combat, and the results support that decision. When asked to choose between action style combat or RPG style combat, 70% chose RPG style combat.
Another decision we wanted to test was our choice of playable races. We know that players will want to differentiate themselves and the more options we can offer, the better. Our current slate of playable Federation races is Vulcan, Bolian, Human, Trill, Bajoran, Andorian, and Tellarite. We not only wanted to know which of those would be most popular (though it wasn't hard to guess), but we wanted to make sure that we weren't making a mistake to exclude others. As you can see, players overwhelmingly want to play Humans, Vulcans, and Klingons when playing as a Starfleet officer. In retrospect, I wish we would have included ex-Borg and Android as choices in the survey. I'm guessing both would have been more popular than several of our current choices.
We gave our panel a list of specialties that doesn't exactly correspond to the choices we plan to offer. As I alluded to in an earlier blog post, Glen has re-factored the character classes (professions) to give players the chance to evolve their character's specialty over time rather than making all the choices up front. I'll let Glen describe the new system in detail when he's ready, but I will say that the general idea is to have a hybrid system that combines some up-front choices with the ability to specialize over time. We'd say that these results tell us that we're on the right track. More on this topic later...
It is not often that a development team has early access to focused feedback from a wide range of players who might eventually play their game. Usually, the first time a team is exposed to player feedback is after it is too late to respond. So we're very pleased with the results since we're in a great position to shore up our best choices and consider areas where we don't align with expectations.
And I should add that while consumer research is valuable, we shouldn't necessarily follow it blindly. In other words, it often provides useful validation and direcional input but it also should be used as just one part of a larger decision-making matrix in combination with development team experience and feedback from the community.
We're looking forward to reading your analysis in the forums.