.: COLUMNS
Behind The Mask
Crossing Borders
Delusions of Grandeur
Guiding Lines
Komic Sutra
Library of Babble
Ouch, That Burns
Pak Talks Comics
Retroflect
Waiting for Trade

 .: VARIOUS
Contest
BF Vault
Links
RSS Feed

 .: ABOUT BF
The Staff
Contact
Site Map
Link To Us
Sponsorship

What We Don't Know
Lowdown by T Campbell, posted May 31, 2007

 

Recent events have forced me to face a fact that I have long avoided.

A strip I write, Penny and Aggie, is popular.

Not Barack Obama popular. Not The Daily Show popular. But popular enough.

For over a year, I had no idea what our numbers were. I avoided this knowledge. Knowing your pageviews can warp your creative judgment: too high and you get overconfident, too low, you get desperate. I felt I'd be less vulnerable to that effect if I waited until the series had found its identity. Besides, at that point I regarded Penny and Aggie as the fun project, and other things I was working on as the moneymakers.

But if I had known the numbers it was getting… it's hard to estimate how much my "statistical abstinence" has cost my artist and me, but it's probably in the thousands. On the other hand, if I had built a business plan around Penny and Aggie and we'd been doing badly, not knowing our figures could have been even more dangerous.

Could the webcomics field be facing a similar problem?

Do we know how well we're doing? Would we know if we had peaked?

What can be done about it?

And should we do anything at all?

Drawing In The Dark

Some cartoonists are probably happy to have no reliable way to track the field's progress. It's about art, not numbers. And there's that overconfidence/desperation thing. But numbers creep into our lives from unexpected angles. The celebrated numbers of Penny Arcade have certainly affected the webcomics field. I may not write a gamer comic (yet), but I look at the Web's bigger hits to help me understand the Web audience.

And as I decide where to take my career now, it does matter to me whether the webcomics audience is continuing to grow, and how much, whether it's continuing to fragment, and how much. A study I did in February implied the webcomics audience had dropped 8 percent over the course of 2006. The data source proved unreliable and the study didn't account sufficiently for new hits or the long tail, so I decided the figures were too flawed to be released to the general public. But I remained disturbed by the finding.

Within the larger webcomics field are a group of savvier businesscartoonists, and most of those don't have the ignorance I cultivated. They know their own numbers. They may know their friends'. Some collectives like Keenspot share the performance metrics of each of their members with each other.

On the other hand, asking for audience numbers is sometimes a faux pas, like asking for an income figure or a driving history. As I've learned from other projects, a poll would probably have a low turnout, especially from "the savvy set." Knowledge is power, and businesscartoonists have no incentive to disclose anything about their numbers that might scare off advertisers. Success will never come from a plan that relies on everyone putting the community's interest ahead of their own.

Other media face the same problem and have come up with some workable solutions. But applying those solutions to webcomics may bring new problems.

Print Kicks The Web's Ass

Print comics' numbers are so much better than webcomics', it's not even funny. Not bigger, necessarily, but better, more accurate.

Say what you will about Diamond Comics Distributors, but its effective monopoly of comic-book distribution means that almost everything in the direct market passes through its doors, and its market share data and growth stats leave little to the imagination. Newspaper cartoonists' income tells them how many papers are carrying their work, and each paper must disclose its circulation to advertisers. Book publishing is a bit more slippery, but The New York Times, Publisher's Weekly, USA Today and Nielsen Media Research have each used their considerable resources to track the book market.

Print deals in physical objects being produced in front of many witnesses, not infinitely reproducible digital objects that anyone can access at any time. Its business models are centralized, while the Web is still free and wild.

The Upper-Crust Accountants

The best hope for statistical tracking seems to be the "Nielsen" approach-- sample the behavior of a select percentage of users and generalize that to the whole. In television, Nielsen ratings have been the gold standard for over fifty years. Nielsen has taken its act online, where it competes with comScore, Inc. and Hitwise. Although no polling method is controversy-free, these three companies' results are highly prized.

And highly expensive. I spent a lot of time talking with representatives from all three companies, and we never established a price tag that was less than four figures. These are B2B companies. comScore's client list includes Pepsi, Borders, Yahoo and Google. Hitwise serves Virgin, Heinz, Honda and MTV.

What's more, comScore and Nielsen don't "drill down" far enough to present useful results for webcomics. Hitwise might, but it'd cost thousands just to test it. I regret to report that my spirit is willing but my wallet is weak, and I know of no one in the field who feels otherwise.

Free Yardsticks... Not Responsible For Height-Related Accidents

That leaves the statistical trackers aimed at the rest of us, and leading the pack in usage is the controversial Alexa, with Quantcast and Compete.com closing the gap. But numbers among the three vary wildly. Quantcast and Compete's counts of "real visitors" (whatever that means) fall far below the quoted statistics for Penny Arcade, PvP and Ctrl+Alt+Del and my own statistics for my own sites.

Quantcast and Compete bite off more than they can chew, but at least they attempt to provide the figures of the most immediate importance to website owners. Alexa is less easily refuted but also less relevant: its data measures the ratio of a site's performance to the performance of the entire Web, which is still expanding and poised for an explosion in China. So a website with steady traffic would slope downward. To slope upward, a site would have to beat the pace of the entire Web's growth. That's not impossible, but it seems like an unreasonable demand, increasingly so as the Web becomes more international and less of an English-speaker's club. And since Alexa makes no effort to convert these figures into "real" numbers (what does ".0004% reach" mean, precisely?), one is left simply musing about one's "Alexa numbers," which, like "Google PageRank," is a copyrighted form of measurement controlled by the rights holder, yet sometimes assigned equal importance to "open source" measurements like "unique visitors" and "conversion ratios." Pray no one ever copyrights the metric system.

Not Even Wrong

Each of these sites proceeds from a flawed methodology. Alexa, one of the first movers in this space and the most established, relies upon the steadily decreasing percentage of Web users who download and use the Alexa toolbar. Widely decried by Internet professionals, it perseveres, too established to be displaced by anything less than a major scandal or a clearly superior product. Neither Quantcast nor Compete is that product.

Quantcast's rhetoric is comfortably modest ("We're good but not yet perfect") and urges users to place its code on their own site to improve their listings. This depends on the cooperation of a smaller set of people than Alexa's toolbar system, but those people have a far greater interest in controlling data than the average Alexa user. What was the problem with this, again? Oh, yeah: "a plan that relies on everyone putting the community's interest ahead of their own."

But what's even worse about Quantcast is the wild swings between numbers for the sites that use its code and the ones that don't. Sore Thumbs, a strip whose numbers are in the vicinity of Penny and Aggie's, has its Quantcast figures insanely inflated relative to other webcomics (see below). How can anyone possibly trust a system that claims to measure all websites accurately, yet shamelessly boosts the figures of its "special friends?"

Compete has a brasher culture, and some of its boasts are backed by facts. It discloses that it uses a larger sample group than comScore's, while Quantcast, Nielsen and Hitwise keep this information off the record. While it uses a toolbar, it supplements the kindness of strangers with ASP data, ISP relationships and tightly controlled surveys, crafting a cocktail less vulnerable to skewing than any one method would be. And it doesn't shy away from the unpleasant truth that client-side traffic-counters sometimes overestimate total traffic.

Unfortunately, there are too many gaps between those facts. Compete's claims are largely unverifiable, because it won't go into specifics about any of its data-mining. We know it surveys 2 million users, but not how many of them are toolbar users, we don't know the contents of the ASPs or polls, and as Google technician Matt Cutts blogged, "Because you don't know which ISPs are selling user data to companies such as Compete or Hitwise, you don't know what biases are baked into those companies' metrics-- and the metrics companies won't tell you."

Scientists have a phrase for untestable theories. They call them "not even wrong." Compete data, like Alexa's and Quantcast's, is not even wrong.

Relatively Speaking

Talking with others about numbers over the years, I've heard many theories. One is that Alexa numbers have greater accuracy the higher in the rankings you get. Another is that while they may not represent actual visitor counts, the relationships between them could produce a reliable "top 10," "top 20" or "top 50." A year ago, Alexa had no serious competitors; now there are two. If relational data was confirmed through two sources, then such data would be-- if not absolutely trustworthy-- at least more trustworthy.

So I experimented.

The following is a result of a survey conducted over two days, using a list of more than 300 websites, copied in February from Wikipedia, Comixpedia.org and my own previously collected data. Not included are major comics publishers who are now venturing into webcomics like Top Cow, newspaper comics posted online, or webcomics-original "publishing brands" like Modern Tales. For the most part I wanted single-comic, single-cartoonist, or single-cartoonist-team sites, though I said "close enough" to webcomics-and-animation studio Explosm.net and to giantitp.com, longtime home of Order of the Stick and more recent home of Erfworld.

I publish this with some reluctance. I'm not afraid of the controversy that has dogged previous surveys of this nature, but I am concerned that it will be misunderstood or misquoted by the press or the blogosphere. The point is not that this is conclusive data: the point is that it isn't. Entirely useless? No, but we don't know how useful it is. It's the statistical equivalent of rumor. Rumor is sometimes fun, and sometimes worth investigating further. But it should never be confused with fact.

Sites Compete.com "people count"
www.explosm.net

235,821

www.vgcats.com

115,277

www.daybydaycartoon.com

98,154

www.toothpastefordinner.com

75,254

www.penny-arcade.com

72,432

www.nataliedee.com

57,297

www.giantitp.com

53,803

www.xkcd.com

51,229

www.nuklearpower.com

46,287

www.megatokyo.com

46,012

www.ctrlaltdel-online.com

44,571

www.coxandforkum.com

37,192

www.questionablecontent.net

32,140

www.leasticoulddo.com

29,298

www.qwantz.com

26,331

www.somethingpositive.net

20,194

www.pbfcomics.com

19,909

www.venisproductions.com

19,676

www.marriedtothesea.com

18,992

www.pvponline.com

18,360




Sites Quantcast Estimated Monthly Uniques
www.explosm.net

115,327

www.daybydaycartoon.com

94,300

www.vgcats.com

74,501

www.sorethumbsonline.com

68,177

www.toothpastefordinner.com

63,115

www.xkcd.com

49,406

www.penny-arcade.com

39,724

www.coxandforkum.com

37,172

www.giantitp.com

33,503

www.nataliedee.com

31,862

www.megatokyo.com

29,900

www.jerkcity.com

29,811

www.ctrlaltdel-online.com

23,961

www.nuklearpower.com

23,433

www.qwantz.com

21,115

www.theultrageeks.com

19,318

www.questionablecontent.net

17,298

www.pbfcomics.com

16,879

www.leasticoulddo.com

16,290

www.filibustercartoons.com

14,549




Sites Alexa Ranking
www.penny-arcade.com

2,312

www.xkcd.com

3,168

www.ctrlaltdel-online.com

3,847

www.explosm.net

3,958

www.vgcats.com

5,016

www.giantitp.com

5,900

www.pvponline.com

8,046

www.pbfcomics.com

10,191

www.questionablecontent.net

11,464

www.megatokyo.com

12,261

www.leasticoulddo.com

14,350

www.sinfest.net

16,518

www.userfriendly.org

16,618

www.nuklearpower.com

18,937

www.phdcomics.com

24,456

www.somethingpositive.net

25,036

www.smbc-comics.com

25,617

www.applegeeks.com

28,640

www.toothpastefordinner.com

30,043

www.dominic-deegan.com

30,714

As you can see, even the top spot is a bone of contention between the three services, and things get more interesting from there. Day by Day is #2 on one chart, #3 on another and not even in the top 20 on Alexa. Penny Arcade is #1 on the Alexa charts but behind Explosm and VG Cats on the other two. I admit there's less variation than I expected, going in. But this ain't nearly good enough.

What Can Be Done?

I confess, I'm eaten up by curiosity about Hitwise numbers and methodologies. If anyone reading this has access to a pre-existing account with Hitwise, they might be in a position to enhance our knowledge.

Of the six companies cited above, Compete is the most likely to change the game. That's still not very likely, but it's already done some things that Alexa and Quantcast would never have dared. If it can retain its competitive advantage while opening its accountancy up for inspection, then we may at long last have an accessible, reliable data set for the webcomics field.

We don't now.

And I can't share Compete's faith in "consumer-based community." Too many people in our community, too many different personalities. Say it with me: "a plan that relies on everyone..."

The best webcomics researchers can do is to be good researchers of the researchers. Question. Probe. Look for independent confirmation from two or more sources, and never simply accept what you're told. Don't confuse "rumor" with fact.

That's a fuzzy approach to numbers, which are supposed to be high-resolution. But as of now, that's the only reasonable response to the world in which we live.

Discuss this article on the Lowdown forum

Latest Articles

The Daily Read: 2/21
Posted February 21, 2008

Fast Breaks for 2/13
Posted February 20, 2008

The Daily Read: 2/19
Posted February 19, 2008

Untangled Web - Part 5
Posted February 19, 2008

The Daily Read: 2/18
Posted February 18, 2008



Print | Web | Film/TV | forum
Copyright © 2002-2008. Broken Frontier, All Rights Reserved Privacy Policy