gladwell dot com logo ARTICLES BOOKS NEWS THE NEW YORKER logo




October 4, 1999
ANNALS OF MARKETING
The Science of the Sleeper
How the Information Age could blow away the blockbuster


In 1992, a sometime actress named Rebecca Wells published a
novel called "Little Altars Everywhere" with a small, now 
defunct press in Seattle. Wells was an unknown author, and the 
press had no money for publicity. She had a friend, however, who 
spent that Thanksgiving with a friend who was a producer of 
National Public Radio's "All Things Considered." The producer read 
the book and passed it on to Linda Wertheimer, a host of the show, 
and she liked it so much that she put Wells on her program. That 
interview, in turn, was heard by a man who was listening to the 
radio in Blytheville, Arkansas, and whose wife, Mary Gay Shipley, 
ran the town bookstore.  He bought the book and gave it to her; 
she loved it, and, with that, the strange and improbable rise of 
Rebecca Wells, best-selling author, began.

Blytheville is a sleepy little town about an hour or so up the 
Mississippi from Memphis, and Mary Gay Shipley's bookstore--That 
Bookstore in Blytheville--sits between the Red Ball Barber Shop 
and Westbrook's shoe store on a meandering stretch of Main Street. 
The store is just one long room in a slightly shabby storefront, with 
creaky floors and big overhead fans and subject headings on the 
shelves marked with Post-it notes. Shipley's fiction section takes 
up about as much shelf space as a typical Barnes & Noble devotes 
to, say, homeopathic medicine. That's because Shipley thinks that 
a book buyer ought to be able to browse and read the jacket flap 
of everything that might catch her eye, without being overwhelmed 
by thousands of choices.  Mostly, though, people come to Mary Gay 
Shipley's store in order to find out what Mary Gay thinks they ought 
to be reading, and in 1993 Mary Gay Shipley thought people ought 
to be reading "Little Altars Everywhere." She began ordering it by 
the dozen, which, Shipley says, "for us, is huge." She put it in the 
little rack out front where she lists her current favorites. She wrote 
about it in the newsletter she sends to her regular customers. "We
could tell it was going to have a lot of word of mouth," she
says. "It was the kind of book where you could say, 'You'll love
it. Take it home.' " The No. 1 author at That Bookstore in
Blytheville in 1993 was John Grisham, as was the case in nearly
every bookstore in the country. But No. 2 was Rebecca Wells.

"Little Altars Everywhere" was not a best-seller. But there
were pockets of devotees around the country--in Blytheville; at
the Garden District Book Shop, in New Orleans; at Parkplace
books, in Kirkland, Washington--and those pockets created a buzz
that eventually reached Diane Reverand, an editor in New York.
Reverand published Wells's next book, "Divine Secrets of the Ya-Ya 
Sisterhood," and when it hit the bookshelves the readers and
booksellers of Blytheville, the Garden District, and Kirkland
were ready. "When 'The Ya-Ya Sisterhood' came out, I met with an
in-store sales rep from HarperCollins," Shipley said. She is a
tall woman with graying hair and a quiet, dignified bearing. "I'm
not real sure he knew what a hot book this was. When he came in
the store, I just turned the page of the catalogue and said, 'I
want one hundred copies,' and his jaw fell to the table, because
I usually order four or two or one. And I said, 'I want her to
come here! And if you go anywhere, tell people this woman sells
in Blytheville!'"

Wells made the trip to Arkansas and read in the back of
Shipley's store; the house was packed, and the women in the front
row wore placards saying "Ya-Ya." She toured the country, and the
crowds grew steadily bigger. "Before the numbers really showed
it, I'd be signing books and there would be groups of women who
would come together, six or seven, and they would have me sign
anywhere between three and ten books," Wells recalls. "And then,
after that, I started noticing mothers and daughters coming. Then
I noticed that the crowds started to be three-generational--there
would be teen-agers and sixth graders." "Ya-Ya" sold fifteen
thousand copies in hardcover. The paperback sold thirty thousand
copies in its first two months. Diane Reverand took out a single-
column ad next to the contents page of The New Yorker--the first
dollar she'd spent on advertising for the paperback--and sales
doubled to sixty thousand in a month. It sold and sold, and by
February of 1998, almost two years after the book was published,
it reached the best-seller lists. There are now nearly three
million copies in print. Rebecca Wells, needless to say, has a
warm spot in her heart for people like Mary Gay Shipley. "Mary
Gay is a legend," she says. "She just kept putting my books in
people's hands."

2.

In the book business, as in the movie business, there are two
kinds of hits: sleepers and blockbusters. John Grisham and Tom
Clancy and Danielle Steel write blockbusters. Their books are
announced with huge publicity campaigns. Within days of
publication, they leap onto the best-seller lists. Sales start
high--hundreds of thousands of copies in the first few weeks--and
then taper off. People who buy or watch blockbusters have a clear
sense of what they are going to get: a Danielle Steel novel is
always--well, a Danielle Steel novel. Sleepers, on the other hand,
are often unknown quantities. Sales start slowly and gradually
build; publicity, at least early on, is often nonexistent.
Sleepers come to your attention by a slow, serendipitous path: a
friend who runs into a friend who sets up the interview that just
happens to be heard by a guy married to a bookseller. Sleepers
tend to emerge from the world of independent bookstores, because
independent bookstores are the kinds of places where readers go
to ask the question that launches all sleeper hits: Can you
recommend a book to me? Shipley was plugging Terry Kay's "To
Dance with the White Dog" long before it became a best-seller.
She had Melinda Haynes lined up to do a reading at her store
before Oprah tapped "Mother of Pearl" as one of her recommended
books and it shot onto the best-seller lists. She read David
Guterson's "Snow Falling on Cedars" in manuscript and went crazy
for it. "I called the publisher, and they said, 'We think it's a
regional book.' And I said, 'Write it down. "M.G.S. says this is
an important book."'" All this makes it sound as if she has a
sixth sense for books that will be successful, but that's not
quite right. People like Mary Gay Shipley don't merely predict
sleeper hits; they create sleeper hits.

Most of us, of course, don't have someone like Mary Gay
Shipley in our lives, and with the decline of the independent
bookstore in recent years the number of Shipleys out there
creating sleeper hits has declined as well. The big chain
bookstores that have taken over the bookselling business are
blockbuster factories, since the sheer number of titles they
offer can make browsing an intimidating proposition. As David
Gernert, who is John Grisham's agent and editor, explains, "If
you walk into a superstore, that's where being a brand makes so
much more of a difference. There is so much more choice it's
overwhelming. You see walls and walls of books. In that kind of
environment, the reader is drawn to the known commodity. The
brand-name author is now a safe haven." Between 1986 and 1996,
the share of book sales represented by the thirty top-selling
hardcover books in America nearly doubled.

The new dominance of the blockbuster is part of a familiar
pattern. The same thing has happened in the movie business, where
a handful of heavily promoted films featuring "bankable" stars
now command the lion's share of the annual box-office. We live,
as the economists Robert Frank and Philip Cook have argued, in a
"winner-take-all society," which is another way of saying that we
live in the age of the blockbuster. But what if there were a way
around the blockbuster? What if there were a simple way to build
your very own Mary Gay Shipley? This is the promise of a new
technology called collaborative filtering, one of the most
intriguing developments to come out of the Internet age.


3. 

If you want a recommendation about what product to buy, you might
want to consult an expert in the field. That's a function that
magazines like Car and Driver and Sound & Vision perform. Another
approach is to poll users or consumers of a particular product or
service and tabulate their opinions. That's what the Zagat
restaurant guides and consumer-ratings services like J. D. Power
and Associates do. It's very helpful to hear what an "expert"
audiophile has to say about the newest DVD player, or what the
thousands of owners of the new Volkswagen Passat have to say
about reliability and manufacturing defects. But when it comes to
books or movies--what might be called "taste products"--these kinds
of recommendations aren't nearly as useful. Few moviegoers, for
example, rely on the advice of a single movie reviewer. Most of
us gather opinions from a variety of sources--from reviewers whom
we have agreed with in the past, from friends who have already
seen the movie, or from the presence of certain actors or
directors whom we already like--and do a kind of calculation in
our heads. It's an imperfect procedure. You can find out a great
deal about what various critics have to say. But they're
strangers, and, to predict correctly whether you'll like
something, the person making the recommendation really has to
know something about you.

That's why Shipley is such a powerful force in touting new
books. She has lived in Blytheville all her life and has run the
bookstore there for twenty-three years, and so her customers know
who she is. They trust her recommendations. At the same time, she
knows who they are, so she knows how to match up the right book
with the right person. For example, she really likes David
Guterson's new novel, "East of the Mountains," but she's not
about to recommend it to anyone. It's about a doctor who has
cancer and plans his own death and, she says, "there are some
people dealing with a death in their family for whom this is not
the book to read right now." She had similar reservations about
Charles Frazier's "Cold Mountain." "There were people I know who
I didn't think would like it," Shipley said. "And I'd tell them
that. It's a journey story. It's not what happens at the end that
matters, and there are some people for whom that's just not
satisfying. I don't want them to take it home, try to read it,
not like it, then not go back to that writer." Shipley knows what
her customers will like because she knows who they are.

Collaborative filtering is an attempt to approximate this
kind of insider knowledge. It works as a kind of doppelgänger
search engine. All of us have had the experience of meeting
people and discovering that they appear to have the very same
tastes we do--that they really love the same obscure foreign films
that we love, or that they are fans of the same little-known
novelist whom we are obsessed with. If such a person recommended
a book to you, you'd take that recommendation seriously, because
cultural tastes seem to run in patterns. If you and your
doppelgänger love the same ten books, chances are you'll also
like the eleventh book he likes. Collaborative filtering is
simply a system that sifts through the opinions and preferences
of thousands of people and systematically finds your
doppelgänger--and then tells you what your doppelgänger's 
eleventh favorite book is.

John Riedl, a University of Minnesota computer scientist who
is one of the pioneers of this technology, has set up a Web site
called MovieLens, which is a very elegant example of
collaborative filtering at work. Everyone who logs on--and tens of
thousands of people have already done so--is asked to rate a
series of movies on a scale of 1 to 5, where 5 means "must see"
and 1 means "awful." For example, Irated "Rushmore" as a 5, which
meant that I was put into the group of people who loved
"Rushmore." I then rated "Summer of Sam" as a 1, which put me
into the somewhat smaller and more select group that both loved
"Rushmore" and hated "Summer of Sam." Collaborative-filtering
systems don't work all that well at first, because, obviously, in
order to find someone's cultural counterparts you need to know a
lot more about them than how they felt about two movies. Even
after I had given the system seven opinions (including
"Election," 4; "Notting Hill," 2; "The Sting," 4; and "Star
Wars," 1), it was making mistakes. It thought I would love
"Titanic" and "Zero Effect," and I disliked them both. But after
I had plugged in about fifteen opinions--which Riedl says is
probably the minimum--I began to notice that the rating that
MovieLens predicted I would give a movie and the rating I
actually gave it were nearly always, almost eerily, the same. The
system had found a small group of people who feel exactly the
same way I do about a wide range of popular movies.

What makes this collaborative-filtering system different
from those you may have encountered on Amazon.com or
Barnesandnoble.com? In order to work well, collaborative
filtering requires a fairly representative sample of your
interests or purchases. But most of us use retailers like Amazon
only for a small percentage of our purchases. For example, I buy
the fiction I read at the Barnes & Noble around the corner from
where I live. I buy most of my nonfiction in secondhand
bookstores, and I use Amazon for gifts and for occasional work-related 
books that I need immediately, often for a specific and
temporary purpose. That's why, bizarrely, Amazon currently
recommends that I buy a number of books by the radical theorist
Richard Bandler, none of which I have any desire to read. But if
I were to buy a much bigger share of my books on-line, or if I
"educated" the filter--as Amazon allows every customer to do--and
told it what I think of its recommendations, it's easy to see
how, over time, it could turn out to be a powerful tool.

In a new book, "Net Worth," John Hagel, an E-commerce
consultant with McKinsey & Company, and his co-author, Marc
Singer, suggest that we may soon see the rise of what they call
"infomediaries," which are essentially brokers who will handle
our preference information. Imagine, for example, that I had set
up a company that collected and analyzed all your credit-card
transactions. That information could be run through a
collaborative filter, and the recommendations could be sold to
retailers in exchange for discounts. Steve Larsen, the senior
vice-president of marketing for Net Perceptions--a firm
specializing in collaborative filtering which was started by
Riedl and the former Microsoft executive Steven Snyder, among
others--says that someday there might be a kiosk at your local
video store where you could rate a dozen or so movies and have
the computer generate recommendations for you from the movies the
store has in stock. "Better yet, when I go there with my wife we
put in my card and her card and say, 'Find us a movie we both
like,'" he elaborates. "Or, even better yet, when we go with my
fifteen-year-old daughter, 'Find us a movie all three of us
like.'" Among marketers, the hope is that such computerized
recommendations will increase demand. Right now, for example,
thirty-five per cent of all people who enter a video store leave
empty-handed, because they can't figure out what they want; the
point of putting kiosks in those stores would be to lower that
percentage. "It means that people might read more, or listen to
music more, or watch videos more, because of the availability of
an accurate and dependable and reliable method for them to learn
about things that they might like," Snyder says.

One of Net Perceptions' clients is SkyMall, which is a
company that gathers selections from dozens of mail-order
catalogues--from Hammacher Schlemmer and L. L. Bean to the Wine
Enthusiast--and advertises them in the magazines that you see in
the seat pockets of airplanes. SkyMall licensed the system both
for their Web site and for their 800-number call center, where
the software looks for your doppelgänger while you are calling in
with your order, and a few additional recommendations pop up on
the operator's screen. SkyMall's system is still in its infancy,
but, in a test, the company found that it has increased the total
sales per customer somewhere between fifteen and twenty-five per
cent. What's remarkable about the SkyMall system is that it links
products from many different categories. It's one thing, after
all, to surmise that if someone likes "The Remains of the Day" he
is also going to like "A Room with a View." But it's quite
another to infer that if you liked a particular item from the
Orvis catalogue there's a certain item from Reliable Home Office
that you'll also be interested in. "Their experience has been
absolutely hilarious," Larsen says. "One of the very first
recommendations that came out of the engine was for a gentleman
who was ordering a blue cloth shirt, a twenty-eight-dollar shirt.
Our engine recommended a hundred-and-thirty-five-dollar cigar
humidor--and he bought it! I don't think anybody put those two
together before."

The really transformative potential of collaborative
filtering, however, has to do with the way taste products--books,
plays, movies, and the rest--can be marketed. Marketers now play
an elaborate game of stereotyping. They create fixed sets of
groups--middle-class-suburban, young-urban-professional, inner-city-
working-class, rural-religious, and so on--and then find out
enough about us to fit us into one of those groups. The
collaborative-filtering process, on the other hand, starts with
who we are, then derives our cultural "neighborhood" from those
facts. And these groups aren't permanent. They change as we
change. I have never seen a film by Luis Buñuel, and I have no
plans to. I don't put myself in the group of people who like
Buñuel. But if I were to see "That Obscure Object of Desire"
tomorrow and love it, and enter my preference on MovieLens, the
group of people they defined as "just like me" would immediately
and subtly change.

A group at Berkeley headed by the computer scientist Ken
Goldberg has, for instance, developed a collaborative-filtering
system for jokes. If you log on to the site, known as Jester, you
are given ten jokes to rate. (Q.: Did you hear about the dyslexic
devil worshipper? A.: He sold his soul to Santa.) These jokes
aren't meant to be especially funny; they're jokes that reliably
differentiate one "sense of humor" from another. On the basis of
the humor neighborhood you fall into, Jester gives you additional
jokes that it thinks you'll like. Goldberg has found that when he
analyzes the data from the site--and thirty-six thousand people so
far have visited Jester--the resulting neighborhoods are
strikingly amorphous. In other words, you don't find those
thirty-six thousand people congregating into seven or eight basic
humor groups--off-color, say, or juvenile, or literary. "What we'd
like to see is nice little clusters," Goldberg says. "But, when
you look at the results, what you see is something like a cloud
with sort of bunches, and nothing that is nicely defined. It's
kind of like looking into the night sky. It's very hard to
identify the constellations." The better you understand someone's
particular taste pattern--the deeper you probe into what he finds
interesting or funny--the less predictable and orderly his
preferences become.

Collaborative filtering underscores a lesson that, for the
better part of history, humans have been stubbornly resistant to
learning: if you want to understand what one person thinks or
feels or likes or does it isn't enough to draw inferences from
the general social or demographic category to which he belongs.
You cannot tell, with any reasonable degree of certainty, whether
someone will like "The Girl's Guide to Hunting and Fishing" by
knowing that the person is a single twenty-eight-year-old woman
who lives in Manhattan, any more than you can tell whether
somebody will commit a crime knowing only that he's a twenty-eight-
year-old African-American male who lives in the Bronx.
Riedl has taken demographic data from the people who log on to
MovieLens--such as their age and occupation and sex--but he has
found that it hardly makes his predictions any more accurate.
"What you tell us about what you like is far more predictive of
what you will like in the future than anything else we've tried,"
he says. "It seems almost dumb to say it, but you tell that to
marketers sometimes and they look at you puzzled."

None of this means that standard demographic data is
useless. If you were trying to figure out how to market a coming-
of-age movie, you'd be most interested in collaborative-filtering
data from people below, say, the age of twenty-eight. Facts such
as age and sex and place of residence are useful in sorting the
kinds of information you get from a recommendation engine. But
the central claim of the collaborative-filtering movement is
that, head to head, the old demographic and "psychographic" data
cannot compete with preference data. This is a potentially
revolutionary argument. Traditionally, there has been almost no
limit to the amount of information marketers have wanted about
their customers: academic records, work experience, marital
status, age, sex, race, Zip Code, credit records, focus-group
sessions--everything has been relevant, because in trying to
answer the question of what we want marketers have taken the long
way around and tried to find out first who we are. Collaborative
filtering shows that, in predicting consumer preferences, none of
this information is all that important. In order to know what
someone wants, what you really need to know is what they've
wanted.

4.

How will this affect the so-called blockbuster complex? When a
bookstore's sales are heavily driven by the recommendations of a
particular person--a Mary Gay Shipley--sleepers, relatively
speaking, do better and blockbusters do worse. If you were going
to read only Clancy and Grisham and Steel, after all, why would
you need to ask Shipley what to read? This is what David Gernert,
Grisham's agent, meant when he said that in a Barnes & Noble
superstore a brand like Grisham enjoys a "safe haven." It's a
book you read when there is no one, like Shipley, with the
credibility to tell you what else you ought to read. Gernert says
that at this point in Grisham's career each of his novels follows
the same general sales pattern. It rides high on the best-seller
lists for the first few months, of course, but, after that, "his
sales pick up at very specific times--notably, Father's Day and
Mother's Day, and then it will sell well again for Christmas."
That description makes it clear that Grisham's books are
frequently bought as gifts. And that's because gifts are the
trickiest of all purchases. They require a guess about what
somebody else likes, and in conditions of uncertainty the logical
decision is to buy the blockbuster, the known quantity.

Collaborative filtering is, in effect, anti-blockbuster. The
more information the system has about you, the more narrow and
exclusive its recommendations become. It's just like Shipley: it
uses its knowledge about you to steer you toward choices you
wouldn't normally know about. I gave MovieLens my opinions on
fifteen very mainstream American movies. I'm a timid and
unsophisticated moviegoer. I rarely see anything but very
commercial Hollywood releases. It told me, in return, that I
would love "C'est Arrivé Près de Chez Vous," an 
obscure 1992 Belgian comedy, and "Shall We Dance," the 1937 Fred 
and Ginger vehicle. In other words, among my moviegoing soul mates 
are a number of people who share my views on mainstream fare but 
who also have much greater familiarity with foreign and classic
films. The system essentially put me in touch with people who
share my tastes but who happen to know a good deal more about
movies. Collaborative filtering gives voice to the expert in
every preference neighborhood. A world where such customized
recommendations were available would allow Shipley's well-read
opinions to be known not just in Blytheville but wherever there
are people who share her taste in books.

Collaborative filtering, in short, has the ability to reshape the book 
market. When customized recommendations are available, choices 
become more heterogeneous. Big bookstores lose their blockbuster 
bias, because customers now have a way of narrowing down their 
choices to the point where browsing becomes easy again. Of the 
top hundred best-selling books of the nineteen-nineties, there are 
only a handful that can accurately be termed sleepers--Robert 
James Waller's "The Bridges of Madison County," James Redfield's 
"The Celestine Prophecy," John Berendt's "Midnight in the Garden 
of Good and Evil," Charles Frazier's "Cold Mountain." Just six 
authors--John Grisham, Tom Clancy, Stephen King, Michael Crichton, 
Dean Koontz, and Danielle Steel--account for sixty-three of the 
books on the list. In a world more dependent on collaborative filtering, 
Grisham, Clancy, King, and Steel would still sell a lot of books. But 
you'd expect to see many more books like "Divine Secrets of the Ya-Ya
Sisterhood"--many more new writers--make their way onto the best-
seller list. And the gap between the very best selling books and
those in the middle would narrow. Collaborative filtering, Hagel
says, "favors the smaller, the more talented, more quality
products that may have a hard time getting visibility because
they are not particularly good at marketing."

5.

In recent years, That Bookstore in Blytheville has become a mecca
for fiction in the South. Prominent writers drop by all the time
to give readings in the back, by the potbellied stove. John
Grisham himself has been there nine times, beginning with his
tour for "The Firm," which was the hit that turned him into a
blockbuster author. Melinda Haynes, Bobbie Ann Mason, Roy Blount,
Jr., Mary Higgins Clark, Billie Letts, Sandra Brown, Jill Conner
Browne, and countless others have recently made the drive up from
Memphis. Sometimes Shipley will host a supper for them after the
reading, and send the proceeds from the event to a local literacy
program.

There seems, in this era of mega-stores, something almost
impossibly quaint about That Bookstore in Blytheville. The truth
is, though, that the kind of personalized recommendation offered
by Mary Gay Shipley represents the future of marketing, not its
past. The phenomenal success in recent years of Oprah Winfrey's
book club--which created one best-seller after another on the
strength of its nominations--suggests that, in this age of
virtually infinite choice, readers are starved for real advice,
desperate for a recommendation from someone they know and who
they feel knows them. "Certain people don't want to waste their
time experimenting with new books, and the function we provide
here is a filter," Shipley says, and as she speaks you can almost
hear the makings of another sleeper on the horizon. "If we like
something, we get behind it. I'm reading a book right now called
'Nissa's Place,' by Alexandria LaFaye. She's a woman I think
we're going to be hearing more from."


 copyright 1999 Malcolm Gladwell