Release 0.9

I recently interviewed Eric Schmidt, CEO of Google, for a video series sponsored by British Telecom. You can see it either at BT.com (registration required) or at interview iTunes. Or you can read on for my unapproved speculations about Google's future. I think it's clear Google needs to move more aggressively into personalization and also into more structured ways of looking at data/metadata. Note that Eric did not necessarily agree…

Personalized search

On personalization: It's inevitable, even though Google is reluctant to get caught up in the privacy issues around user data, especially after AOL's recent fiasco with user data…for which, not surprisingly, it is now getting sued.

Yet it seems clear to me that in order to make its search better, there's little that Google can do other than to understand what each user is looking for. The fat front of search is has been fairly well covered; the long tail needs knowledge about the searcher. Of course, it does no good simply not to collect personally identifiable information. As AOL demonstrated, people give away a lot by their searches. Better, I think, to take the opposite route, and to work with the users: Tell them that you're trying to make their searches better, and ask them to help you. One way to keep management lean is to give managers too much to do, says Schmidt. Make it easy – really easy – for them to turn tracking on and off, and then rely on their common sense to figure out what they want to reveal and what to hide. By trying to be unobtrusive, services end up feeling sneaky. Imagine a service that said: "This is what we know about you, and how." And that then invited you to edit the metadata. In the end, social engineering will be as least as important as technical engineering to do this right. Some early examples include mSpoke and RootMarket, all of which let the user participate.

But once the users want to play, then the system needs to be good at matching behavior and demographics (more…)

Just to set the scene, Dubai is one of seven emirates comprising the United Arab Emirates, a sovereign state; the UAE is a federation of jurisdictions with some local laws and economic regulations, and some central ones.  As everyone I met stressed, the UAE is not Saudi Arabia is not Egypt is not Syria is not Iraq is not Iran…and so forth. Although they may look the same from a Western distance, they are very different places. They have some common problems – most notably a regional political situation that spills over everything – and quite different demographics and local politics internally. 

 
The overwhelming impression is not just cranes everywhere, but that most of what is being built is luxury apartments, hotels or shopping malls.  The Emirates have about 2.7 million residents…and 8 million tourists a year.  The tourists cycle through a modern airport that is itself a huge shopping mall.  (The airport has solved the duty-free liquor problem – i.e. that liquids are forbidden on board - by checking it in for you so you can pick it up at your destination’s baggage claim.) 

There’s a feeling of commercial ebullience in the air: Real estate prices are going up, stock prices are going up, building are going up… and it’s all paid for by oil prices that are going up.

Yet there’s unease, too. Dubai is not just a casino-less Las Vegas on the beach; it’s also a capitalist edifice plopped into a culture that considers interest payments immoral. 

What will happen when all the buildings under construction are finished and a couple of those hundreds of cranes stand idle?  People talk of “Dubai fatigue,” and to the west Qatar (capital city: Doha) is positioning itself as the “new Dubai.”

Then there’s all the political turmoil, visible most recently in the destruction of 14 years of progress in Lebanon, the region’s most liberal, open country (which I visited in 2004).

And when will the oil run out? That’s a good question, even though I believe that will take longer than people expect because it will stretch further, and generate more revenues per barrel, as it becomes scarcer.  (More troubling – or promising, if you’re an investor - is the discovery/development of alternate sources.)

Those tensions are ever-present, and they may account for a bit of the pervasive materialism and the get-rich-quick mentality, which contrast with traditional Islamic values and much of the official story about the place.  That story includes a determination that the Gulf does not want to fritter away its money the way it did during the last oil boom.  This time around, the leaders want to build economies that will be healthy and sustainable even when the oil wealth goes away.  Yes, Dubai Holding is building hotels, but it is also investing in schools and hospitals. Then there’s Media City and Internet City, economic zones devoted to the development of Dubai’s new economy, with special affordances for start-ups.

Yet I still felt that the Dubai establishment - like so many establishments around the world - pays way too much attention to money and official structures and not enough to people issues.  The population of the UAE – very different from most other Gulf and Middle Eastern countries – is mostly foreign.  There’s a thin upper crust – about 7 percent – of locals, or “Emiratis.”  The rest is foreigners – large numbers of migrant/immigrant workers, mostly in service and construction jobs, from India, Pakistan and other countries, many of whom send money “home.”  (None of the employees of the Royal Mirage where we stayed was a native, but they came from more than 20 different countries altogether.)  Higher up the scale are expats: managers and knowledge workers who ply their trades and live in booming housing complexes: They can rent but generally (until just recently?) cannot buy.  They are the brains of the new economy, while the Emiratis are the owners and foreigners provide capital.  Foreigners can invest, but in most cases they need a local partner with at least a 51-percent interest. Of course, over time things will change, and foreign ownership is permitted within the Free Economic Zones, but you can imagine the distortions these rules must engender…

And then there are the real people issues: In South Africa (see previous post), there is affirmative action for a large, underprivileged majority, and the result is that the few well-trained people are leaving government for business posts.  In Dubai, there’s affirmative action for a small, overprivileged minority, who in the past simply played a role as owners or mostly silent partners.  Most of the locals who work, work in government… but the government now wants to push them into more productive private-sector jobs.  The challenges are well described in an excellent Financial Times article I read in the plane on the way home.

 **************

And in case you were wondering about IT: It’s still early.  Businesses are using the Internet, as are higher-income people, but overall penetration is low.

[CORRECTION :  According to this study, Internet use in Dubai is higher than I realized, at 40 percent some years ago. (Thanks, Juri Kaljundi!)  That’s much higher than in the surrounding region - but it’s still not seen as a mass channel by advertisers.  Maybe they are missing something.  For a more recent view, see this article.]

It didn’t seem worth asking about reselling WiFi to one’s neighbors.  As for VOIP, it’s banned in the Emirates, though it was available in Internet City. It is now banned even in Internet City, but the ban may be rescinded. Stay tuned. 

Yet another piece on  click fraud, this time from Businessweek – which of course is an “old medium” itself.  What amazes me is the elephant in the room: This is not an intractable problem. (Ask me about intractable: I just came back from the Middle East!)

Wherever the money flows, so does the information.  People can’t get paid without an arrangement with someone.  Yes, the miracle of the Internet is the ability to do so many things in an automated way, and of course there’s a chain of distribution and marketing partners and whatnot between Google and Yahoo! and the people who actually post the ads… 

But, just as with spyware, this thing won’t get cleaned up until the advertisers – the ones who inject the money into the system in the first place - start requiring more accountability from their partners, starting with Google and Yahoo! and ending where the money ends.   As a collection agent, Google and Yahoo! may not really care…until they are told to care.

 
So perhaps this story from Businessweek is just a sign of the market beginning to work: Advertisers will start to notice, their bosses and shareholders will start to complain, and marketers will get more specific in their contracts about what kinds of sites their ads can appear on.  And Google, Yahoo! et al. may  start to wield their market power on behalf of their advertisers. 

 
All it takes is better record-keeping and effective know-your-partner rules.  Is this some kind of Sarbanes-Oxley for the Web advertising business? 

For the record, I think SOX rule are overbroad where regular business is concerned. 

But in the click-through business as in spyware, where there’s a clear problem, the overhead  of keeping the system clean is better than the “fraud tax” we pay now.  The overhead of leaving the system dirty rewards the worst of its players.

This was a flattering piece in Crain’s New York (subscription only) about the New York Angels, including me, but it said that I had been an early investor in Google.  In fact, I had early stock, but through Kleiner Perkins and Angel Investors.  I  wish I had been that smart, but I’d prefer to correct the record.  Here’s what I wrote to the reporter:

"Thanks for the wonderful story! 

"However, as I made clear [in an e-mail to which you replied],  I did not invest in Google directly.  It was probably an editor’s attempt to make things short and punchy, but it ended up being inaccurate.  

"It’s not a big deal, but I would very much appreciate a brief correction of the facts,  making it clear that I did not mislead you. Thank you!" 
 

Word has slowly gotten out that the PC Forum last March was the last ever.  So let me interview myself:

 
What happened? 

 Well, with very few exceptions, each PC Forum would be better than the previous  year’s.  Yet, even though everyone was happy and the compliments mostly flowed in, I would spend the first couple of days after each Forum in a post-partum depression, thinking through the bits that had not been perfect, eager to do it again and do it better.  

But last March, amazingly, I was satisfied. I t was still not perfect, but I didn’t think I could do it better.  The whole thing flowed smoothly from start to finish, with questions raised by the early speakers resonating through the later discussions. (For the transcripts, see here.) We had finally reached Release 1.0 of PC Forum; it was ready to ship and declare complete.

 But what about the business decision?

 Daphne Kis and I sold EDventure Holdings (including PC Forum) to CNET almost three years ago.  From my perspective, one goal was to put PC Forum into other hands so I could do other things.  That has finally happened, though CNET has elected not to continue it with someone else at the helm.

So do you have a non-compete? Would you like to start over?

 Yes, I have a non-compete, but it’s not even necessary. I have no interest in doing another PC Forum.  I will be continuing to run Flight School, with Aspen Institute (save the dates: June 21-22). And I’ll be doing other events, as moderator, speaker  or panelist rather than organizer.  But I keep telling people they should change and take on new challenges, so that’s what I’m doing. It just took a while!

 I still don’t get it. Won’t you miss it?

 Let me tell you a true story from yesterday:   I was in an old gymnasium on Governor’s Island. There was a basketball lying on the floor, and a basket on a stand. I suppose I must have played basketball in high school, but certainly never since.  So I started trying to get the ball into the basket. It took about ten tries, and when I succeeded I quit.  It took almost 25 years for me to make the goal with PC Forum…

 What will you do instead?

 Well, I’m still an active employee of CNET Networks as editor at large and as the writer of this blog.  I also do a fair amount of private investing, speaking and general Internet troublemaking on the side.  None of that will change.

A client of CNET’s recently asked me to do a presentation on “important events in marketing,” and I got to thinking about it in the pool this morning.  Good marketing sings; it resonates with something that’s already in the recipient’s mind.  And that’s why I love it: A good message  expands when it hits its target. 

Here’s Release 0.9 of what I will say:

Note that this will be a somewhat personal tour, since I’ve been lucky enough to be associated with some of the great examples. 

The two campaigns that I like the most are brilliantly local and clever, in the best sense of the word…

The first was Federal Express.  I was a 20-something security analyst, and Federal Express was trying to get traction with a broad public for its revolutionary overnight delivery service. It did what good marketers did, and considered what made it different.   The first couple of years it focused on “absolutely, positively…overnight,” which was a new and seemingly impossible promise in those days, in messages delivered by an impossibly fast-talking man (produced by Ally & Gargano).

Then the company’s marketing whiz, the late Vince Fagan (one of FedEx’s terminals in Memphis is named after him), realized it needed to reach a broader market; call it the long tail of the overnight-shipping business.  FedEx started a flight of ads assuring “little guys” that it would talk to them too: One featured a single little guy in a small office surrounding himself with noise (radios, bells and the like) so that he could sound important enough when he picked up the phone to call Federal Express. The tagline, “Hello, Federal?” became famous…and so, ultimately, did Federal Express.

Some years later, in the early 80s, I spent a weekend at a company retreat with Michael Dell. Dell did not have particularly memorable ads, but it had a memorable positioning, which is what underlies (more…)

Michael Goldhaber is the real thinker behind the attention economy concept.  Ironically, he hasn’t really gotten the attention he deserves. He wrote an issue of Release 1.0 (my newsletter) way back in the early 90s, and he gave a stunning but mostly ignored talk at O’Reilly’s Web 2.0 conference last year.  (There was just too much going on, and you have to think to get what Goldhaber is saying.)

In  essence, he’s saying no, the attention economy is not just about how to get attention for things so that people will buy them.  Attention itself has value, and people will work hard to earn it… and will pass it out sparingly (if only because they have only so much time to pay attention to things).

I’ve always wanted to create a calculus of attention: One Henry Kissinger can get attention from 20 statesmen or from an audience of 2000 regular people. One of those statesmen can get 100 people each…  

But attention also depends on context:  Why do (would-be) important people  travel with a retinue? It’s not just to handle the money; it’s also to make sure there’s someone around to pay them attention. Take a high-level diplomat and put him in the wrong country, or take a high-powered financier and put him in a hamburger joint in some little town, and he’ll be ignored…

But  there’s a lot more, and it’s pretty radical.  It’s not trading money for attention, he asserts. It’s a really new economy, just as different as our industrial economy was from the feudal economy. (Personally, I think they are cumulative.)  

But pay attention! You can read Michael himself here.

I’m writing this as I leave South Africa – sitting in an airport lounge on WiFi (iPass via South Africa’s Internet Solutions) at the end of my fifth trip here over a span of six years.  I’m a member of the President’s International Advisory Council (PIAC) on the Information Society, which had its meeting over the weekend, and I stayed on for two days for the tenth birthday meeting of its ISP association,  ISPA,  in Johannesburg. 

Thanks for inviting me, Alan!   (Pictures galore on Flickr, including my favorite, which shows a sign welcoming visitors to the Johannesburg Zoo Lake Park. It has four green squares with icons of things you are invited to do; and below that are sixteen icons of  things you are forbidden to do.)

I’m thinking back to what my fourth-grade teacher often told me: “I wouldn’t criticize you if you couldn’t do so much better!”

South Africa has had so much promise for so long…  One reporter asked were we happy with the SA government’s directions and its willingness to take our advice… That’s not the problem; the problem is the speed with which teh advice  is implemented – or not.

But first – if you’re my typical reader, rather than a South African sharing my frustration with the delays – why should you care?  Because South Africa is the de facto first country of Africa, especially of sub-Saharan Africa. Like the US, it is not always loved, but it matters. Other African countries look to it for examples, to say nothing of skills and funding.  If South Africa succeeds, the rest of Africa may follow. South Africa has the most developed economy, arguably the best government and the best leaders, a moral edge in its victory over apartheid…but all those edges are wearing thin as it fails to exploit its advantages.  

South Africa has approximately 47 million people, of which approximately 3.7 million have Internet access - 2 million at work and about half a million at school or university.  The rest are at home or in other shared services. As for accounts, there are about a million dial-up accounts and about 280,000 broadband accounts, some of them residential. 

But Internet service is expensive: about $120 US for a typical 512K residential account.Indeed, one of the big problems in South Africa is the high cost of telecommunications, both analogue and digital.  There’s a shortage of (more…)

“I’ve seen this movie before and it doesn’t end well,” mutters a VC whose start-up is short of cash.

“We’ll always have Paris,” Rick tells Ilsa near the end of the movie “Casablanca.” 

Both these scenarios illustrate a favorite point of David Waltz, the natural-language expert whom I met while he was senior scientist at Thinking Machines: “Words are not in themselves carriers of meaning, but serve merely as pointers to shared experience.” …or something like that! The meaning should come clear from my examples.

In each case, there’s something complex and familiar that both parties recognize – something well beyond the capacity of words to represent without a sentient, intelligent being to condense them into a pointer at one end and to revive the words into meaning at the other.

When that doesn’t happen, you get scenarios like these: “Let’s have lunch sometime,” says Mr. Big Shot.

“Yes, that would be great!” says Little Worm. “Next Tuesday?”

“Ermm, actually, I’m quite busy next week… In fact, I’m tied up the rest of the month.”

Or “I’d like a red dress that flatters my figure,” says the shopper. She looks at the billowing red tent the saleswoman produces and says, “That is not what I meant. That is not it, at all.”

Natural language rules?

This is all to set up a series of posts on a current fascination of mine, pattern recognition. Pattern recognition means that you recognize a common pattern in a variety of instances…and that you can also produce instances to illustrate the pattern (which of course is exactly what I am trying to do here – illustrate a general theme of pattern recognition with examples).

Most computer programs say “if A, then do B.” Pattern recognition helps you determine whether A is true.

Pattern recognition takes a variety of forms, from object recognition and facial recognition to natural-language processing, which might more aptly be called “meaning recognition.” Pattern recognition ranges from recognizing a person in a crowd (useful to certain government agencies) to recognizing who’s likely at fault in a dispute, who is probably committing fraud, whether Juan’s a good match for Alice next door, which people will like a certain movie, what pitch is most likely to land a new advertising account, who designed a particular dress.

The inputs range from images to descriptions of behavior and numerical data, to natural language. You can do a lot of pattern recognition just with statistics, but only if you have enough data – and outcomes – or models, to start with. (That’s partly why I’m so hopeful about pattern recognition; there is more data everywhere, from people’s buying habits to GPS records of their movements, sensor data about all the things we see, electronic medical records that someday will follow a few standard formats so we can match behavior, therapies, genomes and outcomes.)

Some people are good pattern-matchers without ever articulating what they do; some (yes) recognize and can explain exactly what they are doing. (That is, in tech-speak, some people work like a neural net, producing results from a black box, while others work like an expert system, following explicit rules.)

And other people can read pattern-describing self-help books till they are blue in the face and still not recognize the situations in which they should apply the advice. Consider this piece of advice, for example: “Don’t ever ask a prospect who has said no to change his mind. Just give him a new proposition that he can agree with.” That’s how pattern recognition by a good salesman – and non-recognition (by the prospect who agrees with something that restates what he rejected before) – can work in business.

But back to software. A couple of weeks ago Stefanie Olsen of News.com wrote a piece about new forms of search. “Google is not the end of history,” I (and evidently a few other people) told her. Nor is search the final application…

Using pattern recognition in a variety of situations is the next, very diverse, frontier. If natural-language search is – or will be – the fat front of natural-language recognition, a wide variety of applications will be its long tail. Let me run through some examples from which you can divine my meaning. (I’ll be posting more of these over the next few days as I get the details fact-checked.)

I’ll start with the beginning of current history, Google. Google indexes words and phrases, and then uses the presence of those words plus popularity (the number of webmasters’ links to a particular page) to determine the ranking of the results – a list of pages where the search terms appear. In fact, Google’s search algorithms do a little more than that – fooling around with synonyms, eliminating stop words, possibly noting some metadata (authors and dates, for example) and other undisclosed “tuning” – but it is concerned with words, not meanings. And all it indexes or analyzes is text on the Web; it knows nothing about anything that is not in words, on the Web.

The future lies in moving beyond both those constraints. One is going beyond the Web, into “real life” and other media, such as television and films (and advertising); more on that later.

The other is expanding search (and other capabilities) to the meanings of those words on the Web: that is, to concepts, story lines, relationships – verbs, not nouns. Time-Warner acquiring AOL, for example, is very different from AOL acquiring Time-Warner… yet Google could not distinguish between those two. What’s enticing but not yet widespread is the ability not just to find relevant content, but to put the content into more regular form: for example, to build a table showing ten acquisitions of parts manufacturers by vehicle makers, listing the acquiring company, the acquired company and the amount paid including both stock and securities.

Google can get you lots of relevant (and irrelevant) articles, but it can’t fill in such table. It can get you a list of movies with Jennifer Aniston in them, but only IMDB tk link (compiled by people, often working for studios eager to promote their movies) can tell you the ones in which Jennifer Aniston starred.

That could change if we get better at natural-language understanding. Reuters is already pretty good at the generate-a-table-of-acquisitions task. However, as I heard the story, it also wanted to produce such news stories automatically by extracting data from press releases, but the reporters objected and insist on writing these formulaic stories by themselves.

Aside from whatever Powerset (mentioned by Olsen on News.com; I am an investor and it is still in stealth) is doing, there are lots of companies using natural-language pattern recognition in ways well beyond plain old search. Often, it’s a two-way process: The user supplies some information, the system makes some educated guesses, and ultimately a situation is recognized with the user doing quality control by saying, “Yes, that is right.” Then the appropriate action can be taken – whether it’s to remedy a situation (a dispute or a disease, for example) or to take advantage of an opportunity (an undervalued stock or an attractive purchase).

Here’s one you may not have thought of this way (or heard of at all), but then, pattern recognition is my job. SquareTrade is in the business of resolving petty business disputes; its major partner is eBay, for whom it provides dispute-resolution services to eBay buyers and sellers. (It also does quality checks and offers “good-behavior” seals.) The system begins by asking the user to fill in a form with simple questions: What was the item? What was the problem? What are the amounts at stake? [Disclosure: I have a small investment in SquareTrade.] 

SquareTrade uses a fairly limited vocabulary and resolves a fairly limited range of disputes. You could probably argue that it’s not AI at all. But that’s not really the point. The point is that it recognizes patterns: “This is the kind of problem where the product didn’t meet the buyer’s specs,” vs. “This is where it was broken during shipping,” or “This is one where the buyer changed his mind.” For each of these situations (As) there are fairly standard resolutions (Bs): “Send the product back and charge the seller two-way shipping costs,” “Fix the product and split the cost of repairs,” or “Send the product back, refund the money minus the cost of two-way shipping, but give the buyer a black mark for changing his mind.”

Imagine if we could do a better job of recognition, agreeing on the facts, and resolve more disputes with reference to a generally accepted set of rules. Imagine if SquareTrade could understand disputants’ free-form descriptions of the events rather than just have them fill in forms. And imagine if we could represent the body of law as an expert system, and then accurately recognize the situations and figure out which laws to apply. This is a bit of a digression; more on expert systems later.)

In another domain, using natural-language parsing without much regard to meaning, Vantage Laboratories uses recognition of grammar and usage  to grade SAT essays.

Similar techniques apply in health care: recognizing patterns to diagnose diseases, and then applying standard treatments. The challenge here is accurate observation – and the kind of probing a good doctor can do to uncover problems the patient is unaware of or doesn’t want to discuss.

More, and more intelligent automation, has been the dream in health care and education for decades, and it is beginning to happen here and there. One factor is that traditionally you needed the doctor or nurse to observe things. Now, there are devices that hook up to the patient directly and can feed physical data into a monitoring device. Those systems will find only what they have been trained to look for, but that’s a good start.

Of course, a good doctor or a good teacher is always better than a machine, but if there are not enough good doctors or good teachers, then machines can help fill the gaps and handle the routine cases. But more on real-world recognition next time…

Coming up soon: Recognizing real-world objects and the business models that could support .

…and recognizing consumers’ buying patterns – and the objects they buy

DISCLOSURE: I do not know how to drive a car. In fact, I have been weightless (on two Zero-G flights, for 16 minutes) longer than I have ever driven a car-like vehicle - twice, a golf cart in Florida long ago, and an electric car in Aspen this summer, for a minute apiece and with a co-pilot at my side. I have never bought a car, shopped for car insurance or an auto loan. I am as innocent about oil changes as a 50’s-era movie Dad was about diaper changes.

On the other hand, while I don’t know anything about driving a car, I do know something about the impact of transparency and online sales on marketplaces. I was on the advisory board of Orbitz until its acquisition by Cendant, and saw the delicate dances among the online services, the travel agents and the airlines.

All that probably makes me uniquely qualified to appreciate Zag.com, The first generation of car sites offered information – pricing transparency. Zag goes one step further and offers process – managing the workflow of buying a car and handling the related tasks of getting a loan and buying insurance. the latest run at making auto-buying more efficient, from Scott Painter, the man who founded CarsDirect.com in the ‘90s. Painter helped create the current environment of increasingly transparent car prices, with some help from Kelly Blue Book and Edmunds. And now, with Zag.com, he’s leading the Internet trend from cost-per-click (or lead) to cost-per-transaction.

Of course, most of the news about the car industry right now is pension problems, production cuts by GM and Ford, rumors of bankruptcy (probably floated in order to get the unions to "cooperate"), and gas prices and mileage. The overall US auto business, about $700 billion per year for new ($500B) and used ($180B) cars, hasn’t grown more than a percent or two a year for 20 years.

But dealers and buyers are having their own challenges.

On one side, consumers know now a lot more about car pricing, but they don’t have the negotiating power they could get as members of a buying group. And they have to go through a cumbersome process (more…)