The Long Tail

Long Tail vs. Lessig

David Hornik, a smart lawyer-turned-VC friend of mine who teaches a class on Intellectual Property and Business at Stanford Business School, recently gave his students an interesting assignment: Consider the apparent contradictions between the Long Tail and the work of Stanford law professor (and Wired columnist) Larry Lessig in the context of advising Congress on the future of copyright law.

Good question. Let me take a whack at it myself.

I've noted before that some have spotted a conflict between Lessig and the Long Tail. Specifically, Lessig argues that most creative work doesn't have value for long and doesn't need as much protection as it gets. The Long Tail, in contrast,  argues that as the limitations of shelf space diminish and older content is kept available we are finding that demand continues for longer than we thought.

In Free Culture, Lessig writes (page 225):

Of all the creative work produced by humans anywhere, a tiny fraction has continuing commercial value. For that tiny fraction, the copyright is a crucially important legal device. For that tiny fraction, the copyright creates incentives to produce and distribute the creative work. For that tiny fraction, the copyright acts as an “engine of free expression.”

But even for that tiny fraction, the actual time during which the creative work has a commercial life is extremely short. As I’ve indicated, most books go out of print within one year. The same is true of music and film. Commercial culture is sharklike. It must keep moving. And when a creative work falls out of favor with the commercial distributors, the commercial life ends.

Contrast that with The Long Tail:

You can find everything out there on the Long Tail. There's the back catalog, older albums still fondly remembered by longtime fans or rediscovered by new ones. There are live tracks, B-sides, remixes, even (gasp) covers. There are niches by the thousands, genre within genre within genre: Imagine an entire Tower Records devoted to '80s hair bands or ambient dub. There are foreign bands, once priced out of reach in the Import aisle, and obscure bands on even more obscure labels, many of which don't have the distribution clout to get into Tower at all.

So Lessig says the commercial life of creative work is short. The Long Tail says it is, well, long. Lessig concludes that copyright is overprotecting stuff that doesn't need it, since it doesn't sell anymore. What does the Long Tail say about that?

On the face of it, it really does seem to disagree. I've argued, for instance, that the concept of "out of print" is anachronistic and will soon go away. Even worse (for the not-worth-protecting argument), I think the availability of archive content could lead to a real boom in new content as it attracts/creates a generation of remixers and others who can find ways to find new value in old wine.  More content available + more people who want to do stuff with content = more commercial potential. Whether you like copyright or not, it's no longer safe to say that it's irrelevant for older material.

But there's another way to look at this that offers a neat bridge between the two views. Many of those extracting new value from old content are not the original creators or rights-holders. Some of them are repurposing older material, and others are aggregators who have found ways to find new markets for material that's fallen beneath the commercial radar. Either way, they typically aren't the original record label, film studio, publishing house, TV production company or any of the other names that might be on the copyright declaration. They are someone else, probably someone entirely unexpected. This is, after all, the dawn of Remix Culture.

What's changed is the presumption that the primary rights-holder is the best at extracting the commercial potential of creative material. Instead, anyone can do it: the advertising company that remixes an old movie to sell a car; the Linux t-shirt done Warhol-style, or just plain old DJ magic. What you need to encourage this multiplicity of commercialization potential is tiered alternatives to one-size-fits-all copyright, from allowing derivative works (good marketing!) to shorter terms for the sake of the remix-culture social good. I can't think of a better example of that than Lessig's own Creative Commons, which has already become the license of choice for the right side of the Tail, where the commercial imperative is less all-consuming. 

So, bottom line: the Long Tail ends up in the same place Lessig does, but via a different path--the diversification of commercial potential rather than the absence of it. I hope that the hypothetical Member of Congress to whom Hornik's students were addressing their arguments can see the big picture. The shelf life of creative works may be growing, but so is the social cost of locking them up at the sole discretion of the primary rights-holder.

At the same time, I'm delighted to see that more and more content producers are discovering the appeal of voluntarily renouncing some of their statutory copyright protections (which is what adopting a Creative Commons license does) to encourage their work to take on a life beyond that they originally intended. The Long Tail is full of such cross-fertilized fare. We don't yet know how the money will eventually flow through this collaborative network, but it's clear that without more flexible intellectual property rights, it won't flow at all.

Long Tail FAQ: What about producers?

I've been planning to start a Long Tail FAQ, beginning with a question I get all the time:  " I can see what the Long Tail offers consumers and distributors, but are there equal advantages for producers?" The answer to that will be at least one chapter of the book, so it's a lot for a FAQ. But the short form is "Yes". The supply side of the Long Tail is a rich topic that neatly compliments the demand-side focus of most of my writing to date.

Kevin Kelly has now prodded me into FAQ action with a great production example--how to self-publish on Amazon. It's a comprehensive, step-by-step guide, and it begins by answering the "why?":

I began listing self-published material on Amazon because I wanted a way to reach the wider public with my stuff but I did not want to have to deal with shipping out copies to each customer who ordered on my website. For a small-timer like me, mailing out, and keeping track of onesies and twosies is very disruptive for my day job. By having my stuff on Amazon, Amazon's mighty enterprise became my shipper (they are very good at this), so the only place I have to ship my copies to is to their warehouse.

More importantly, as popular as my website may or may not be, it doesn't compare to the traffic headed to Amazon to search for books and DVDs. By having my stuff pop up among the big publisher's offerings for "similar books" or even in reader's lists and guides, my titles gain a greater chance to be seen and ordered. In a certain way, unless your stuff is available on Amazon, it ain't available.

This example encapsulates two big Long Tail advantages for producers. The first is access to market.  Only a distributer/retailer with effectively infinite warehouse and shelf space would be willing to carry Kevin's self-published books (as distinct from his very successful commercially-published books, Out of Control and New Rules for the New Economy.) Even then, the process has to cost the retailer as close to nothing as possible, something Amazon accomplishes by making the entire transaction self-service.

The second advantage is low-cost marketing. Recommendations, whether by software filters or network-amplified word of mouth (from blogs to viral emails), can be far more powerful than traditional advertising, especially for niche products aimed at narrow markets. This isn't a cure-all for obscurity: much of what's produced far down in the Tail just isn't very good. Other products are good but simply aren't for everyone, and the people they are for aren't easy to find.  And there recommendations can make all the difference in connecting niche producers and niche consumers.

Two other quick points about how the picture is improving for the producer: As Kevin notes, the tools of creativity--from cheap digital video gear to GarageBand--are being democratized at a dizzying pace. Every Mac now ships with music, video and photo editing software that rivals the professional tools of just a few years ago. If you have talent, there's little stopping you from doing something about it, and millions are. Although this isn't a consequence of the Long Tail (it's just a function of the spread of digital technology), it does have the happy effect of populating the Tail at a pace never before seen.

Finally, it's worth noting that commercial success is not the only (or even main) reason to be a Long Tail producer. Most authors write books not to get rich but to reach a readership, whether it be to enhance their academic reputation, market their consultancy, or just leave a mark on the world.  The Long Tail effect may not pay their rent, but it will find them a bigger audience, and if what they're offering is really good it may be dramatically bigger. As Kevin puts it, self-publishing on Amazon "is not a way to make money; it's a way to distribute your message." As powerful as the Long Tail is for commerce, it's potentially even more powerful for communications. And for producers of all sorts, that's good news.

In Defense of Endism

First, apologies for the thin posting last week. I was in New York  for the usual frenzy of back-to-back meetings, both Long Tail and otherwise.  (I did get to see the Gates, but sadly only at night. It's not the same).  I'll try to catch up with a few quick posts here on some loose threads, and then return to your regularly scheduled stream of observations, answers and other half-bakery.

One of the eyebrow-raisers in the media world last week was the transcript of Michael Wolff's keynote remarks to the 2005 SIIA Information Industry Summit in New York City. [Disclosure: Michael is the media columnist for Vanity Fair, which is a corporate sibling of Wired. ] It was controversial (so much so that he reportedly asked for it to be removed from the site that originally posted it) because in the midst of some quite smart observations about the challenges facing mainstream media he made a rather broad swipe at the Wall Street Journal, claiming that it lost its currency with the chattering classes when it put its online content behind a subscription wall.

I, however, thought it was interesting for an entirely different reason: flagrant "endism".

In the Q&A, according to the transcript, Wolff suggested that Consumer Reports was also suffering from consumer rejection due to its policy of charging for access to its website. "It once occupied a very clear position in the opinion market, and it now occupies a kind of marginal position," he said. "Marginal" is one of those words that means "effectively dead" and represents the sort of wording that Wolff wields often to great effect; his latest book, after all, is "Autumn of the Moguls".

A questioner then challenged him, and his response was far more reasonable:

AUDIENCE MEMBER: Consumer Reports has almost two million paying subscribers, so I'm just curious. This thing you keep saying about how paying for content devalues it ... I'm confused by that. Can you just elaborate on that a little?

WOLFF:  Well, I think Consumer Reports at one time was the brand in product evaluation. That's what you would say: "Check Consumer Reports." But if you want to buy something now, it's "Check the Web." It's not that Consumer Reports doesn't have a business, but it has certainly lost its position as the grail of product evaluation.

Actually, Consumer Reports says it has about 4 million subscribers, which by anyone's estimation makes it non-marginal (although I don't get the magazine, I do pay $26/year to subscribe to its excellent website). Vanity Fair, or for that matter Wired, would love to have anything close to that.  But I think I know what Wolff meant. To understand his claim, you have to realize that there are three kind of people, which being a science geek, I will describe in physics terms (that noise is the sound of a readership stampeding for the exits):

    A) Position People
    B) Velocity People (first derivative)
    C) Acceleration People (second derivative)

Category A people think: "4 million subscribers is a lot. Consumer Reports must be doing something right."

Category B people think: "It used to be 4.2 million. Consumer Reports is in decline."

Category C people think:  "They lost 200,000 readers in three years! Consumer Reports is dead."

Now I should quickly add that I have no idea if Consumer Reports is indeed losing readers; I just made those figures up for illustrative purposes (and to add to the evidence that blogs are not journalism).  The point is that you probably know all three kinds of people I've described, and you may have noticed that while the first two perspectives are accurate, the third, which is clearly false, is the one that gets all the attention. People are drawn to grand overstatement, especially if it's in service of a broader point.

The pinnacle of grand overstatement is endism, the declaration that the moment something stops growing it is effectively dead. In this, Wolff is in good company. If a corporation stops growing, Wall Street investors not only treat it as roadkill but usually clamor for the CEO's head on their wall. After a former colleague of mine at The Economist wrote "The Death of Distance" about the communications revolution, another colleague wrote a piece noting the several hundred other business  books whose titles start the same way. And we at Wired do it all the time, including our new cover story on "The End of Radio" (although in fairness we did add an "as we know it").

The reason endism is not necessarily a sin is that all three of those perspectives are legitimate. It is, of course, important for some people to keep their feet on the ground and focus on the absolute reality of what's around them, thus category A. It is also important for some people to keep an eye on trends, thus category B. Finally, a few people should be looking around the corner, trying to spot leading indicators that may turn into trends and finally into absolute reality, which leads us to category C, where Wolff and Wired occasionally meet.

At The Economist the unofficial internal motto was "simplify, then exaggerate". This is obviously a successful model in sure hands; the art of analysis is knowing the line between dramatic distillation and  reductio ad absurdum. The secret, of course, is to be right. If Radio is indeed about to be transformed by technology and readers are persuaded by our evidence and arguments (as I'm sure they will be; it's an excellent package, which should be online later this week), then they will understand our meaning of "End".

We at Wired, like Wolff, live in the whiplash world of acceleration space, where tiny fluctuations in trend velocity can either be blips or the beginnings of the next big thing. We're pretty good at telling one from the other, and thus I think our occasional endism is seen as the effective rhetorical device it is. Industries really do crumble and reshape, and it's our mandate to spot the signs first.

As for Wolff, he too has a point. As much as I like Consumer Reports' website, I tend to go to Amazon reviews and even Google first because I'm more likely to find a recent review of whatever I'm thinking about buying. And I'm a subscriber;  non-subscribers don't even have that choice. I think of Consumer Reports mostly for big-ticket items and consumer durables, like cars and washing machines. That's enough to keep me as a subscriber, but it's not enough to retain the position as "grail of product evaluation".  Does that make it "marginal"? Perhaps not, but it's worth noting all the same.

 

Can blog spam be solved like email spam?

As always, the gnarly questions that come up when I give a speech are the best part of the experience. On Thursday I spoke at The Media Center's Emerging Technology conference at Stanford University, where I was asked a toughie: could bad actors drown the Tail in a tide of  spam and other robotic graft?

The fear, to put it simply, is that the Tail is fragile and that niche quality can easily be overwhelmed by industrial-scale automated marketing, fraud and vandalism. After all, the signal-to-noise ratio of the Tail is already low; could digital grafitti push it to insignificance? The fall of the once-great Usenet, which collapsed under the weight of autospam and trolls, is the cautionary example.  Will the same happen to the blogosphere and all the trust networks and other recommendation mechanisms that we use to pull diamonds from the rough?

I think the answer is no. At the conference I said, controversially, that I felt a combination of technologies had finally come together to turn the corner on the email spam problem and I didn't see why the same approach wouldn't work for comment spam and other automated web scams.  In short, I think the good guys, backed by smart technology, will win in the end. (Ross Mayfield did a fine job covering the talk here.)

My confidence comes primarily from seeing the spam problem, which was until recently deemed an out-of-control epidemic, get reduced to a very manageable two-or-three-a-day annoyance in my own case. I use four spam filters, two on the server and two on the client, and the result is the graph above, taken from just one day. Although more spam than real email is sent to me, almost all of it is filtered out with virtually no false positives.

(Note: I know I may not be typical, but I do think I'm not a total outlier either. I have a public email address, so I get a good bit of spam. And of those four filters, three are automatically provided by my company, ISP or email software. I only pay for the fourth. In other words, most people's problem are not as bad as mine and their solutions are equally within reach, so their experience shouldn't be much worse than my own.)

I call this multiple-filter approach a "cocktail therapy", intentionally invoking the drug combinations that have shown such success in fighting AIDS. The best strategy is to attack simultaneously on many front.

My spam-blockers are a combination of Bayesian filters and networked blacklists. Although they all play a role, I think the most powerful among them is Cloudmark's SafetyBar, which uses collaborative filtering. Its client software fingerprints email that you identify as spam and sends that fingerprint to all the other Cloudmark clients out there. If enough people with enough good track records call something spam, it probably is. The client software automatically moves a blacklisted email into the spam folder for everyone else, saving them the trouble of clicking on it themselves. (If someone mistakenly calls something spam and others reverse it, the mistaken user's rating will go down, reducing their influence until they earn enough confirmatory credit again). This is The Wisdom of Crowds at work, market forces applied to fight automated crime.

I think we can combat the problem of comment and trackback spam using the same forces--democracy and shared information. The computers that are depositing links to porn sites in my comments and trackbacks (now all deleted, albeit by hand) are doing the same at other blogs. Yet the implicit data contained in my deletions is lost if it isn't shared with those other sites. A little bit of software can make all the difference, recording the thumbs-down "votes" of vigilant bloggers pruning their comments of spam and automatically passing that information on to other blogs, where the deletions can happen with no human intervention.

There are, to be sure, differences between email and comment spam. For starters, most comment spam occurs in the parts of the web only visited by machines: dusty archives patroled just by spiders and, unlike your own inbox, rarely seen by humans. Furthermore, Google has added financial fuel to the fire by making links, even if nobody follows them, essentially free money, which is more than can be said for most herbal Viagra email spams.

But the similarities are even more striking. Already, an open source project called MT Blacklist (MT=Moveable Type, the blogging software from SixApart that underlies the TypePad hosting services that I use, too) is showing how collaborative comment-spam filtering might work. It's still pretty crude and not yet automatic, but Jay Allen, its creator, describes the promising future of the project here.  Another advance is Google's "no-follow" tag, which allows bloggers to deprive comment spammers of Google juice

The low-hanging fruit for this is hosted blogging services such as TypePad, where everything is done on a central server and the information about who-is-deleting-what can be easily shared.  I was pleased to hear that the Cloudmark and SixApart folks have met to talk about this. "We're going to be adapting the lessons of the email spam world to comment spam," Michael Sippey, SixApart's VP for Products told me.  "We're looking at ways to include community feedback mechanisms into that filtering process. " No announcements yet, but I'd be surprised if within the year my trackback and comment spam wasn't automagically cleaned up as rapidly as my inbox now is.

Google's Long Tail

On Tuesday Google had their analyst day, which was focused on articulating their strategy and explaining the underlying dynamics of their remarkable success. I've described Google as a "Long Tail company" before, so I was delighted to see that this is the way they now describe themselves, too. You can watch the webcast here (the image above is slide 10) The San Jose Mercury News report (free reg req'd) on the meeting also discusses the Long Tail strategy.

What Google has done is to find and monetize the Long Tails of both advertisers and publishers. These include millions of small companies and individuals who may never have advertised before, at least not nationally. They were considered sub-scale--too small to be worth a call or visit from an ad salesperson, possible too small to even think of themselves as an advertiser at all. But Google ads are self-service, cheap, and performance based (pay-per-click), which all combine to dramatically lower the barrier to entry.

Matching these advertisers are hundreds of thousands of previously sub-scale "publishers", from blogs to niche commercial sites. Most are too small to have their own ad sales business, but they can now run relevant Google ads by just adding a few lines of HTML to their site.  About half of Google's business now comes from such "partners", rather than from ads sold against search results themselves.

Eric Schmidt, Google's CEO, explained how these millions of small-to-midsized customers represent a huge new Long Tail ad market. "The surprising thing about The Long Tail is just how long the tail is, and how many businesses haven't been served by traditional advertising sales," he said. Google now has revenues of more than $1 billion a quarter, a least half of which is made up of Long Tail advertisers by this definition. This is, needless to say, just a glimpse of what's still to come.

Briefly noted

I've had a lot of email and comments encouraging me to stick with longer essays here, which is much appreciated. I'll try to get in one or two a week. In the meantime, I'll satisfy my own target of daily posts by extending my "Long Tail comment elsewhere" format to the main bar here, making it part of the feed. (The Six Apart folks have not yet implemented RSS in TypeLists, which is what the sidebar is, although they say that's on the way).

I'll start with a fantastic post on RSS by Kevin Laws, a venture capitalist who helped me articulate and think though the original Long Tail theory. RSS, as you know, is a really big deal and an important step in maximizing the potential of the Long Tail.  The key element of the publish-and-subscribe model is that it allows readers to indicate an interest once (by subscribing to a feed), and then get relevant content when it's published, without any thought or effort required.

Kevin describes this with the best analogy I've heard yet for RSS: Tivo for the Web:

One direction [in which RSS is expanding] relates to automating tasks for you. This is basically the return of agent technology. Now that a wider variety of web sites are available in machine readable format, it should be possible to tell your computer things like "tell me when an article about gnosticism appears". While this is similar to the stored searches on Google, the fact that RSS aggregators are closer to real-time makes this more valuable. The best analogy is "Tivo for the Web" - specify web sites to definitely "record" and the agent can also record a selection of potentially interesting web posts.

Unlike email, RSS feeds aren't mixed in with personal messages and subject to spam filtering, and there's no obligation for content to be written in the form of a newsletter.  The atomic element of RSS is the single post, which works perfectly well in the context of a reader that lays out many posts on a page  (like the fab Bloglines, which has just been sold to my friends at Ask Jeeves.)

The reason this is so important to driving demand down the Tail is that RSS feeds can provide a constant stream of links and suggestions for products and media that you otherwise wouldn't have heard of. Best of all, they don't have to be from conventional media and blogs; they can simply be notifications of availability or updates on what's selling where.

One of the best examples is Netflix's wide range of RSS feeds, which include a new release feed. It's a quick read, complete with paragraph summaries, of a hundred or so new DVDs a week. Two clicks on one that grabs my attention and it's in my queue, soon to appear at my door. They've also got RSS feeds for the latest bestsellers in niche categories that range from Independent and Foreign to Gay and Lesbian and Anime.  Invariably interesting, and often useful.

When theory meets practice

You know that post where I argued that as more viewer read via RSS the subscription model would favor the occasional thoughtful post rather than many random miniposts designed to drive traffic? Well, there was one bit I got wrong: I underestimated the ego reward of watching visitor stats climb during the day.

The problem with RSS is that there's no good way to track who's reading what. For full feeds like my own there's no need to click through and thus generate a page view. So there's a strong temptation for me to post more frequently to stir up more linkage and thus measurable traffic from outside the feed, even if that doesn't necessarily mean many more readers.  Sad but true.

Unfortunately I'm in book-writing mode, which means that all ideas tend to spiral out into thousand-word essays, far too long for blog conventions. Plus it's hard to write an essay a day along with the book and my day job.  Thus a policy reversal: until either my ego needs diminish or RSS readership becomes measurable, I'm going to try my hand at shorter posts that riff off of other people's thoughts--like, well, all the other bloggers.

Rather than using such ugly words as "derivative" or even "lazy", I'd like to spin this as an exuberant embrace of intellectual sampling. This is, after all, the Remix Age.

Transistors don't have sex on the roof

(Although I'm now back home, I'm catching up on my posts from India while the impressions from that extraordinary trip are still fresh)

In Bombay, a hypercompressed city of 17m people (nearly the population of Australia), the value of the individual is bound to be diminished.  Even knowing that, I was struck by the casualness with which servants, driver and other laborers are deployed.

By casual, I mean this sort of thing: On the chance that you might want a car and may not want to flag one of the 65,000 taxis (true number) that circulate constantly, virtually all of them empty, a driver is asked to wait all evening outside your meeting place.  Houseboys actually appear at the ring of a bell; it's a favorite game of one friend's three-year-old. "Send your man" is the suggestion for any errand, no matter how slight. And street vegetable stands home-deliver orders of less than a dollar.

The class divide in India is extreme; democracy has arrived but not egalitarianism. Yet there is little complaining. Waiting hours on end on the whim of richer people is considered a relatively good job compared to the alternatives.

Coming, as I do, from the land of the overscheduled, the amount of idleness on display in Bombay is shocking. This is a nation of grossly underutilized labor and talent.  As I looked out a restaurant window one evening on the tide of empty Ambassador cabs lit by blue neon from within, I wondered whether this overabundance of human potential has an analogy in technology.

The enabling insight of the Silicon Age was Carver Mead' s counterintuitive 1980 call to "waste transistors".  Until then most of the focus in software was on writing tight, efficient code that made the most of the scarce (and typically shared) computing resources available at the time. But the era of the personal computer was dawning, and with it the arrival of more MIPS on the desktop than most people knew what to do with. Mead suggested that programmers imagine that transistors--computing power--were free. Rather than protecting them, they should think of ways to exploit them without a second thought.  And thus was born such computationaly extravagant eye-candy as the graphical user interface--and the Mac and Windows.

The same can now be argued for storage, memory and bandwidth. The original iPod's 10GB hard drive seemed profligate compared the flash memory of the average MP3 player of the time, but it introduced the concept of taking your entire music library with you everywhere you went, rather than having to anticipate ahead of time what mood you might be in when a moment to listen came. And that is why Manhattan is now a sea of white earbuds. Likewise, I carry a month's worth of  email on my Treo because there's no reason not to; 1GB SD cards are cheap.

In all the computers that surround us, the CPU idles virtually all the time, spinning its gigahertz wheels waiting for us to give it a command. Carver Mead's advice has been taken. Standing on the street in Bombay, I think of 65,000 taxi drivers doing the same.

Is the analogous lesson of India "Waste People"? Is India's key competitive advantage the ability to think of people not as individuals but as available resources, deployable in the millions? It may look that way from a Silicon Valley being depopulated by outsourcing,  but I think the answer is actually no.

I imagine that somewhere in one of the planning ministries of New Deli there is a technocrat who is nevetheless thinking that way. But then I think of the call center workers I've met and, more importantly, their parents, who hold the managers of the operation accountable for the moral and professional integrity of their daughters.  I think of the 90% annual turnover rate at these centers (comparable to the employee churn in the American fast-food industry). And I think of the security guards in the parking lots who are there not to protect property but to stop the newly independent 20-year-olds from having sex in the cars. The doors to the roof are locked for the same reason.

Until you come to India you cannot really understand what a traditional society it is, and how important the family remains (arranged marriage is still the norm). This is the limiting factor in India's ability to scale its outsourcing industry. The call centers have tapped out the local population of qualified kids and must now import them from around the country. Rather than living at home until they are married (and sometimes after), these young men and women are now living in shared apartments in the big city and working all night with members of the opposite sex, to say nothing of the exposure to American culture. The natural happens in a culture not yet ready to accept it.

It is statistically tempting to look at the 30m Indians entering the workforce each year, many of them college educated and English-speaking, and see them as cheap and available labor on a scale unimaginable in the West. Yet meet a few and you are reminded that each one of them is a person with a family, and culture sometimes takes precedence over economics. Carver Mead's slogan created Silicon Valley, but its analogy may not work in India. Outsourcing will hit its limits in India's traditions, and their ability to absorb the social effects of globalisation. There is no Moore's Law of culture.  Transistors don't quit under parental pressure. Transistors don't have sex on the roof.

The end of objectivity

For one of the book's chapters I've been exploring some of  the profound differences between scarcity thinking and abundance thinking. The Long Tail is all about abundance--unbounded shelf space, as it were, leading to profligate inventory expansion: everything, all the time.  But there are many other manifestations of abundance all around us. One of them is in my own industry, media, where the end of distribution bottlenecks is changing some of the fundamental rules of journalism.

The traditional premium on impartial journalism is a function of media scarcity: if you are the main or sole source of news you have an obligation to be balanced. That was certainly once true of America's newspapers, which in a big country are distributed by city, almost invariably in ones or twos. And the rest of American media took its journalistic-standards lead from newspapers.

But the UK is different in that it has long had a national newspaper market. Thus there was no news scarcity and newspapers differentiated themselves by taking sides.

Today in the US the newspaper is fading, as is its influence on American journalism: news and information is becoming a commodity. What will rise as a differentiating competitive advantage? I'd argue that it's not so much pure opinion and political partisanship (although that's been the case on radio) as it is sensibility and worldview.

Perhaps the best example of sensibility is The New Yorker, which has a distinctive voice and perspective that, one assumes, had its origins in the cultural life of the Upper East Side of Manhattan (disclosure: they're our corporate sibling at Conde Nast).  You'd never confuse it with a newspaper--it assumes too much of the reader, both in intellegence and attention span, and appeals by making its audience feel like they've joined a somewhat exclusive club of smart, sophisticated people.

But sensibility doesn't have to be posh. Maxim and FHM have a sensibility (embrace your inner dog), as does MTV. Perhaps the best examples are blogs, which at their best have a distinctive and human voice, driven by the interests, values and sensibility of their author.

Worldview, on the other hand, tends to take the form of writing that does not so much seek to be balanced and comprehensive as it does to argue a case or give informed perspective and analysis, often reflecting a consistent philosophy (environmentalism, libertarianism,  globalism, and plenty of positions that aren't "isms", too). 

Examples include my alma mater The Economist (worldview: free markets), Fox News (American triumphalism), and my own Wired (change is good). What worldview shares with sensibility is that the writer's voice is louder than in traditional journalism, and his/her own observations and reactions are less suppressed.

I see both of these as part of the fall of "dispassionate media" and rise of what, by contrast, one might call "passionate media".  I think passionate media is the only kind that will cut through the blur of commodification in the years to come. And I think that we, as readers (and writers!) can handle the lack of quasi-impartial hand-holding just fine.

Dan Gillmor calls all of this "the end of objectivity".  I agree.

Why Social Software Makes for Poor Recommendations

Another interesting question that came up at the Nokia workshop was whether social software networks, such as Friendster, would prove to be a good source of recommendations to filter the Tail. At first blush it makes sense that they would.  After all, whose recommendations would you trust more than those of your friends? But the truth, parodoxically enough, is that strangers typically do an even better job.

The problem with social software as a recommendation network has its roots in the problem of social software itself. "Friend" is a pretty blunt instrument when it comes to describing relationships, especially in matters of taste. The sad reality is that most of my friends  have rotten taste in music (I don't hold it against them), while the music recommendations I actually follow are mostly from people I've never met, be they Rhapsody editors or MP3 blogs.  Same for virtually every other narrow category where I need advice; odds are that the real subject matter experts aren't anyone I know.

In other words, the assumption that there's a correlation between the people I like and the products I like is a flawed one.  To use an analogy, Bill Joy, the co-founder of Sun Microsystems, famously uttered this truism (now known as Joy's Law):  "No matter who you are, most of the smartest people work for someone else." The same might be said of recommendations. No matter who you are,  someone you don't know has found the coolest stuff.

Compounding the problem, the people whose recommendations I trust in music are different from those whose recommendations I trust in movies. Gadgets are yet another group of mavens, as are games and books. Indeed, although I have dozens of "trust networks" (usually formed by reputation and experience, not personal relationships), most of them have nothing in common with each other, and almost none of them I consider friends. Some of them aren't even human--they're software.

In a sense, you can think of all your filters as being part of orthogonal trust networks, often with the only common member being yourself.  They rarely, if ever, overlap. Thus any service that tries to condense all of your different planes of influence into a single dimension is going to fail, at least as far as useful recommendations go. That isn't to say that such services shouldn't offer playlist sharing and Amazon wishlists, only that I'm likely to find better advice elsewhere. 

The filters that work best for me typically earned my trust by liking some of the same things I did, then turning me on to new stuff that I liked even more.  I really don't need to know anything more about these people, other than that they've got more time than me and are willing to listen to a lot of junk in search of undiscovered gems. And they, in turn, don't care much about me. When it comes to recommendations, friendship is overrrated.