The Long Tail

The case against the Shuffle

I was at Nokia in Helsinki on Tuesday running a workshop on the Long Tail. I was impressed by how much brainpower they had in house, and how much they were inviting in. Just in the past week I'd been preceded by Henry Jenkins, Cory Doctorow, Esther Dyson, CK Prahalad and probably a few others that I'm forgetting about.

One of the interesting questions that came up during the workshop was about Apple's new iPod Shuffle. In provacateur mode, I described it as a "value subtract" product, arguing that the lack of a display would limit its success.  I got a fierce reaction from one of the Nokia's researchers, who thinks it will be a huge hit because of its passive entertainment value: with near-zero effort on your part, the Shuffle offers a fresh selection of music from your collection, both randomly copied from your hard drive and played in random order. As Apple's marketing puts it: "iPod Shuffle adds musical spontaneity to your life. Lose control. Love it."

This was a good opportunity to put Long Tail theory to the test--what would the rules I've set out predict about the Shuffle's commercial future? The answer: they argue against it. Here's why.

You can think of the thousands of songs on your hard drive as a Long Tail of sorts. Even though you've bought, ripped, or downloaded them all, they're not all your personal "hits". Some of those tracks are songs you don't care for on albums you otherwise like; others are albums you wish you hadn't bought or ripped in the first place. And yet others are songs that you've simply grown tired of.  In other words, the signal-to-noise ratio in your own collection can be nearly as variable as that in any commercial music service.

The difference is that the commercial services provide recommendations, editors' picks, bestseller lists and collaborative filtering to suppress most of the junk. This is important: a key element in extracting value from any Long Tail is the ability to find the diamonds in the rough, which gets harder the further into the rough you go.  In other words, a Long Tail without good filters is just noise.

So that, in a nutshell, is the case against the Shuffle.  For anyone with a big music collection (thousands of tracks) a random walk through their entire library is statistically likely to be an unwelcome reaquaintance with mistaken purchases, whim rips, filler album tracks and embarrassing ghosts of music taste past.  And if you're anything like me, that gets annoying real fast.

Now, I know that the shuffle feature can be turned off and you can load the device with any playlist you want.  But then it's just another mini music player, save the quite useful feature of telling you which track is playing and allowing you to choose another without having to memorize their order. That's what I meant by "value subtract." No doubt they'll still sell millions, such is Apple's brand and momentum. But I don't think the Shuffle will have anything like the impact of the original iPod itself.

Maximum City

There's an old joke in the newspaper business that the best way to turn a front page story into a paragraph on page 50 is to add the words "in India". Which is as good an argument for a blog as I can think of. Having zero marginal costs of paper and ink, I'm free to spill a few bytes over the next few days touching on what I found most remarkable in this extraordinary hotspot of globalization (and don't worry, it's not totally off topic; stick around and you'll see why).

India is a true interzone. It's an economic force that is transforming Silicon Valley, yet is one of the poorest countries in the world (about half  the GDP/capita of China and behind many African nations). Boomtown cities like Bangalore are building luxury housing as fast as they can pour the concrete, yet a third of the country can't read or write. I came because you can't understand the world today without understanding India. But I also had some specific questions, which will be the subjects of my next posts. This one is just to set the scene.

I started in Bombay (as it's generally called in English; "Mumbai", the official name, is more commonly used in Hindi) which is, I will not be the first to say, the bomb. There really is nowhere like it on the planet. Nothing with its mind-boggling density (1m people/square mile in places), its vertiginous social divisions, its baroque history and, yes, its abject poverty. This is a place where the dazzling 24/7 digital hubs of the outsourced economy stand just yards from the worst slums I've ever seen, and that's saying something. It is simultaneously exhilarating (technology, music, really interesting social change) and terrifying (roads--yikes!, disease, deprivation).

Like the emerging industrial powerhouses of east Asia (China, Korea, Taiwan), Bombay has become an obligatory stop on the travel schedule of the global CEO. Unlike the others, it still sometimes puts those CEOs in the hospital on a drip. To hear the somewhat hysterical warnings, water is practically poison: hot and cold running giardia, dysentery and hepititus B. You can't imagine how hard it is to take an entire shower with your lips tightly sealed. Everyone I know who fell victim can recall their moment of slipped vigilance: one remembered too late the washed salad; another forgot about the ice cubes. So far I'm fine. I've been careful. But I'm thinking about the single grape (unpeeled fruit!) I absent-mindedly ate off my plate last night. I couldn't concentrate on the conversation for ten minutes afterwards. 

One of the complaints I used to have as a foreign correspondent was how hard it is to find places in the world that still feel foreign. What's foreign? How about this: 2am, driving back from a state-of-the-art call center in the middle of Bombay, my driver is slaloming through rubble in a scene that would look like Fallujah but for the Brahman cows grazing in the fast lane. On the shoulder a half-naked five year old girl is squatting to pee on a huge slab of broken concrete, lit by a fire of burning garbage. The billboard behind her advertises the latest BlackBerry. India!

My guide to Bombay was Maximum City, Suketu Mehta's beautifully-written collection of stories about Bombay's dark soul, from crime bosses to bar girls. What I liked best about it is how Mehta uses these stories to explain how and why this place got so simultaneously broken and vibrant. One example: the Rent Act, which was put in place as a temporary measure to freeze rents and avoid profiteering during World War II. It has subsequently proved impossible to repeal because, naturally enough, tenants vastly outnumber landlords. Because many rents don't come close to reflecting the actual value or cost of a property, owners don't bother investing in maintaining the buildings. And so you get the defining characteristic of Bombay living: beautiful interiors in decrepit exteriors.

In just two days, I visited MTV India, a Wipro call center, the Tata Institute of Fundamental Research to see the makings of India's first astronomy satellite, three colleges, a labor activist meeting to organize call-center workers, and a series of street retail stores (to test this theory). I attended a lecture on jazz in historic Bollywood and witnessed a proper Indian wedding, which is a dancefest that would put the marching bands of Drumline to shame. I interviewed venture capitalists, consultants, industry analysts and call center workers.

I also hung out with a zillion great people, starting with my friends at Monitor, and extending to fellow editors and even bloggers. And I was taken under the wing of the incomparable Manjeet Kripalani, whose title of Business Week's India bureau chief hardly hints at her superconnected place in Bombay society. Thanks Manjeet!

Tomorrow: Bangalore.

What is it with the Japanese and robots?

Two things have pushed me into an off-topic post:

• The phrase "capable of non-autonomously operating a backhoe" in this slashdot story about dancing Japanese robots.
• This beer-pouring robot I had the good fortune to encounter in the United lounge at Narita. (Warning: 45mb 1MB file with no nudity. Also not a real robot; just a fancy soda machine, although the two pouring angles and the separate "head" nozzle are a nice touch.)  UPDATE: apologies for my jetlag-induced idiocy. Windows Movie Maker compressed it to 2% its original size. Doh!  Download Robotender.wmv

It is, of course, not news that the Japanese have a thing for robots. It is, however, getting worse. Way worse. Like robot overlord worse. Because this is a blog, I can advance three random reasons for why this might be:

1. History. Japan is an industrial nation with a high-cost labor force. It also has a tradition of being insular, if not xenophobic. How to grow the nation's industrial capacity without importing foreign workers? Robots.
2. Demographics. Japan is the oldest country in the world. Old people need assistance, and the erosion of the traditional family structure means that there aren't necessarily younger people around to help. What's the solution? Robots.
3. Astroboy.

PS: Am I the only one to think it's slightly ironic that while the bartender has been replaced by a machine, women are still employed to bow in the elevator lobby?

Away from my desk

This is my travel intenerary for the next two weeks:

SF->Tokyo->Singapore->Mumbai->Bangalore->Mumbai->Frankfurt->
Helsinki->Zurich->Davos->Zurich->Manchester->Chicago->LA->SF

Certain portions of this trip will involve four children, ages two to seven.

Posting will be light.

Long Tail TV: Conclusion

So far I've been pretty fully in armchair economist mode here (is there any better reason for a blog?), but it's time to go beyond the charts and trend stats and actually ask: What kind of TV are we talking about, anyway? Are there, for instance, any examples?

There are. But first, let's crisp up what I mean by Long Tail TV. The definition of the Long Tail in this context is: "content that is not available through traditional distribution channels but could nevertheless find an audience." For the most part, that's niche content. It may not have been niche when it was made or niche everywhere but it counts as niche now where you live. This could include:

1) TV shows that are made but not broadcast in your area:

• Channels your cable provider doesn't carry
• Foreign TV
• Local sports and events from places you aren't

2) Old TV shows:

• TV from the archives, from ancient to relatively recent
• Current shows that you missed and forgot to record

3) Video of any sort that is  made but not broadcast (the video found on the Internet Archive's moving image collections, which ranges from the Prelinger Archives to SIGGRAPH animations, is a great example.)

• Independent films
• Commercials (which are broadcast but not scheduled and findable)
• Amateur video, including news
• Commercial/corporate video intended for targeted audiences

4) Video that could and would be made if only there were a good way to find an audience for it. (Steve Rosenbaum is blogging on this, too). The best sense of what that might be can be found by looking at the online video that's been made since the broadband web became a reality.

• Political video mashups from MoveOn. Skateboarders taping and distributing their stunts and spills. Any number of witness videos. Amateur porn.  Videogame machinima. Etc...
• The sort of thing this article about JibJab Media (home of  South Park-like fare such as "This Land") celebrates. Such web video, the article says, is "spawning a cottage industry of digital movie Fellinis hoping to make their mark in the nascent world of online short films."
• Endless numbers of reality shows.

How will we find it? Well, Yahoo's very impressive video search, including Media RSS,  is a start. (I used it to find that Halo 2 Physics Hacks video that I was looking for.) Combined with a good recommendation engine such as ChoiceStream, things could get interesting.

Finally, I'll end by once again quoting Thomas Hawk:

If today I watch CSI Miami, but on my weekends go out hang gliding and am a huge hangliding fan, when the California hang gliding championships end up being broadcast through a microcontent platform I will end up watching that instead of CSI.

If today I watch some network television but even more than my network television I love reading author Hunter S. Thompson, and my microcontent platform brings me a talk by Hunter S. Thompson from the University of Wyoming I will end up watching that instead of CSI.

If I am 16 and my favorite band is not what hits the charts but rather the latest skate punk music thing, then the custom skate punk music shows that can easily be created and delivered to my microcontent platform will be much more interesting to me than American Idol.

Sounds good to me.

Long Tail TV: Wishlist

Here's an idea, in part inspired by Mark Cuban's gift of three great business ideas for the future of television. It's a way, which I'd love to see someone implement, to help blogs and other word of mouth drive TV taste (with the presumed effect of pushing some demand down the Tail). It might also be a great value-add feature to help smart DVR makers stand out from the crowd.

The problem with DVRs is that when I'm sitting in front of the TV I don't want to program it, I want to watch it. Likewise, when I hear great TV recommendations that I'd like to record, I'm typically nowhere near a TV--I'm talking to colleagues, out and about, or just reading blogs.  I'd like to be able to act on those recommendations while they're fresh, rather than making a mental note (which I'll forget) to program the DVR when I get home.

Now, I realize that that TiVo has online programming and I can do the same on my Media Center using Orb (Microsoft is beta testing a more full-featured remote programming service using MSN). But it's still not easy enough. All of them require that I log into some site, hunt or search through a schedule and then pick some options (even worse,  if you're using TiVo you need to do this at least an hour before the show). Don't even think about doing it on a cellphone from a bar, which is actually perhaps the most likely place that people would be talking about the TV they like.

Instead, what I'd like is is a dedicated browser plug-in that would make this much easier for recommendations found online. When I see mention of a TV show that I want to record, I'd like to highlight it and right-click (Mac users insert your own favorite shortcut here).  A new option on the pop-up menu that appears would be "Record to DVR". If I select that, the app would do a quick search on the phrase, returning with enough information to let me choose the particulars of what I want. 

(George Hotelling notes that TiVo does have a way to write a permalink for a show that will take you directly to that show on TiVo Central Online. That's great, but it requires the author of the blog post or news article or review to specifially code the link with TiVo remote programming in mind. What I'm proposing requires no forethought on the part of the author, or even the recognition that some reader might be using the mention of a show this way.)

If you'll forgive the five-minute Photoshop hack, the PC version might look something like this (apols to Evan for using his blog in vain):

---------------

Long Tail TV: Part III

I promised to offer some thoughts on what the shift from Broadcast to Narrowcast to Nanocast will mean for the future of prime time. And there is a simple answer: it's sure not good for the upfront. But the more detailed answer is just as interesting. I'll start with a little history.

There are quite a few precedents for Long Tail effects, ranging from the introduction of the printing press to the arrival of the Sears Roebuck catalog on the prairie. Practically every time there's been an expansion of access to a wide variety of goods, we've seen shifts in the demand curve toward niches. The fact that everyone buys/watches/listens to or otherwise consumes the same few things doesn't necessarily mean that that's what they all want. It may be just that that's what's available.  Plenty of "hits" are artifacts of scarcity, not the triumph of common taste.

Television has already gone through one such shift. The introduction of cable TV in the late 1970s began as a rapid expansion of the number of channels available in the average American home, a trend that continues today with the switch from analog to high-capacity digital cable. In 1980, the average home could watch any one of 16 channels (probably using an antenna on the roof); nearly 80% of them were watching one of the three main networks in prime time. Today, nearly 95% of those homes have cable (AKA "multichannel" in the chart below), with an average of 110 channels each. And now only half of them watch the networks in prime-time, despite the addition of Fox as a fourth network:

But there’s another way to look at that diving market share line. A drop from 75% to 50% over the period shown means that 25% of viewers had shifted to watching non-Network TV instead. In other words, a quarter of the audience had migrated down the Tail, adding to the quarter that were already there.

Now to be fair, you might not consider MTV or HBO part of the Tail. We’re still talking about channels that are watched by millions of people each night. But as the number of channels available in the average home goes from 80 (analog cable) to 200 (digital cable) to infinity (Internet TV), it’s easy to see the shift towards less concentrated audiences continuing and more people migrating to truly niche content.

“Prime Time” is already meaningless to a TiVo user. So is “channel”. Soon “available in my area” will be, too. As the instruments of scarcity dissolve, audiences are flowing more naturally across the spectrum of available content, usually at a cost to the hits.

To flesh this out,  I spent some time with Howard Look, who runs TiVo's applications and  interface team. He was kind enough to run some numbers on TiVo users' behavior based on what they've got programmed as season passes. The first chart shows the percentage of users who have any of the top 100 shows programmed to record. The second extends that to the top 7,000 shows, averaged over windows of 100 shows (click for larger versions).

What these charts show is that the top 24 season passes make up 25% of  total season passes, and the top 80 season passes make up 50% of the total season passes. But what that means is that the bottom 6820 season passes make up the other 50%. In other words, the Tail is about the same size as the Head. Which is exactly the same ratio that emerged after the introduction of cable, but now on a massive scale.

(A definitional aside: Why draw the line between Tail and Head at the top 80 shows? In truth, because that's where the 50% marker happened to fall. But there's some logic to it, too. Nielsen only reports audience share for the top 100 shows, which pretty much defines where the industry draws the line between hits and misses. 80 is close enough to 100 to make the point.)

Is it possible that there's a fractal geometry at work here--that when distribution bottlenecks go away human nature tends to divide demand equally between a few hits and a lot of niches, regardless of the scale? That instead of the 80-20 Rule we know so well from our world of constraints, unconstrained demand is actually closer to 50-50? It will take a lot more examples for me to prove that's the case, but it's a very interesting coincidence at least.

That's enough to chew on for today. Tomorrow, I hope to finish up this series with some  thoughts about making and finding the video that will populate tomorrow's Long Tail TV.

Long Tail TV: Part II

Yesterday I wrote about a flood of  new Internet-based technologies and products that aim to break the broadcast monopoly on TV, potentially creating an open distribution platform for any video content. This, like the smashing of distribution bottlenecks everywhere, could shift consumer taste from hits to niches, creating a Long Tail of demand (yes, I've now programmed those words into a macro to save on finger wear).

Before I turn to how that may change the way we watch, I should correct one point. I loosely described this streamed and downloaded TV as "IP TV" because it uses the Internet as its distribution network.  I am now reminded (embarrassingly enough, by an article in my own magazine) that the term usually refers  specifically to video served by existing cable and telephone companies using Internet protocols, rather than content streamed across the wide-open Internet by third-party services.

I'm more interested in the latter case--video over the Web--because it's more likely to lead to an open marketplace in which anyone can play.  I don't, however, think that an an entirely bottoms-up self-service market such as eBay will dominate. Programming is not a commodity and the quality of the shows, which is the most important thing, is not easy to ascertain without trusted advice and recommendations. Instead, what seems to work best for entertainment media is a somewhat more structured environment, such as  Amazon and iTunes, where critics' reviews, customer comments and expert taxonomies can all combine to help people find great stuff fast.

As Thomas Hawk puts it:

"The first major player who builds a team of editors and begins not only publishing the microcontent of the long tail to their platform, but even more importantly builds a comprehensive searchable guide complete with rankings, recommendation technology and other features, will have a huge advantage over the rest of the field."

I realize that this could be SBC or Comcast as easily as TiVo or Netflix, but after seeing the mess the cable and phone companies made of video on demand in its first incarnation, I'm skeptical that they'll move quickly enough and get a critical mass of content. In general it's better to let infrastructure companies provide infrastructure and content companies provide content. But either way,  what's most important is that the process of getting video onto these platforms be cheap, simple and transparent.

The best way to do that is to start with a good standard for packaging video to be distributed by such services that can incorporate rich metadata (from keywords to closed captions to full scripts) and, when wanted, DRM tied to some payment system. Needless to say, there are plenty of contenders for this, from the proprietary Windows Media to MPEG4, and no shortage of companies working on ways to extend them.  On top of that, there are companies thinking about ways to clear legal rights to video content on an industrial scale so they can package entire libraries for distribution, the way The Orchard does in music.

Coming back from CES, I felt that I had seen the beginnings of a Cambrian explosion in new forms of television, with the momentum of the DVR+broadband+home networking combining to reshape TV as much as cable did two decades ago.

Jeff Jarvis, who has been driving this meme, describes what he thinks is about to happen:

" In the future of exploding TV, a few months away, anybody can create video programming and do it inexpensively with new equipment and tools; they can distribute it online and they can "market" it (that is, it can be found) thanks to metadata and search and links. All this levels the playing field ....I feel the need for a Death of Networks summit. Coffee's on me."

I suspect the outcome will not be a single winner but a wide range of tiers of service, from p2p (read: Bittorent) distribution of  Pro-Am content, either free or unlockable for a fee, to commercial services that offer top quality and  loads of recommendation features. Some consumers will be willing to pay a premium to stream content in real time; others will pay less and wait for it to trickle into their DVR. And yet others will create a new Napster generation, not just comfortable seeking out exactly what they want to watch online but expecting it to all be there, probably for free.

As Mark Cuban sees it:

"We are about to enter an era where kids can do a search on google, icerocket.com, yahoo and other search engines  and get all the  video they want of TV broadcasts. Put in a topic. Boom. All the video you could ever want. Put in a name. There it is. Video and transcripts to go with it."

Exciting stuff. Tomorrow I'll write about where this content will come from and what the shift from broadcast to narrowcast to nanocast will mean for the future of Primetime.

Long Tail TV

I spent a few days at CES in Las Vegas last week, mostly to get an impressionistic sense of the hot categories in consumer electronics. Aside from the freakish, seventh-seal scene of snow on the Strip and way, way too many huge plasma flatscreens, what struck me most was the explosion in innovation around freeing TV from its distribution shackles.

As your thumb crawls through your several hundred digital cable channels, TV may appear anything but shackled. Yet it is. What seems like everything imaginable is instead a very thin slice on the video world. The existing channel structure mostly rewards focused programming with enough depth to fill a 24/7 window every day of the year. So the DIY channel and History en Espanol now pass muster, but the Halo 2 Physics Hacks channel does not.  An acceptable loss, you say?  How about last year's great season on Bravo, long ago overwritten by your DVR to save space?

Both the channel-centric reality of TV and its ephemeral nature are artifacts of the distribution bottleneck of cable broadcast.  TV is still in the era of limited shelf space, while the lesson of the Long Tail is that more is always better. The growth of cable capacity over the past decade pales next to the growth in video creation over the same period and the size of the potential microaudiences for anything and everything. TiVo may have helped by at least taking the tyranny of time out of the equation, but we are nowhere near the iTunes  model of being able to download everything ever made, anytime.

There are several reasons for this. First, TV has some gnarly rights issues. Content is locked up in all sorts of ways, starting with broadcast network carriage agreements and finishing with syndication.  Then there are the costs of streaming high-quality video, which amount to about $2 per hour. Finally, and perhaps most importantly, there is the small matter that most TV is free to viewers, subsidized by advertising.  It's not clear how to make downloaded video a good medium for advertising, given that you can't know in advance when it will be seen and TiVo-trained viewers are getting all too good at ad-skipping.

At CES a host of companies were showing and talking about ways to get around these problems, extending the celestial jukebox model to IP TV through subscriptions, pay-per-view or some form of advertising support yet to be developed and made skip-proof:

• TiVo announced that it was shifting its focus from bundling deals with cable and satellite companies to delivering video from the web, including movies on demand from Netflix.
• Microsoft discussed its own progress in developing its Media Center 2005 system into an IP TV platform that any content provider could use to distribute their own wares.
• Akimbo, another TV-over-the Internet company, continued to build its library of streamable video and discussed its strategy to be available on Media Center PCs in addition to its current dedicated DVRs.  Dave Networks showed off a similar service, which instead of a DVR uses software running on a PC + an "extender" (slim streaming video receiver) combo like the Media Center. I also spoke to executives at Vidmark, a new company still in stealth mode, which has similar plans (details when they decloak).
• Meanwhile  iFilm and AtomFilms are offering more and more streaming video and short films, including iFilm's brilliant Viral Video service.  
• Yahoo! and Google are launching video search services on the route to delivering IP TV of their own.
• Then there's Blog Television

I'll be blogging more tomorrow about what Long Tail TV might look like once these companies get traction. But while you're waiting, here's a teaser in the form of Jeff Jarvis explaining "How to Explode TV News in Four Easy Steps".

Definitions: Final Round!

The call for definitions produced a tremendous showing from a lot of  very smart readers, both in comments and emails. At this rate I'll have you all writing the entire book for me in true Stone Soup/Open Source fashion.  Thanks in advance!

Quite of few of you were able (easily, by the looks of it) to improve upon my own efforts at defining The Long Tail.  For what it's worth, my definition D--"The Long Tail is about the economics of abundance—what happens when the bottlenecks that stand between supply and demand in our culture start to disappear and everything becomes available to everyone"--got the most votes, although I wouldn't call it a mandate by any means.

I'm not quite ready to pick a winner yet (or more likely remix good bits into an all-star mashup) but I can at least offer a finalist list. Ideally, the three key elements of the Long Tail that a good definition should explain are: 1) the shift from hits to niches; 2) the economics of abundance (infinite shelf-space effect); 3) the aggregation of many small markets to make a big one. 

These are the ones the are closest so far (several were articulated by more than one person; in those cases I've usually picked the shortest one; I've also tweaked a word or two):

Best Definitions:

• "The Long Tail is the realization that the sum of many small markets is worth as much, if not more, than a few large markets." --Jason Foster
• “The Long Tail is what you get when the obscure becomes ubiquitous.”-- Eric Akawie
• “The Long Tail is the 80% of stuff that didn't used to be worth selling.”--Greg
• "The Long Tail is the story of how products that were once considered fringe, underground or independent now collectively make up a market that rivals the bestsellers and blockbusters." --Bob Baker

Best Slogans:

• “Trickle-up economics!”—Joshua Wood
• “The end of the 80/20 Rule!”—Eric Etheridge
• “Everything, all the time!”—Jim Treacher (channeling Don Henley)
•  “A small number times a very big number equals a big number!”—Me (nominated by several readers)

Best Puns:

•  “Life in the vast lane.” –Jim Treacher
•  "An embarrassment of niches."--Kevin Kelleher