When you think about it, today's vibrant Long Tail actually arose in two steps. The first was the abundance boom of the 1990s that arrived with an explosion of variety on supermarket shelves (thanks to revolutions in supply chain efficiency and globalization) and the miraculous appearance of the wide-open Internet. Both were impressive in their own way, but neither were enough. In fact, many people found them oppressive.
In the physical world, the experience of trying to make a wise purchase among 150 kinds of toothpaste or three dozen kinds of iced tea introduced the concept of the "tyranny of choice." Studies showed that people actually felt worse about their purchases when they had more to choose from (did I make the right decision?), and often even bought less because of it.
Online, in those pre-Google days, the flat hierarchy of hypertext and the trueism that 90% of everything is crap led to much hand-wringing over "information overload" and learned distinctions between mere information and true understanding. After wandering lost for days in the unfamiliar wild of sophisticated amateurs and clueless companies, the early online explorers could only agree that there sure was a lot there. Whether it was good remained to be seen.
What was missing in both these cases was the necessary second step: not just the presence of lots of things and information, but the arrival of information about them--"data" about products, and "metadata" about information itself.
For the physical goods, this can take the form of something as simple as Amazon's "rank by bestselling" lists to more complex background information such as reviews, price comparisons, version histories and manufacturing details. Even a little such information can make a huge difference in helping a consumer make a decision.
As Amazon's Jeff Bezos explains it, for a product that a potential purchaser has a great deal of interest in, no amount of information is too much: from reader and trade reviews to service records, the more they can learn about a product the more comfortable they are buying it. But for products that they just don't care much about, even something as simple as knowing what most other people bought can make the difference between being frozen by overwhelming choice and purchasing with confidence.
On the supermarket shelf today the only information you have about all those toothpastes is whichever brands have been lodged in your brain, price, the promotional text on the boxes, and whatever display placement the manufacturer has paid for. None of that is very helpful, and it certainly isn't independent. But imagine if you could add just one simple bit of information: what sold most in the last hour? Suddenly you are able to tap the wisdom of the crowd. This is what information about products allows, and why the online retailers, which can offer that most easily, have been more successful in enticing consumers to venture down the tail towards less familiar products.
By the same token, Google offers information about information, namely how many other people thought a given page was important enough to link to it. As a result, its ranked search results are like the Amazon category bestseller lists: when in doubt, start at the top. Collaborative filtering does the same for music and film, drawing connections between the uneasily categorized and revealing popularity. And several companies are working on ways to use cameraphones as barcode readers, bridging the divide between the bare goods in the physical world and the rich information online.
In all these instances, data and metadata can structure a flat hierarchy and bring order out of chaos. The variety boom may have created the Long Tail, but it takes information about that variety to entice people down it.