The real meaning of 80/20

I am as guilty as, well, nearly everyone else of sloppily defining the "80/20 Rule" to mean whatever I want. Pareto's principle really is an all-purpose widget, broadly applicable to almost anything humans do (and, of course, the behavior of atoms in a Bose-Einstein condensate). But it's time to crisp things up, at least as far as the Long Tail goes.

The general articulation of the principle is: "for many phenomena 80% of consequences stem from 20% of the causes." (Vilfredo Pareto [shown] first articulated this in 1906 when he noticed that 80% of the property in Italy was owned by 20% of its population).

In my original piece I explained it as follows:

The 80/20 rule is all around us. Only 20 percent of major studio films will be hits. Same for TV shows, games, and mass-market books - 20 percent all.

We're stuck in a hit-driven mindset - we think that if something isn't a hit, it won't make money and so won't return the cost of its production.  We assume therefore that there is little demand for the stuff that isn't carried by Wal-Mart and other major retailers; if people wanted it, surely it would be sold. The rest, the bottom 80 percent, must be subcommercial at best. We assume, in other words, that only hits deserve to exist.

So in this case I was using 80/20 to refer to the notion that 80% of the revenues come from 20% of the products. Furthermore, I was echoing the Hollywood mantra of hit-driven economics: the top 20% of films will make virtually all the profit, subsidizing the loss-making bottom 80%. Like this:

In other words, if you could only know ahead of time which films would be hits, the economically rational decision would be to make just 20% as many of them. And, indeed, this is largely what DVD rental and retail outlets do: since they do know which films were hits in the box office, they mainly stock and push just those. As a result, theatrical hits from the past year make up 90% of the transactions in even video superstores (other titles are stocked, but in smaller numbers and are not promoted).

The stores do this because shelf space is expensive, attention is scarce and there are few good ways to offer consumers information about products to help them with their choices.  Ironically, the real world, despite its ability to flood our five senses, is still a pretty narrowband  information channel when it comes to things like "people like you bought...", "IMDB says..." and "rank by...".

My thesis in the continuing research on the Long Tail is that as distribution and information bottlenecks diminish, many 80/20 markets will become closer to 50/50 markets.  What I mean by that is that the demand curve will flatten somewhat, so that the few "hits" that now generate 80% of the revenues will instead make up about half the market, and the many niches that now represent 20% of the revenues will, in aggregate, make up the other half. The reason is that more goods will be available and they will become easier to find as information about products gets richer and recommendations drive demand down the tail.

That's revenue. The profit implications could be even more dramatic. We're seeing that in many markets (especially those for digital goods that can be delivered online) the retail margin on small sellers can be as high as for the blockbusters. The marginal cost of storing and delivering a track on iTunes is the same tiny amount whether it's top ten or bottom half-million. Because storage space is essentially free, the market can become non-discriminatory:  10 sales each of 1,000 niche tracks is, to first approximation, economically identical to 10,000 sales of one hit track. So rather than all the profit coming from the top 20%, it too can evolve to be split evenly between hits and niches. Thus the Long Tail can, in aggregate, be as profitable as the head.

However, here's where it gets tricky. In the above definitions, I've been referring to 80% and 20% of the existing markets. But the Long Tail is about expanding those markets to include products and customers that are currently "sub-economic"--traditionally not worth carrying or otherwise including at all. How does that fit into the traditional 80/20 construction?

Good question. I'm afraid I may have confused many people on this point, because in the charts in my original piece I imply that the Long Tail refers exclusively to products that are not available in standard retail, which is to say outside the existing 80/20 world entirely. That's one perfectly reasonable definition of the Long Tail (see below), at least for products that have a retail presence, but it doesn't help us in our quest for 80/20 clarity. The fact that in the Netflix example this beyond-retail category happens to represent 20% of rentals only muddies the waters further. No wonder interpretations of this point have been all over the map.

(In all the Long Tail markets that I've looked at so far the revenues from products not available in traditional retail range from 10%-30% of the total. Those figures have all risen since my article last year, but I'll save the latest numbers for the book. Also note that I've revised my Amazon estimates after more consultation with them. )

Making matters worse, virtually none of the media markets I've looked at actually show an 80/20 distribution today. In most instances, they're much more hit-driven: typically a single digit percentage of products represent 80% of the revenues. In other words, the 80/20 Rule not only won't hold in the future, but it doesn't even hold today. Argh!

Needless to say, this is a mess and I'm going to have to be a lot clearer in the book. As I see it there are two possible solutions:

1. Keep my definition of the head category constant even as the tail expands, so I'm at least comparing apples to apples. Thus my thesis would be: As more niche products become available, the hits (whatever percentage of the total number of products they may be) that now make up 80% of the revenues will decline to 50%. And the niches will grow to make up the rest.
   
2. Find another way to draw a line between head and tail and simply observe how the expansion of one affects the other and the market as a whole. Inevitably, that's going to be subjective and will be different in each category. Also, in most cases it won't fit neatly into either the 80/20 or 50/50 rules. As in the Netflix example above, I could use typical bricks-and-mortar inventory to represent head, and the rest tail. In TV, I could use Nielsen-ranked shows (the top 100) to define the head. In music, perhaps the best metric is the Soundscan top 1,000 (the Billboard 100 is probably too few to be a fair representation, given how many albums are released each year). For advertising, companies that spend more than $10,000 a year on ads. And so on...

Number 1 is conceptually neat but somewhat arbitrary in its line-drawing. Number 2, while reflecting real-world dividing lines, doesn't fit neatly into a Pareto context.  On balance, I'm leaning towards Number 1--neat is good--but we'll see how the next batch of data comes in to see if it's doable. Rigor is great, but plenty of people smarter than me have ended their careers still trying to turn their ideas into the grand unified theory of everything. Me, I'll happily settle for almost everything.

Comments

[fling93]

Does it really matter if 80/20 becomes 50/50? Isn't the real key to convince retailers they'll get a higher return on marginal dollars spent on extending the tail rather than creating more or bigger hits? Depending on the cost, that could be true even in an 80/20 world.

Or did I miss the whole point of this post? I did find it pretty confusing.

[Christopher Kemp]

Your division in #1 between niche and hit (or 20 and 80) may be arbitrary, but it's readily visible. Few would disagree that it makes sense, and it works for the generalizations you're making.

[cori schlegel]

I agree with Chrisopher; you don't need to apply Pareto's rule in such a rigid fashion as long as you make it clear that you're using it in a more, um, metaphorical sense.

Most readers won't look to apply it with rigor to their thinking about your ideas and as long as you disclaim your looseness I don't think it's an issue.

[Kaleberg]

There are a lot of curves that can meet the 80:20 rule. According to one source (http://www.hpl.hp.com/research/idl/papers/ranking/ranking.html) the Pareto curve is of the form:

y = x ^ -a

where a is a parameter that describes the curve. Has anyone poked around and snarfed a few A9 usage statistics or Amazon rankings to figure out what this parameter is for various phenomena.

The integral, that is, cumulative sales assuming constant pricing, is clearly just another Pareto curve of a different order and scaling.

Lowering prices or increasing marketing of items on the long tail should just move things up and down the curve and change its scale, not its overall shape. If not, you run into the smallest uninteresting number problem and get things like the best selling poorly selling DVD title not quite a blockbuster list.

In the real world, a seller has to pick a range of stuff to sell. Some go for the meaty high end. Some move farther down the list, and then charge a premium. Very few Chagalls are sold every year, but selling one can be lucrative.

Sellers have limits on the range of items they can sell, that is, the range of the integral of the curve that determines how much money they make. Long tail businesses can cover a VERY broad range with an extremely low cost of goods sold.

You can fit the curve and tweak all sorts of numbers, but the real thing is how this trend towards increasing range works. Computerization enables chains of stores who share software and other infrastructure. This can reduce the variety of goods sold, but if you compare a modern supermarket with its SKUs and PLUs and a corner bodega, the supermarket probably stocks more different goods.

We are moving to both increasing standardization and increasing variety. (Hmm, do I believe that?)

[brian]

Chris, you still haven't solved the producer's paradox. Of a finite amount of resources (time+money+people), how should a producer choose between projects with which to invest and support? Some projects cannot be done on commision alone like between an author and a publisher of an already finished work.

Theatrical movies require substantial upfront investment and Hollywood studios must decide how to allocate their budgets between competing projects. Blockbuster films are often greenlighted because studio executives have confidence that at least a substantial portion of them will recoup their costs in the aggregate. They cannot toss a few bucks at just any Tom, Dick, or Marry screenwriter who may think his script will reach some niche audience.

Also, remember that of the SAG membership, only a select few become household names and enjoy the benefits of celebrity. Hollywood has no supply problem of wannabe stars, yet studios still recruit the same handful of costly actors and actresses to headline their films.

[Francis Hamit]

In my own electronic publishing through Amazon and the other online retailers, I've noticed that a few titles carry most of the volume. Since the topics are all over the map, I suspect that there is a bandwagon effect here. They continue to sell well because they've sold well in previous months and are nearer the beginning of the list.

I'm still waiting for the Google Print program to kick in. (We are "processed" but not "live" yet.) I'm hoping that sampling will make a difference and I suspect that ads revenues might be as important (or even more so) than actual sales.

We continue to add titles, but recently sales of single articles at $1.95 have been far more than bundled product, even though the cost per article is lower. It has become obvious that price alone will not sell a title.

[Jakob Nielsen]

You will never reach a 50/50 distribution in the sense that the top 50% of products account for 50% of the revenue. That would mean a completely flat demand curve where everybody were exactly equal. There are very few markets where that can be expected. In almost every domain, some things are more popular than others.

Note, that when you talk about an x/y distrubition, it's not necessary that x+y=100. For example, there could be a 60/20 distribution, where 60% of your revenues came from the top-selling 20% of products. That's a likely direction for the Pareto principle to move if the Long Tail shows sufficient strength.

Finally, note the difference between manufacturers and resellers. For the manufacturer (e.g., movie studio), it only makes sense to product best-selling products, if only they could identify in advance what those would be. But for the resellers, it makes sense to stock tail-like products since they don't have to carry the investment in producing these non-best-sellers in the first place.

(Also, for the manufacturer, once a dud is produced, the fixed cost is a sunk cost and doesn't matter to the marginal thinking about what to do next. So if the variable cost of keeping a non-bestseller in inventory are less than the contribution margin from sales, then you should keep selling it, even if you regret having invested in it.)

[brian]

I take Jakob Nielsen's point about sunk cost duds, but as he and I point out that still does not mean that producers have an incentive to make and market niche products, which is what the long tail largely rests upon as an economic theory.

[Christopher Kemp]

I don't believe it matters whether or not Universal or any other major movie producer finds incentive to make and market niche products: those products are already being made by other producers. The point of the Long Tail, as described here, is that new models of reselling are allowing for increased access to and profit from those niche products. The volume of movies and music (as two examples) produced relative to the volume heavily marketed is staggering. It is the changing perspective of the reseller that's important.

Second, x+y must equal 100. 60% of your profits might come from 20% of the products, but 40% profit must then come from the other 80% products.

[Francis Hamit]

There are two other constraints at work here. One is time, which is very inelastic. You cannot "save" time, in the sense of storing it up the way you can goods or capital. The appeal of electronic distrbution is that it costs less time to distribute material already created and low demand products can become incrementally profitable.

Which leads us to the second constraint. The theory of incremental income says that if you can make a marginal profit from an economic activity then you should. The question then becomes "what is your time worth?" There are other ways to get paid than money of course, but generally, if you can make a little more than you spend then you should. Except that your time may be better spent in an alternative activity or even at rest. preparing for something that brings a higher return. In theory, the Long Tail will uncover profits that would otherwise be lost. As a practical matter, since you can't recover the time spent to make them available, the opportunity may not be worth the investment required to realize it. Time is money.

[fling93]

The decision-making behind production and distribution are very different, and it's important to keep in mind that the primary implication of the Long Tail is on the distribution side.

The costs of production are also going down, but that's an independent issue.

[Nisarg Kothari]

to Jakob Nielsen:

The 50/50 isn't the first and second halves. It's the head and the tail. The head is maybe the top 10% of the total, the tail is the rest. So instead of the top 10% being 80% of revenue, it will be 50% of revenue.

[MarkN]

I think there needs to be more clarity on what it is that you plot along the X axis. Is it units of Widget X sold? Or total profit for Widget X's sold? Or expected margin per Widget X.

It seems to me that if the choice of X axis is arbitrary then you just keep looking for some other statistic to plot until you find one that agrees with your proposition. The movie example is $/movie. The NetFlix example is rentals/movie.

My vote is for expected margin per widget. This lends itself to a precise microeconomic interpretation in terms of supply and demand.

My sense is that long-tail economics is heavily related to the practise of yield managment where customer segmentation and differential/discriminatory pricing are stock and trade. It would be interesting to characterize the "customer" at various points along different long-tails.

[Chris Anderson]

Jakob,

Nisarg is right. The thesis I'm proposing is the 20% of products that now make up the 80% of revenues will decline to 50% of revenues in the future. In other words (putting aside all the caveats I discuss in the post) that means that the 20% of products in the Head would make up 50% of the revenues and the 80% of products in the Tail would make up the other 50%. So it's still a powerlaw distribution, just a much flatter one.

At Etech today I showed some real world examples of this effect actually showing up already. One was in TV channel viewership after the arrival of cable and in DVD rental through interactive kiosks vs shelves.

[Jakob Nielsen]

Well, the definition you quoted for the 80/20 rule is "for many phenomena 80% of consequences stem from 20% of the causes."

This means that when you talk about an x/y rule, x is a percentage of the consequences (e.g., sales) and y is a percentage of the causes (e.g., number of products). Thus, x and y are percentages of different things and do not need to add up to 100%.

Of course, you can use a new definition if you want, but then you have deliberately abandon the original definition.

Now you are talking about an x/y/z rule, where x% of the consequences stem from z% of the causes and y% of the consequences stem from (100-z)% of the causes. You have to make z explicit, or there will be no way to know what you mean by, say, a "50/50 rule."

For example, you could say, "if the head is defined as the 20% best-selling products, sales used to be split 80/20 between head and tail, and new the split will be closer to 50/50." (Every time you present a ratio between head and tail, you also have to present a definition of the head, unless you have one fixed definition throughout the book.)

[Barry Welford]

I think all the above goes to show that the misnamed Pareto Principle should move over and be replaced by the Anderson Principle or Long Tail Principle as I suggest in a blog posting today at http://blog.cre8asite.net/bwelford/index.php?id=P258

[Quickflix Review]

I just read THe Long Tail, It weas a fantastic read which opened my eyes, Now on to my FREE download

[xmas gifts]

The 80/20 rule applies to cases where products or customers are different. Some will be more popular or heavier users than others, and the "powerlaw" demand curve that results can be generalized as 80/20. But in your case, the move to flat fees made all the customers the same. That's why you ended up with a totally flat demand curve and the 80/80 effect you noticed.