Microstructure in the Long Tail

In this post I'm going to attempt the death-defying stunt of connecting social network theory, wealth distribution, the behavior of atoms in a gas and gangsta rap.  Coherence would be a plus.

In my Etech speech I listed several interesting economic questions I was hoping to answer about the Long Tail in the book. One of the was whether the LT had a "fractal dimension", by which I meant whether the shape of the popularity curve in the many niches is the same, at a smaller scale, as the overall shape of the total demand curve. Functions that have this characteristic are said to exhibit self-similarity at multiple scales. I speculated that this was indeed the case in many industries, and that the Long Tail is in fact made up of many "minitails" (below), all adding up to the powerlaw ("Pareto") shape we know so well.

This is important because it explains why a very effective network-effect (viral word-of-mouth) recommendation system, which is essential in driving demand down the tail,  does not actually do the opposite: drive content up the tail to further amplify hit/niche inequality. The explanation, I argue, lies in the only semi-permeable membrane between niches and mass-markets. Popularity exists at multiple scales, and ruling a clique doesn't necessarily make you the homecoming queen.

It's possible, for example, to be the most popular drum-and-bass artist at the very head of the drum-and-bass popularity curve, but that doesn't mean that you're about to knock 50 Cent off his top-ten perch. Music is made up of thousands of niche micromarkets, miniature ecosystems that, when smooshed together into an overall ranking, look like one Long Tail. But look closer and each has its own head and tail.

My question was whether these minitails resemble the Long Tail under a microscope, or whether they have a different shape. (Clearly this depends on the size of the niche, but I was focusing on relatively substantial ones where the n is large enough to be statistically significant). 

This is a reasonable question. Ducan Watts and Albert-Laszlo Barabasi's work on "small worlds" suggests that the microstructure of small social networks is distinctly different from large ones, going from a near-broadcast relationship to narrowcast to multilateral as the group size shrinks. That would tend to lead to very different popularity curves, assuming that social network theory does indeed apply to the shape of the Long Tail (further research for my to-do list).

I'm still gathering data to answer this empirically, but in the course of my digging I came across (thanks to Julian Bond) a fascinating New Scientist article about new "econophysics" thinking on the best-known of the Pareto distributions--wealth. It explains why we appear to have two economic classes, one in which the rich grow richer and the other in which the poor stay poor:

In 1897, a Paris-born engineer named Vilfredo Pareto showed that the distribution of wealth in Europe followed a simple power-law pattern, which essentially meant that the extremely rich hogged most of a nation's wealth (New Scientist, 19 August 2000, p 22). Economists later realized that this law applied to just the very rich, and not necessarily to how wealth was distributed among the rest.                                            

Now it seems that while the rich have Pareto's law to thank, the vast majority of people are governed by a completely different law. Physicist Victor Yakovenko of the University of Maryland in College Park and his colleagues analyzed income data from the US Internal Revenue Service from 1983 to 2001. They found that while the income distribution among the super-wealthy - about 3 per cent of the population - does follow Pareto's law, incomes for the remaining 97 per cent fitted a different curve - one that also describes the spread of energies of atoms in a gas
(see graph at right)

In the gas model, people exchange money in random interactions, much as atoms exchange energy when they collide. While economists' models traditionally regard humans as rational beings who always make intelligent decisions, econophysicists argue that in large systems the behavior of each individual is influenced by so many factors that the net result is random, so it makes sense to treat people like atoms in a gas....The atoms assume an exponential distribution of energy when they are in thermal equilibrium, and pushing the gas away from this state takes a lot of energy. It could prove similarly difficult to shift an economy to a different state.

Basically, it's so hard being poor that there's no time to work on getting rich. Is that true for struggling artists at the butt end of the tail, too? I've assumed that demand can shift down the tail and quality can rise up it, almost without limit. But there may indeed be a threshold at which this egalitarian mobility no longer works.

Comments

[Etienne]

Isn't that effect generated by "limited rationality”? Following your arguments from the very beginning, I’ve always been bothered by this behavior - wondering ever if it only generate a limited side effect or major bias?
From my perspective, one reason why shelves have an end (out of physical capacities) is people have a major factor to deal with: time!
Don’t you believe the pattern “time and resources” might be matching the 3% you mentioned above (time is often “outsourced” for that population)??

Thereby, in common everyday life behaviors, shelves have to be “pushed” to consumers with limited rationality, more than being simply exhaustively offered: by heavy brands, buzz, blog… And the most effective: social pressure! (What IT people seem to be finally handling, under concept such as “social TV”). Don’t you believe your theory might be missing that dimension?

[Etienne]

To illustrate my word, if you where to discover ourmedia, or atomfilms (or any unlimited shelf) today what would you he consume ? why ?

[Rishab Ghosh]

Start talking about fractal effects in long tails, and you risk even worse confusion than the 80/20 - 50/50 one, thanks to shifting definitions of the long tail. The best, consistent way to draw the tail is as a cumulative distribution function, in the form of a Lorenz Curve. That way, the slope of the curve can only ever increase as you go up the curve, and any (x,y) point on the curve always shows that the top x% of [songs, people, books, whatever] represent y% of [sales, wealth, pages, whatever].

As such, "minitails" as you draw them are pretty but meaningless, and would not occur in a lorenz curve. It's not clear what they could possibly mean, either, since if points on the long tail represent individual products you could not possibly have "minitails". If points on the long tail represent product categories, of course you could have "minitails" (pareto distributions) for individual products in individual categories, and like just about any other natural or self-organised phenomena, they will also exhibit pareto distributions. Either way, the points on the "minitails" cannot represent the same items as the points on the "large tail", or they are meaningless.

[Rishab Ghosh]

Start talking about fractal effects in long tails, and you risk even worse confusion than the 80/20 - 50/50 one, thanks to shifting definitions of the long tail. The best, consistent way to draw the tail is as a cumulative distribution function, in the form of a Lorenz Curve. That way, the slope of the curve can only ever increase as you go up the curve, and any (x,y) point on the curve always shows that the top x% of [songs, people, books, whatever] represent y% of [sales, wealth, pages, whatever]. This also has the advantage of allowing you to calculate the gini coefficient, a well known (in economic literature) measurement of the concentration of a distribution.

As such, "minitails" as you draw them are pretty but meaningless, and would not occur in a lorenz curve. It's not clear what they could possibly mean, either, since if points on the long tail represent individual products you could not possibly have "minitails". If points on the long tail represent product categories, of course you could have "minitails" (pareto distributions) for individual products in individual categories, and like just about any other natural or self-organised phenomena, they will also exhibit pareto distributions. Either way, the points on the "minitails" cannot represent the same items as the points on the "large tail", or they are meaningless.

[Francis Hamit]

I can certainly relate to the real world effect of being too poor to get rich.

As a writer, I've had one of the best possible day jobs; freelance journalist. The net result is hundreds of magazine articles in print under my byline from which I was able to make a modest living, but which also do not seem to have much value in the aftermarket so far. They are the ultimate niche products.

The net effect of this career wise is that, until I got some money to allow me the time to work on very big projects taking years to realize, I was on a treadmill of production of these articles. Time is money and vice versa, so those hundreds of articles also represent about a dozen novels and stage plays which didn't get written, much less published and produced.

I now have the first of a series of Civil War novels close to completion and my stage play"Memorial Day" will have a showcase in the Envision program at the Masquers Playhouse in Point Richmond, California May 13th to 15th.

Conversely, the last magazine article I wrote and published was in 2002. For a creative person, time to actually do the work is the one immutable resource.

Outside that three percent you mentioned, most of us are buying and selling time, the one thing we can't get back. I think that the Long Tail works because it saves people time searching for information products. Mass media pushes out information. Niche marketing pulls it in to those who really want it.

There are other ways to get paid than money and many people are content with that. The classic "poor struggling artist" always has to make paying bills their first concern. That places limits on creativity and originallity. That last is always a hard sell going in.

[Tim Keller]

For a definitive answer you should check the current issue of the Proceedings of the National Academy of Sciences. The cover story is Subnets of scale-free networks are not scale-free: Sampling properties of networks (looks like you have to pay for the full piece).

Tim

[bill fischer]

Some quick thoughts on an admittedly broad post:

I'm strongly inclined towards Rishab's notions regarding the use of a cumulative distribution function.

If one must bring fractal geometry or minitails into this it might be useful at a SKU level but less so at a niche level. A single SKU as it moves down the tail is likely to continuously loop back upon itself since any spike in the amplitude of demand will push it back up the tail briefly before its fall. In aggregate one might see a collection of minitails.

Since popularity has a time component it might be useful in discussions of a marketplace to plot a third axis and discuss "the Great Plains" instead of a "long tail."

Recommendation engines lever items from the head of the tail to raise up items in the tail. This serves to push the Gini coefficient of the curve closer to zero. When recommendation engines incorporate momentum (biggest movers, daily popularity, top searches, etc.) they can dramatically move an item up the tail which adds to equality in a marketplace.

In terms of a governor on the ability of a tail item to move up a tail perhaps Hollywood's "Q" rating might be a useful device. Whereby the percentage of individuals exposed to a SKU and the percentage of those who like said SKU will limit its movement up the tail. Marketing dollars can impact box office but not on a pari pasu basis.

[Arthur De Vany]

All the processes that are stable under mixture, choice, and summation converge to the class of infinite variance stable distributions. They have Paretian tails.

The central part of the econophyics distribution is a Bose-Einstein condensate.

Both these topics are covered in my Hollywood Economics: how extreme uncertainty shapes the film industry. I show that long-tailed, stable distributions capture both the dynamics and end of run statistics of the movie business. Another feature of these processes is their self-similarity (things look the same from different perspectives). I use SS to explain the contracting and financing of movies as well as cost, returns, and revenue.

[David Palmer]

Chris, I'm sure there's a certain amount of randomness in the exchange of money, but just because people's behavior is irrational doesn't mean it's random. If it were, the odds of someone buying one product would be the same as the odds of them buying another. Like roulette. The wealth distribution among the bottom 97% might not follow a Pareto curve, but I'll bet their (our) spending habits do.

[Terry Rock]

Another related theoretical problem I've run into when doing research in this area is defining the market. We looked at mutual forbearance as an outcome of multi-unit/multi-market competition, but the results are highly contingent on how one defines the market.

For example: does the Drum & Bass superstar *really* compete with 50 Cent? Maybe from the perspective of a retailer with limited shelf space, but my guess is that their consumer market overlap is negligible. It will be assymetrical for sure: 100% of the Drum & Bass fans could be 50 Cent fans, but maybe .00001% of 50 Cent fans are Drum & Bass fans. So... are these really the same "market?" When are they and when aren't they?

This matters because referral dynamics in any given market must be highly dependent on these overlaps. Perhaps what's under any given long-tail distribution is the cumulation of the area within millions of Venn diagrams of, eg, "50 Cent Fans + Drum & Bass Superstar Fans", "50 Cent Fans + Country Music Superstar Fans." The fat part of the distribution (is it the "short/fat head?") would perhaps best be described as "50 Cent Fans + Ludacris Fans" and "50 Cent Fans + Eminem Fans," or, as we commonly refer to it... "Rap/R&B/HipHop Fans." They're a market because their tastes rather predictably and symmetrically overlap. You get a long tail only with asymmetry.

In this view, then, you could indeed conceive of the distribution as fractal. Simply add one more category to your Venn Diagram. Now, instead of "50 Cent Fans + Drum & Bass Superstar Fans" you've got "50 Cent Fans + Drum & Bass Superstar Fans + Weird Electronica Star Fans"... etc. etc.

The more I read your work, the more I'm convinced that at the core of this is simply the age-old mantra "know thy customer." Only in the past, it wasn't so lucrative to be precise because there wasn't the technology to make it cost effective. Now there is, so: Know thy customer and thy customer's friends!

[Paul]

just a semantic note, but in the article you quote- they mention the exponential curve of the gas molecule distribution. An exponential curve does NOT have a long tail, if your present definition of "long tail" is power law distribution. While they may look similar to the eye, try plotting each on a log-log scale. The "long tail" becomes an unremarkable straight line, and the exponential curve still shows curvature.

[deus ex Machina]

the exponential part of the wealth ditribution is to do with the charasteristic of the different asset classes that are differentially held by wealth deciles. the top few percent hold most of the stocks and are able to reinvest. the middle percentiles mostly hold wealth in their own homes and dont reinvest so they only get a compound return not a power law return.

I make some sweeping observations about power law wealth distributions inb my blog http://vic.org

[Tom N]

Fascinating...

Regarding the income distributions, the movenement between groups and the ages of the groups needs to be compensated for in any study.

(and hopefully that was) Amoung people who's life experience would plot towards the middle of the curve on a 50 year plot, on an annual plot they'd likely have some years at the extremes...

...especially true for entrepneurial types, those that had employee stock options, those in cyclical fields, and just plain old wage earners that may take a windfall on some small investement they made that paid off 20 years later.

Other anomalies could be the descrepancies of cost levels in differing areas of the country where the NYC firefighter might be far off into the "very rich" if his income were put into a rural alabama context (and a handyman with a fully paid for home in alabama spending a good part of his time hunting, fishing taking both recreation and some food outside of the cash economy wouldm't be nearly so poor as his income showed)

The first case, the point in life and income voitilty, would tend to extend the points at either end, while the region differences might tend to fattten the tails (?)

Thats similar to the structure of the curve you're talking about to a degree.

But I'm not sure if you've addressed it, but understanding the tail over different time streams rather than a snapshot could lead to better understaning and perhaps flattening of what seems so extreme.

(then again it might not, but certainly it might be addressed)

[Azeem Azhar]

Chris, Suggest you check out Phillp Ball's 'Critical Mass' which has a hsitory of sorts of 'econophysics' (or did we mean psychohistory?)

[hyokon]

Regarding this issue, I agree with Chris Anderson mostly.

I think the most effective way for a poor startup (including artists) is to create a market, not just a product in an existing market. What does it mean to create a market? Create a product that's different enough so that users cannot associate with an existing category.

[jeux pc]

Microstructure in the Long Tail This “fractal dimension” interests me a lot – I’ve seen Long Tail patterns in the various subcultures I’ve been involved in – could it be a natural phenomena held back by traditional market economics? Or something?