Why better DVD recommendations are worth a million bucks

Today Netflix announced a million-dollar contest for the team who can come up with the best improvement to its DVD recommendations. Aside for the obvious points that better recommendations are, well, better and the best thing of all is to get volunteers to do the work, why is this such a priority for Netflix, which is known for having pretty good recommendations already?

The simple answer is that search, recommendations and other filters tend to drive demand down the tail, from the hits down to the niches where minority tastes are often better satisfied. Aside from happier customers, this also has some clear economics benefits for Netflix. It so happens that older titles, well down the Long Tail of time, are both cheaper to acquire and tend to get higher ratings than new titles (mostly because they've passed the test of time and have moved beyond the fog of hype that accompanies new releases). Not only that, but Netflix can also buy fewer of them, since as you go further down the curve the demand is spread out over more titles and there's less of a need to buy stacks of expensive new blockbusters to satisfy the rush of rentals requests around the release date.  

To explain why this is such a good business move for Netflix, let's start with the economic picture on the DVD acquisition side. The following chart was primarily meant to show how margins can get better in the tail, but the thing to notice in this discussion is that acquisition costs fall dramatically with age.

Furthermore, DVD renters are much happier with the DVDs they get from recommendations (which tend to be to older DVDs) as opposed to new releases, as shown by the following slide from a presentation that Netflix's Jim Bennett gave at the Recommenders06 conference in Bilbao last month. Note that once again, the most expensive DVDs for Netflix are the new releases ("NR"); the least expensive are the ones that were rented as a result of a recommendation (the left two)

(Each bar refers to a way that a customer found the video that they rented. Again, "NR" is New Releases; "Interest" refers to the recommendations you get when you add a DVD to your queue and you get a "Other films you might enjoy..." suggestion. "Recs" are recommendations based on rental history and ratings.)

Evidence of the power of Netflix's recommendations to drive demand down the tail can be seen in the following data comparison, which shows Netflix rentals versus the industry as a whole (mostly bricks-and-mortar rental stores such as Blockbuster, hence the labeling). I've plotted the demand curves on a log-log scale so the powerlaws look like relatively straight lines and I've indexed them to the same scale for comparison purposes. The difference in the slopes of the lines shows that Netflix is considerably more niche-centric than the traditional industry retailers such as Blockbuster.

Recall that there are two factors that create functioning Long Tail markets: 1) massive increase in product variety, and  2) massive improvement in findability.  In the above chart, for the first 20,000 titles or so the inventory is the same between Netflix and the aggregate bricks-and-mortar retailers. But because Netflix has search and recommendations, two things that are hard to do in a traditional store, it wins the findability contest hands down. That's why its stats show that niche titles (rank 400 and below) are rented far more often: people are able to easily find them even after they've left the new release promotion shelves in physical stores.

So let's put this all together and bring it back to Netflix's recommendations contest. As shown above, the more Netflix can get customers to rent DVDs from recommendations, rather than new releases, the more money it makes and the more satisfied the customer. The prize will go to the team that can improve those recommendations by 10% (measured by correspondence to customer ratings). I don't know what the exact effect of a 10% improvement in accuracy would have on the demand curve, but I can illustrate the basic effect with the following conceptual chart.

Is that worth $1 million? Netflix clearly thinks so. If there were any doubt about the importance of filters to a functioning Long Tail, this should put it to rest.

Comments

[Hashim]

When I saw this competition, I thought to myself - when will this be covered on the LT blog? :)

In general, are the economics better for most products down the Long Tail, or is Netflix unique?

[Bob]

Netflix has another advantage in squeezing customers 'down the tail': it is a particularly bad way to rent new releases. This effect should be accounted for somehow. I am guessing that the number of subscribers who rent new releases from 'guaranteed in stock' B&M places is not insignificant but I could be wrong.

Thus, while its stats do indicate the importance of niches the causes cited for this - namely variety and findability - may not tell the whole story. Perhaps the supply of new releases vs. demand should be included.

Netflix's queue functionality works to drive this, too, by automatically sending a different movie when #1 (etc) is not available. Since new releases seem more likely to be 'out of stock' it would be possible for one customer to receive many non-new releases before receiving one new release and thus squeezing more and more area into the tail.

Ultimately, if people are satisfied than it doesn't matter. Everybody wins and the emphasis on improving the recs is right on.

[David]

I'll bet the folks at Netflix feel that the million $ prize itself (and the publicity it generates) will generate enough new business that it doesn't matter if anyone improves their recommendations enough to win it. In fact, if nobody wins they'll have gotten an enormous boost in public attention without spending a penny. Pretty smart.

[Matt C]

"by 10% (measured by correspondence to customer ratings)"

Could someone explain this a little better?

[Ian]

Why is Netflix a bad way to rent new releases? They'll ship out a new release the day before it releases so it arrives at your door the day it hits the open market, and then I can keep it as long as I want. Seems pretty efficent to me.

Netflix does a good job of keeping valuable customers happy. I think people that see "out of stock" on new releases a lot are the ones that aren't actually profitable to Netflix due to the volume that they rent. In a very real sense, Netflix has no interest in making these customers happy. I rent 4-6 movies a month on the 4-out at a time plan and haven't seen "out of stock" in years, so I assume they want to keep me.

[Jakob Nielsen]

Matt: "10% improvement" presumably means that if users' average rating of recommended films was 4.1 with the old system, then it should be at least 4.5 with the new system.

Actually, there are many ways of calculating differences between average ratings, and it will be almost impossible to achieve a 4.5 rating if the scale goes from 1-5, because of the ceiling effect of the top rating.

A more fair way would be to say that out of the users who didn't give the top rating in the past, at least 10% should give at least one point more with the new system. In other words, throw out those users who already gave the top rating, because they can't give a higher rating, even if they are more happy.

Another point:

It makes sense that the rating for recommended DVDs is higher than the rating for "interest" DVDs. In the latter case, the system only knows that one data point about the user's interests, whereas in the former, it knows all his/her previous rentals as well as the satisfaction score they provided if they rated these previous rentals.

However, it's a bit of a puzzle that search should score lower than recommendations, because search is something the user is actively seeking out. One option is that the search engine is really bad and doesn't find the films that the user really wants. (I don't know if they sort the search results by recommendation score, but that would be an obvious thing to try if it's not done.)

Another option, of course, is that people simply don't know what they want. I will be willing to believe that, but I still think that the quality of the search engine should be the first suspect, given how poorly search performs in user testing of most other e-commerce sites.

[Bob Carpenter]

The contest rules clearly state the 10% criterion in terms of root mean square error (RMSE). According to their FAQ, guessing the global average scores 1.05 on this metric and their current system, Cinematch, based on linear regression, scores 0.95. I'm surprised the baseline's so good.

[hiodjio]

白癜风治疗白癜风白癜风治疗白癜风白癜风白癜风治疗白癜风治疗治疗白癜风白癜风治疗白癜风白癜风白癜风治疗白癜风治疗白癜风白癜风白癜风白癜风白癜风

斑秃治疗斑秃治疗斑秃斑秃斑秃治疗斑秃斑秃斑秃治疗斑秃斑秃治疗神经性斑秃斑秃斑秃的原因斑秃斑秃脱发

带状疱疹生殖性疱疹生殖性疱疹疱疹疱疹带状疱疹治疗疱疹治疗疱疹疱疹治疗疱疹疱疹疱疹性咽颊炎生殖性疱疹疱疹治疗疱疹病毒性疱疹

狐臭狐臭治疗治疗狐臭狐臭狐臭治疗狐臭狐臭治疗狐臭狐臭狐臭治疗狐臭狐臭狐臭治疗腋臭治疗狐臭

牛皮癣治疗牛皮癣牛皮癣治疗牛皮癣牛皮癣治疗牛皮癣牛皮癣牛皮癣治疗牛皮癣牛皮癣牛皮癣治疗牛皮癣治疗牛皮癣牛皮癣治疗牛皮癣牛皮癣

[Rick Presley]

I'm wondering if NetFlix will see Amazon do an end run with their Unboxed service. Now all you have to do is download videos. No messing with mailers and actual DVDs. Amazon will allow you to download to two PCs and two portable devices for $1.99 per episode or you can order a whole season.

Right now their selection isn't that great, but downloadable and portable video (that can play on your DVR through your home theater system) may sound the deathknell to not only Blockbuster, but NetFlix as well.

My guess is that NetFlix already has a plan to leapfrog Amazon. Or at least I hope so. It's also interesting to see my DVD player doesn't have a chance to gather dust before being replaced by a DVR device which may be replaced by a wireless network card from my PC straight into the TV. For that matter, the DVR, PC and TV may all morph into a single system all controlled from my cell phone.

[Jean-Baptiste Rudelle]

At the end of the day, 10% increase in their predictions is worth much more than $ 1 million.
But then, it depends on the bargaining power of each party.

[Felix McGlade]

Your premise that better recommendations are worth $1M to Netflix is wrong. It necessarily has to be worth only $50K per year to make sense to Netflix. See, Netflix is playing a trick on the contestants. The bar for $1M is so high that it's highly improbably that anyone will achieve it given the data. However, Netflix will get the best results that thousands of people around the world can achieve, fueled by dreams of $1M, for just $50K a year. Even a 9% improvement still only gets 50K. This is a brilliant twist on crowd-sourcing.

[Greg Banville]

Ian said:

I think people that see "out of stock" on new releases a lot are the ones that aren't actually profitable to Netflix due to the volume that they rent. In a very real sense, Netflix has no interest in making these customers happy.

This is what puts potential customers like me off Netflix. I'm accustomed to being treated as if I were a valuable customer wherever I shop and I really don't like the idea that using a service too much is going to get me treated like second class. Such as by being put on the slow list or whatever. It isn't the customers who decided to call it an "unlimited" number of dvds per month. The customers aren't lining up to trade in their ration tickets for food either, they are paying, and deserve to know up front exactly what they can get for what they pay instead of being underserved by Netflix at the company's discretion, as they reserve the right to do in their terms of service. At least today the customers who read the TOS know what they are getting into, but it took a lawsuit to even get Netflix to own up to their practices even in the fine print.

I look forward to competition between Netflix, Amazon and Blockbuster Online, maybe one of the services will distinguish itself by being a but more honest about what their customers are getting for what they are paying.

[Margaret Hedstrom]

Chris, I enjoyed your talk yesterday at the I-School conference in Ann Arbor. I agreed with most of what you had to say, but I would like to pose a question. My sense from your talk was that if entrepreneurs can figure out how to exploit the "after" or "hidden"market, most of the niche content will be taken care of. My question is this: What is the role of public institutions (libraries, archives and museums) in serving the niche market? Haven't these institutions always been supporting niche interests because market solutions did not work?

[Hunter McDaniel]

Another reason Netflix wants to drive demand down the long tail is that this puts them in a better bargaining position with the content providers. Netflix has the least leverage on big new releases, which a significant number of subscribers want RIGHT NOW. For older or niche releases Netflix can afford the risk of not reaching agreement with owner and thus is in a better position to dictate terms.

Greg, I would suggest to you that the TOS conditions you refer to have no practical impact on the vast majority of subscribers, and that those few who are affected still get a very high level of service. Any business that offers 'unlimited' service has strategies to keep the very heaviest users from breaking the model; it just happens to be easier to see with Netflix.

[Eddie Schneider]

I was shocked, shocked to read about video store closings in Manhattan today...

[san]

I look forward to competition between Netflix, Amazon and Blockbuster Online, maybe one of the services will distinguish itself by being a but more honest about what their customers are getting for what they are paying条码条码机条码打印机 仓库管理条码枪