Netflix is a company that (for a monthly fee) allows you to check out and return movies through the mail. A large number of their customers request movies based on Netflix’s “Cinematch” suggestions, which recommends other movies the customer might like based on the movies he or she has previously ordered. In October 2006, Netflix offered a $1,000,000 prize to anybody who could improve the company’s recommendation algorithm by 10% or more. It proved to be an very difficult and interesting contest, with various mathematical (and other) twists and turns along the way, some of which are sketched here. The prize was recently claimed by a seven-person team of statisticians, machine-learning experts and computer engineers from the United States, Austria, Canada and Israel.
Amazingly, the winning team’s algorithm was mathematically identical to another team’s algorithm, which was submitted only 20 minutes after the winning team’s entry. Still, the second place team is not crying about it:
Yet the scientists and engineers on the second-place team, and the employers who gave many of them the time and freedom to compete in the contest, were hardly despairing. Arnab Gupta, chief executive of Opera Solutions, a consulting company that specializes in data analytics, based in New York, took a small group of his leading researchers off other work for two years. “We’ve already had a $10 million payoff internally from what we’ve learned,” Mr. Gupta said….”So for us, the $1 million prize was secondary, almost trivial.”
The article from which that second quote is taken notes that this type of corporate-led open competition could become a new model for improving business, and mentions a number of other such competitions that have cropped up.