A Case of ROI and the Crowd

netflixCrowdsourcing the public can bring to light ideas that would never have surfaced through internal exploration. The more open your innovation campaign, the greater the quantity and diversity of the resulting data. But what to do with all of this data? You can look at the most popular ideas for implementation, but will they be successful? Do the ideas most popular with the crowd translate into innovations with the highest ROI?

In 2006, Netflix issued a public challenge: improve the predictive rating system that recommends movies for Netflix users. Specifically, improve upon the accuracy of Cinematch, the system used by Netflix at that time, by at least 10%. Offering $1,000,000 as a reward, the only rule Netflix established was that those who submitted must explain and make publicly available how they solved the puzzle.

After nearly three years, the 10% barrier was finally surpassed. Although Netflix issued the grand prize check at a publicized event, the algorithms that won the award were never implemented. According to Netflix, the winning system was tested, and the output simply did not warrant the time it would take to implement – the return on investment was not strong enough.

By the time the prize had been claimed, Netflix’s instant watch option had grown from a side feature to the main event. Streaming videos through devices such as a roku or xbox had changed the format through which viewers made their selection. In a blog response to the win, Netflix explains that 75% of viewer choices are now made based on recommendations from the system. Even more interesting, Netflix found that viewers made significantly different choices when using instant watch for videos they would be viewing immediately, than for rentals they would receive and watch in the future. The prize winning algorithms would only have improved accuracy for physical video rentals, which had become the minority.

Opening the innovation and ideation process to the crowd is a great way to test the public pulse. But ROI is a complex metric, and requires a deeper evaluation of the crowdsourced data. On Tuesday June 10th, IdeaScale’s CEO, Rob Hoehn, will introduce ReviewScale – our new decision matrix software that ensures ROI. Sign up for the webinar now, and discover how ReviewScale can guide you through the implementation stage to success.

Leave a Reply