Learning from Mixed Signals in Online Innovation Communities

We study how contributors to innovation contests improve their performance through direct experience and by observing others as they synthesize learnable signals from different sources. Our research draws on a 10-year panel of more than 55,000 individuals participating in a firm-hosted online innovation community sponsoring creative t-shirt design contests. Our data set contains almost 180,000 submissions that reflect signals of direct performance evaluation from both the community and the firm. Our data set also contains almost 150 million ratings that reflect signals for learning from observing the completed work of others. We have three key findings. First, we find a period of initial investment with decreased performance. This is because individuals struggle to synthesize learnable signals from early performance evaluation. This finding is contrary to other studies that report faster learning from early direct experience when improvements are easiest to achieve. Second, we find that individuals consistently improve their performance from observing others’ good examples. However, whether they improve from observing others’ bad examples depends on their ability to correctly recognize that work as being of low quality. Third, we find that individuals can successfully integrate signals about what is valued by the firm hosting the community, not just about what is valued by the community. We thus provide important insights into the mechanisms of how individuals learn in crowdsourced innovation and provide important qualifications for the often-heralded theme of “learning from failures.”