IDEAS home Printed from https://ideas.repec.org/a/inm/orisre/v27y2016i1p27-48.html
   My bibliography  Save this article

Rate or Trade? Identifying Winning Ideas in Open Idea Sourcing

Author

Listed:
  • Ivo Blohm

    (Institute of Information Management, University of St. Gallen, 9000 St. Gallen, Switzerland)

  • Christoph Riedl

    (D’Amore-McKim School of Business and College of Computer and Information Science, Northeastern University, Boston, Massachusetts 02115; and Institute for Quantitative Social Science, Harvard University, Boston, Massachusetts 02138)

  • Johann Füller

    (School of Management, University of Innsbruck, A-6020 Innsbruck, Austria)

  • Jan Marco Leimeister

    (Chair for Information Systems, Kassel University, 34121 Kassel, Germany; and Institute of Information Management, University of St. Gallen, 9000 St. Gallen, Switzerland)

Abstract

Information technology (IT) has created new patterns of digitally-mediated collaboration that allow open sourcing of ideas for new products and services. These novel sociotechnical arrangements afford finely-grained manipulation of how tasks can be represented and have changed the way organizations ideate. In this paper, we investigate differences in behavioral decision-making resulting from IT-based support of open idea evaluation. We report results from a randomized experiment of 120 participants comparing IT-based decision-making support using a rating scale (representing a judgment task) and a preference market (representing a choice task). We find that the rating scale-based task invokes significantly higher perceived ease of use than the preference market-based task and that perceived ease of use mediates the effect of the task representation treatment on the users’ decision quality. Furthermore, we find that the understandability of ideas being evaluated, which we assess through the ideas’ readability, and the perception of the task’s variability moderate the strength of this mediation effect, which becomes stronger with increasing perceived task variability and decreasing understandability of the ideas. We contribute to the literature by explaining how perceptual differences of task representations for open idea evaluation affect the decision quality of users and translate into differences in mechanism accuracy. These results enhance our understanding of how crowdsourcing as a novel mode of value creation may effectively complement traditional work structures.

Suggested Citation

  • Ivo Blohm & Christoph Riedl & Johann Füller & Jan Marco Leimeister, 2016. "Rate or Trade? Identifying Winning Ideas in Open Idea Sourcing," Information Systems Research, INFORMS, vol. 27(1), pages 27-48, March.
  • Handle: RePEc:inm:orisre:v:27:y:2016:i:1:p:27-48
    DOI: 10.1287/isre.2015.0605
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/isre.2015.0605
    Download Restriction: no

    File URL: https://libkey.io/10.1287/isre.2015.0605?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. repec:cup:cbooks:9780511771576 is not listed on IDEAS
    2. Einhorn, Hj & Hogarth, Rm, 1981. "Behavioral Decision-Theory - Processes Of Judgment And Choice," Journal of Accounting Research, Wiley Blackwell, vol. 19(1), pages 1-31.
    3. David Tilson & Kalle Lyytinen & Carsten Sørensen, 2010. "Research Commentary ---Digital Infrastructures: The Missing IS Research Agenda," Information Systems Research, INFORMS, vol. 21(4), pages 748-759, December.
    4. Martin Spann & Bernd Skiera, 2003. "Internet-Based Virtual Stock Markets for Business Forecasting," Management Science, INFORMS, vol. 49(10), pages 1310-1326, October.
    5. Christian Hildebrand & Gerald Häubl & Andreas Herrmann & Jan R. Landwehr, 2013. "When Social Media Can Be Bad for You: Community Feedback Stifles Consumer Creativity and Reduces Satisfaction with Self-Designed Products," Information Systems Research, INFORMS, vol. 24(1), pages 14-29, March.
    6. Gerrit Kamp & Peter Koen, 2009. "Improving the Idea Screening Process within Organizations using Prediction Markets: A Theoretical Perspective," Journal of Prediction Markets, University of Buckingham Press, vol. 3(2), pages 39-64, August.
    7. Quentin Jones & Gilad Ravid & Sheizaf Rafaeli, 2004. "Information Overload and the Message Dynamics of Online Interaction Spaces: A Theoretical Model and Empirical Exploration," Information Systems Research, INFORMS, vol. 15(2), pages 194-210, June.
    8. Hun‐Tong Tan & Elaine Ying Wang & Bo Zhou, 2014. "When the Use of Positive Language Backfires: The Joint Effect of Tone, Readability, and Investor Sophistication on Earnings Judgments," Journal of Accounting Research, Wiley Blackwell, vol. 52(1), pages 273-302, March.
    9. Justin Wolfers & Eric Zitzewitz, 2004. "Prediction Markets," Journal of Economic Perspectives, American Economic Association, vol. 18(2), pages 107-126, Spring.
    10. Yuxiang Zhao & Qinghua Zhu, 2014. "Evaluation on crowdsourcing research: Current status and future direction," Information Systems Frontiers, Springer, vol. 16(3), pages 417-434, July.
    11. Goodwin, Paul & Lawton, Richard, 1999. "On the asymmetry of the symmetric MAPE," International Journal of Forecasting, Elsevier, vol. 15(4), pages 405-408, October.
    12. Peter Todd & Izak Benbasat, 1999. "Evaluating the Impact of DSS, Cognitive Effort, and Incentives on Strategy Selection," Information Systems Research, INFORMS, vol. 10(4), pages 356-374, December.
    13. Christina Ann LaComb & Janet Arlie Barnett & Qimei Pan, 2007. "The imagination market," Information Systems Frontiers, Springer, vol. 9(2), pages 245-256, July.
    14. Moez Limayem & Gerardine DeSanctis, 2000. "Providing Decisional Guidance for Multicriteria Decision Making in Groups," Information Systems Research, INFORMS, vol. 11(4), pages 386-401, December.
    15. Franke, Nikolaus & Shah, Sonali, 2003. "How communities support innovative activities: an exploration of assistance and sharing among end-users," Research Policy, Elsevier, vol. 32(1), pages 157-178, January.
    16. Stefan Luckner & Christof Weinhardt, 2007. "How to Pay Traders in Information Markets: Results from a Field Experiment," Journal of Prediction Markets, University of Buckingham Press, vol. 1(2), pages 147-156, July.
    17. Christian Slamka & Wolfgang Jank & Bernd Skiera, 2012. "Second‐Generation Prediction Markets for Information Aggregation: A Comparison of Payoff Mechanisms," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 31(6), pages 469-489, September.
    18. Stefan Luckner & Christof Weinhardt, 2007. "How to pay traders in information markets? Results from a field experiment," Artefactual Field Experiments 00107, The Field Experiments Website.
    19. Karan Girotra & Christian Terwiesch & Karl T. Ulrich, 2010. "Idea Generation and the Quality of the Best Idea," Management Science, INFORMS, vol. 56(4), pages 591-605, April.
    20. Brian Spears & Christina LaComb & John Interrante & Janet Barnett & Deniz Senturk-Dogonaksoy, 2009. "Examining Trader Behavior in Idea Markets: An Implementation of GE's Imagination Markets," Journal of Prediction Markets, University of Buckingham Press, vol. 3(1), pages 17-39, April.
    21. Sujan, Mita, 1985. "Consumer Knowledge: Effects on Evaluation Strategies Mediating Consumer Judgments," Journal of Consumer Research, Journal of Consumer Research Inc., vol. 12(1), pages 31-46, June.
    22. Robin Hanson, 2003. "Combinatorial Information Market Design," Information Systems Frontiers, Springer, vol. 5(1), pages 107-119, January.
    23. Chris Forman & Nicolas van Zeebroeck, 2012. "From Wires to Partners: How the Internet Has Fostered R&D Collaborations Within Firms," Management Science, INFORMS, vol. 58(8), pages 1549-1568, August.
    24. Easley,David & Kleinberg,Jon, 2010. "Networks, Crowds, and Markets," Cambridge Books, Cambridge University Press, number 9780521195331, October.
    25. Bo Cowgill & Eric Zitzewitz, 2015. "Corporate Prediction Markets: Evidence from Google, Ford, and Firm X," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 82(4), pages 1309-1341.
    26. repec:reg:rpubli:460 is not listed on IDEAS
    27. Deborah Compeau & Barbara Marcolin & Helen Kelley & Chris Higgins, 2012. "Research Commentary ---Generalizability of Information Systems Research Using Student Subjects---A Reflection on Our Practices and Recommendations for Future Research," Information Systems Research, INFORMS, vol. 23(4), pages 1093-1109, December.
    28. Kevin J. Boudreau & Eva C. Guinan & Karim R. Lakhani & Christoph Riedl, 2016. "Looking Across and Looking Beyond the Knowledge Frontier: Intellectual Distance, Novelty, and Resource Allocation in Science," Management Science, INFORMS, vol. 62(10), pages 2765-2783, October.
    29. Ozer, Muammer, 2005. "Factors which influence decision making in new product evaluation," European Journal of Operational Research, Elsevier, vol. 163(3), pages 784-801, June.
    30. Iris Vessey & Dennis Galletta, 1991. "Cognitive Fit: An Empirical Study of Information Acquisition," Information Systems Research, INFORMS, vol. 2(1), pages 63-84, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Lele Cao & Vilhelm von Ehrenheim & Sebastian Krakowski & Xiaoxue Li & Alexandra Lutz, 2022. "Using Deep Learning to Find the Next Unicorn: A Practical Synthesis," Papers 2210.14195, arXiv.org, revised Jun 2024.
    2. Samer Faraj & Georg von Krogh & Eric Monteiro & Karim R. Lakhani, 2016. "Special Section Introduction—Online Community as Space for Knowledge Flows," Information Systems Research, INFORMS, vol. 27(4), pages 668-684, December.
    3. Thomas Görzen & Dennis Kundisch, 2019. "When in Doubt Follow the Crowd: How Idea Quality Moderates the Effect of an Anchor on Idea Evaluation," Working Papers Dissertations 57, Paderborn University, Faculty of Business Administration and Economics.
    4. Ho Cheung Brian Lee & Sulin Ba & Xinxin Li & Jan Stallaert, 2018. "Salience Bias in Crowdsourcing Contests," Information Systems Research, INFORMS, vol. 29(2), pages 401-418, June.
    5. Sander, Julian, 2024. "The Role of Emotions in Investment Decisions: The Effects of Vividness of a Crowdfunding Campaign Video," Thesis Commons 6gptv, Center for Open Science.
    6. Weiquan Wang & Jingjun (David) Xu & May Wang, 2018. "Effects of Recommendation Neutrality and Sponsorship Disclosure on Trust vs. Distrust in Online Recommendation Agents: Moderating Role of Explanations for Organic Recommendations," Management Science, INFORMS, vol. 64(11), pages 5198-5219, November.
    7. Julia Troll & Ivo Blohm & Jan Marco Leimeister, 2019. "Why Incorporating a Platform-Intermediary can Increase Crowdsourcees’ Engagement," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 61(4), pages 433-450, August.
    8. Dahlander, Linus & Beretta, Michela & Thomas, Arne & Kazemi, Shahab & Fenger, Morten H.J. & Frederiksen, Lars, 2023. "Weeding out or picking winners in open innovation? Factors driving multi-stage crowd selection on LEGO ideas," Research Policy, Elsevier, vol. 52(10).
    9. Dominik Dellermann & Nikolaus Lipusch & Philipp Ebel & Jan Marco Leimeister, 2019. "Design principles for a hybrid intelligence decision support system for business model validation," Electronic Markets, Springer;IIM University of St. Gallen, vol. 29(3), pages 423-441, September.
    10. Tat Koon Koh & Muller Y. M. Cheung, 2022. "Seeker Exemplars and Quantitative Ideation Outcomes in Crowdsourcing Contests," Information Systems Research, INFORMS, vol. 33(1), pages 265-284, March.
    11. Christoph Riedl & Tom Grad & Christopher Lettl, 2024. "Competition and Collaboration in Crowdsourcing Communities: What happens when peers evaluate each other?," Papers 2404.14141, arXiv.org.
    12. Ho Cheung Brian Lee & Jan Stallaert & Ming Fan, 2020. "Anomalies in Probability Estimates for Event Forecasting on Prediction Markets," Production and Operations Management, Production and Operations Management Society, vol. 29(9), pages 2077-2095, September.
    13. Christoph Riedl & Victor P. Seidel, 2018. "Learning from Mixed Signals in Online Innovation Communities," Organization Science, INFORMS, vol. 29(6), pages 1010-1032, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ivo Blohm & Christoph Riedl & Johann Fuller & Orhan Koroglu & Jan Marco Leimeister & Helmut Krcmar, 2012. "The Effects of Prediction Market Design and Price Elasticity on Trading Performance of Users: An Experimental Analysis," Papers 1204.3457, arXiv.org.
    2. Edoardo Gaffeo, 2013. "Using information markets in grantmaking. An assessment of the issues involved and an application to Italian banking foundations," DEM Discussion Papers 2013/08, Department of Economics and Management.
    3. Patrick Buckley & Fergal O’Brien, 0. "The effect of malicious manipulations on prediction market accuracy," Information Systems Frontiers, Springer, vol. 0, pages 1-13.
    4. Patrick Buckley & Fergal O’Brien, 2017. "The effect of malicious manipulations on prediction market accuracy," Information Systems Frontiers, Springer, vol. 19(3), pages 611-623, June.
    5. Riekhof, Hans-Christian & Riekhof, Marie-Catherine & Brinkhoff, Stefan, 2012. "Predictive Markets: Ein vielversprechender Weg zur Verbesserung der Prognosequalität im Unternehmen?," PFH Forschungspapiere/Research Papers 2012/07, PFH Private University of Applied Sciences, Göttingen.
    6. Karimi, Majid & Zaerpour, Nima, 2022. "Put your money where your forecast is: Supply chain collaborative forecasting with cost-function-based prediction markets," European Journal of Operational Research, Elsevier, vol. 300(3), pages 1035-1049.
    7. Pavel Atanasov & Phillip Rescober & Eric Stone & Samuel A. Swift & Emile Servan-Schreiber & Philip Tetlock & Lyle Ungar & Barbara Mellers, 2017. "Distilling the Wisdom of Crowds: Prediction Markets vs. Prediction Polls," Management Science, INFORMS, vol. 63(3), pages 691-706, March.
    8. Ledyard, John & Hanson, Robin & Ishikida, Takashi, 2009. "An experimental test of combinatorial information markets," Journal of Economic Behavior & Organization, Elsevier, vol. 69(2), pages 182-189, February.
    9. Coulomb, Renaud & Sangnier, Marc, 2014. "The impact of political majorities on firm value: Do electoral promises or friendship connections matter?," Journal of Public Economics, Elsevier, vol. 115(C), pages 158-170.
    10. Ujwal Kayande & Arnaud De Bruyn & Gary L. Lilien & Arvind Rangaswamy & Gerrit H. van Bruggen, 2009. "How Incorporating Feedback Mechanisms in a DSS Affects DSS Evaluations," Information Systems Research, INFORMS, vol. 20(4), pages 527-546, December.
    11. Snowberg, Erik & Wolfers, Justin & Zitzewitz, Eric, 2013. "Prediction Markets for Economic Forecasting," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 657-687, Elsevier.
    12. Martin Spann & Bernd Skiera, 2009. "Sports forecasting: a comparison of the forecast accuracy of prediction markets, betting odds and tipsters," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 28(1), pages 55-72.
    13. Li Chen & Paulo Goes & Wynd Harris & James Marsden & John Zhang, 2010. "Preference Markets for Innovation Ranking and Selection," Interfaces, INFORMS, vol. 40(2), pages 144-153, April.
    14. Liangfei Qiu & Subodha Kumar, 2017. "Understanding Voluntary Knowledge Provision and Content Contribution Through a Social-Media-Based Prediction Market: A Field Experiment," Information Systems Research, INFORMS, vol. 28(3), pages 529-546, September.
    15. Justin Wolfers & Eric Zitzewitz, 2004. "Prediction Markets," Journal of Economic Perspectives, American Economic Association, vol. 18(2), pages 107-126, Spring.
    16. Clayton Arlen Looney & Andrew M. Hardin, 2009. "Decision Support for Retirement Portfolio Management: Overcoming Myopic Loss Aversion via Technology Design," Management Science, INFORMS, vol. 55(10), pages 1688-1703, October.
    17. Coulomb, Renaud & Sangnier, Marc, 2014. "The impact of political majorities on firm value: Do electoral promises or friendship connections matter?," Journal of Public Economics, Elsevier, vol. 115(C), pages 158-170.
    18. Robin Hanson, 2006. "Designing real terrorism futures," Public Choice, Springer, vol. 128(1), pages 257-274, July.
    19. Strijbis, Oliver & Arnesen, Sveinung, 2019. "Explaining variance in the accuracy of prediction markets," International Journal of Forecasting, Elsevier, vol. 35(1), pages 408-419.
    20. van Bruggen, G.H. & Spann, M. & Lilien, G.L. & Skiera, B., 2006. "Institutional Forecasting: The Performance of Thin Virtual Stock Markets," ERIM Report Series Research in Management ERS-2006-028-MKT, Erasmus Research Institute of Management (ERIM), ERIM is the joint research institute of the Rotterdam School of Management, Erasmus University and the Erasmus School of Economics (ESE) at Erasmus University Rotterdam.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:orisre:v:27:y:2016:i:1:p:27-48. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.