IDEAS home Printed from https://ideas.repec.org/a/inm/orisre/v28y2017i1p137-158.html
   My bibliography  Save this article

Cost-Effective Quality Assurance in Crowd Labeling

Author

Listed:
  • Jing Wang

    (School of Business and Management, Hong Kong University of Science and Technology, Clear Water Bay, Kowloon, Hong Kong)

  • Panagiotis G. Ipeirotis

    (Leonard Stern School of Business, New York University, New York, New York 10012)

  • Foster Provost

    (Leonard Stern School of Business, New York University, New York, New York 10012)

Abstract

The emergence of online paid micro-crowdsourcing platforms, such as Amazon Mechanical Turk, allows on-demand and at-scale distribution of tasks to human workers around the world. In such settings, online workers come and complete small tasks posted by employers, working for as long or as little as they wish, a process that eliminates the overhead of hiring (and dismissal). This flexibility introduces a different set of inefficiencies: verifying the quality of every submitted piece of work is an expensive operation that often requires the same level of effort as performing the task itself. A number of research challenges arise in such settings. How can we ensure that the submitted work is accurate? What allocation strategies can be employed to make the best use of the available labor force? How can we appropriately assess the performance of individual workers? In this paper, we consider labeling tasks and develop a comprehensive scheme for managing the quality of crowd labeling: First, we present several algorithms for inferring the true classes of objects and the quality of participating workers, assuming the labels are collected all at once before the inference. Next, we allow employers to adaptively decide which object to assign to the next arriving worker and propose several heuristic-based dynamic label allocation strategies to achieve the desired data quality with significantly fewer labels. Experimental results on both simulated and real data confirm the superior performance of the proposed allocation strategies over other existing policies. Finally, we introduce two novel metrics that can be used to objectively rank the performance of crowdsourced workers after fixing correctable worker errors and taking into account the costs of different classification errors. In particular, the worker value metric directly measures the monetary value contributed by each label of a worker toward meeting the quality requirements and provides a basis for the design of fair and efficient compensation schemes.

Suggested Citation

  • Jing Wang & Panagiotis G. Ipeirotis & Foster Provost, 2017. "Cost-Effective Quality Assurance in Crowd Labeling," Information Systems Research, INFORMS, vol. 28(1), pages 137-158, March.
  • Handle: RePEc:inm:orisre:v:28:y:2017:i:1:p:137-158
    DOI: 10.1287/isre.2016.0661
    as

    Download full text from publisher

    File URL: https://doi.org/10.1287/isre.2016.0661
    Download Restriction: no

    File URL: https://libkey.io/10.1287/isre.2016.0661?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Maytal Saar-Tsechansky & Prem Melville & Foster Provost, 2009. "Active Feature-Value Acquisition," Management Science, INFORMS, vol. 55(4), pages 664-684, April.
    2. Nikolay Archak & Anindya Ghose & Panagiotis G. Ipeirotis, 2011. "Deriving the Pricing Power of Product Features by Mining Consumer Reviews," Management Science, INFORMS, vol. 57(8), pages 1485-1509, August.
    3. A. P. Dawid & A. M. Skene, 1979. "Maximum Likelihood Estimation of Observer Error‐Rates Using the EM Algorithm," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 28(1), pages 20-28, March.
    4. Wolfgang Ketter & John Collins & Maria Gini & Alok Gupta & Paul Schrater, 2012. "Real-Time Tactical and Strategic Sales Management for Intelligent Agents Guided by Economic Regimes," Information Systems Research, INFORMS, vol. 23(4), pages 1263-1283, December.
    5. I. Robert Chiang & Vijay S. Mookerjee, 2004. "A Fault Threshold Policy to Manage Software Development Projects," Information Systems Research, INFORMS, vol. 15(1), pages 3-21, March.
    6. Gediminas Adomavicius & Alok Gupta & Dmitry Zhdanov, 2009. "Designing Intelligent Software Agents for Auctions with Limited Information Feedback," Information Systems Research, INFORMS, vol. 20(4), pages 507-526, December.
    7. Antonio Moreno & Christian Terwiesch, 2014. "Doing Business with Strangers: Reputation in Online Service Marketplaces," Information Systems Research, INFORMS, vol. 25(4), pages 865-886, December.
    8. Maytal Saar-Tsechansky & Foster Provost, 2007. "Decision-Centric Active Learning of Binary-Outcome Models," Information Systems Research, INFORMS, vol. 18(1), pages 4-22, March.
    9. Christina Aperjis & Ramesh Johari, 2010. "Optimal Windows for Aggregating Ratings in Electronic Marketplaces," Management Science, INFORMS, vol. 56(5), pages 864-880, May.
    10. Zhiqiang Zheng & Balaji Padmanabhan, 2006. "Selectively Acquiring Customer Information: A New Data Acquisition Problem and an Active Learning-Based Solution," Management Science, INFORMS, vol. 52(5), pages 697-712, May.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Fügener, A. & Grahl, J. & Gupta, A. & Ketter, W., 2019. "Cognitive challenges in human-AI collaboration: Investigating the path towards productive delegation," ERIM Report Series Research in Management ERS-2019-003-LIS, Erasmus Research Institute of Management (ERIM), ERIM is the joint research institute of the Rotterdam School of Management, Erasmus University and the Erasmus School of Economics (ESE) at Erasmus University Rotterdam.
    2. Ruyi Ge & Zhiqiang (Eric) Zheng & Xuan Tian & Li Liao, 2021. "Human–Robot Interaction: When Investors Adjust the Usage of Robo-Advisors in Peer-to-Peer Lending," Information Systems Research, INFORMS, vol. 32(3), pages 774-785, September.
    3. Xuan Bi & Mochen Yang & Gediminas Adomavicius, 2024. "Consumer Acquisition for Recommender Systems: A Theoretical Framework and Empirical Evaluations," Information Systems Research, INFORMS, vol. 35(1), pages 339-362, March.
    4. Tomer Geva & Maytal Saar‐Tsechansky, 2021. "Who Is a Better Decision Maker? Data‐Driven Expert Ranking Under Unobserved Quality," Production and Operations Management, Production and Operations Management Society, vol. 30(1), pages 127-144, January.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dominik Gutt & Jürgen Neumann & Steffen Zimmermann & Dennis Kundisch & Jianqing Chen, 2018. "Design of Review Systems - A Strategic Instrument to shape Online Review Behavior and Economic Outcomes," Working Papers Dissertations 42, Paderborn University, Faculty of Business Administration and Economics.
    2. Micha Kahlen & Karsten Schroer & Wolfgang Ketter & Alok Gupta, 2024. "Smart Markets for Real-Time Allocation of Multiproduct Resources: The Case of Shared Electric Vehicles," Information Systems Research, INFORMS, vol. 35(2), pages 871-889, June.
    3. Borchert, Philipp & Coussement, Kristof & De Weerdt, Jochen & De Caigny, Arno, 2024. "Industry-sensitive language modeling for business," European Journal of Operational Research, Elsevier, vol. 315(2), pages 691-702.
    4. Lingfang (Ivy) Li & Erte Xiao, 2014. "Money Talks: Rebate Mechanisms in Reputation System Design," Management Science, INFORMS, vol. 60(8), pages 2054-2072, August.
    5. Yixin Lu & Alok Gupta & Wolfgang Ketter & Eric van Heck, 2019. "Dynamic Decision Making in Sequential Business-to-Business Auctions: A Structural Econometric Approach," Management Science, INFORMS, vol. 65(8), pages 3853-3876, August.
    6. Meghana Deodhar & Joydeep Ghosh & Maytal Saar-Tsechansky & Vineet Keshari, 2017. "Active Learning with Multiple Localized Regression Models," INFORMS Journal on Computing, INFORMS, vol. 29(3), pages 503-522, August.
    7. Apostolos Filippas & John Horton & Joseph M. Golden, 2017. "Reputation in the Long-Run," CESifo Working Paper Series 6750, CESifo.
    8. Yingfei Wang & Inbal Yahav & Balaji Padmanabhan, 2024. "Smart Testing with Vaccination: A Bandit Algorithm for Active Sampling for Managing COVID-19," Information Systems Research, INFORMS, vol. 35(1), pages 120-144, March.
    9. Kaiquan Xu & Stephen Shaoyi Liao & Raymond Y. K. Lau & J. Leon Zhao, 2014. "Effective Active Learning Strategies for the Use of Large-Margin Classifiers in Semantic Annotation: An Optimal Parameter Discovery Perspective," INFORMS Journal on Computing, INFORMS, vol. 26(3), pages 461-483, August.
    10. Wolfgang Ketter & Karsten Schroer & Konstantina Valogianni, 2023. "Information Systems Research for Smart Sustainable Mobility: A Framework and Call for Action," Information Systems Research, INFORMS, vol. 34(3), pages 1045-1065, September.
    11. Ransome Epie Bawack & Samuel Fosso Wamba & Kevin Daniel André Carillo & Shahriar Akter, 2022. "Artificial intelligence in E-Commerce: a bibliometric study and literature review," Electronic Markets, Springer;IIM University of St. Gallen, vol. 32(1), pages 297-338, March.
    12. Alain Bensoussan & Radha Mookerjee & Vijay Mookerjee & Wei T. Yue, 2009. "Maintaining Diagnostic Knowledge-Based Systems: A Control-Theoretic Approach," Management Science, INFORMS, vol. 55(2), pages 294-310, February.
    13. Apostolos Filippas & John J. Horton & Joseph M. Golden, 2019. "Reputation Inflation," NBER Working Papers 25857, National Bureau of Economic Research, Inc.
    14. Foster, Joshua, 2022. "How rating mechanisms shape user search, quality inference and engagement in online platforms: Experimental evidence," Journal of Business Research, Elsevier, vol. 142(C), pages 791-807.
    15. Mochen Yang & Gediminas Adomavicius & Gordon Burtch & Yuqing Rena, 2018. "Mind the Gap: Accounting for Measurement Error and Misclassification in Variables Generated via Data Mining," Information Systems Research, INFORMS, vol. 29(1), pages 4-24, March.
    16. Jorge Mejia & Shawn Mankad & Anandasivam Gopal, 2019. "A for Effort? Using the Crowd to Identify Moral Hazard in New York City Restaurant Hygiene Inspections," Information Systems Research, INFORMS, vol. 30(4), pages 1363-1386, December.
    17. Xuan Bi & Mochen Yang & Gediminas Adomavicius, 2024. "Consumer Acquisition for Recommender Systems: A Theoretical Framework and Empirical Evaluations," Information Systems Research, INFORMS, vol. 35(1), pages 339-362, March.
    18. Jinsoo Park & Hamirahanim Abdul Rahman & Jihae Suh & Hazami Hussin, 2019. "A Study of Integrative Bargaining Model with Argumentation-Based Negotiation," Sustainability, MDPI, vol. 11(23), pages 1-21, December.
    19. Joe Cox & Daniel Kaimann, 2013. "The Signaling Effect of Critics - Evidence from a Market for Experience Goods," Working Papers CIE 68, Paderborn University, CIE Center for International Economics.
    20. Baxendale, Shane & Macdonald, Emma K. & Wilson, Hugh N., 2015. "The Impact of Different Touchpoints on Brand Consideration," Journal of Retailing, Elsevier, vol. 91(2), pages 235-253.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:orisre:v:28:y:2017:i:1:p:137-158. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.