IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2502.06387.html
   My bibliography  Save this paper

How Humans Help LLMs: Assessing and Incentivizing Human Preference Annotators

Author

Listed:
  • Shang Liu
  • Hanzhao Wang
  • Zhongyao Ma
  • Xiaocheng Li

Abstract

Human-annotated preference data play an important role in aligning large language models (LLMs). In this paper, we investigate the questions of assessing the performance of human annotators and incentivizing them to provide high-quality annotations. The quality assessment of language/text annotation faces two challenges: (i) the intrinsic heterogeneity among annotators, which prevents the classic methods that assume the underlying existence of a true label; and (ii) the unclear relationship between the annotation quality and the performance of downstream tasks, which excludes the possibility of inferring the annotators' behavior based on the model performance trained from the annotation data. Then we formulate a principal-agent model to characterize the behaviors of and the interactions between the company and the human annotators. The model rationalizes a practical mechanism of a bonus scheme to incentivize annotators which benefits both parties and it underscores the importance of the joint presence of an assessment system and a proper contract scheme. From a technical perspective, our analysis extends the existing literature on the principal-agent model by considering a continuous action space for the agent. We show the gap between the first-best and the second-best solutions (under the continuous action space) is of $\Theta(1/\sqrt{n \log n})$ for the binary contracts and $\Theta(1/n)$ for the linear contracts, where $n$ is the number of samples used for performance assessment; this contrasts with the known result of $\exp(-\Theta(n))$ for the binary contracts when the action space is discrete. Throughout the paper, we use real preference annotation data to accompany our discussions.

Suggested Citation

  • Shang Liu & Hanzhao Wang & Zhongyao Ma & Xiaocheng Li, 2025. "How Humans Help LLMs: Assessing and Incentivizing Human Preference Annotators," Papers 2502.06387, arXiv.org.
  • Handle: RePEc:arx:papers:2502.06387
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2502.06387
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Dirk Bergemann & Alessandro Bonatti, 2019. "Markets for Information: An Introduction," Annual Review of Economics, Annual Reviews, vol. 11(1), pages 85-107, August.
    2. Fabian Herweg & Daniel Muller & Philipp Weinschenk, 2010. "Binary Payment Schemes: Moral Hazard and Loss Aversion," American Economic Review, American Economic Association, vol. 100(5), pages 2451-2477, December.
    3. Barron, Daniel & Georgiadis, George & Swinkels, Jeroen M., 2020. "Optimal contracts with a risk-taking agent," Theoretical Economics, Econometric Society, vol. 15(2), May.
    4. Elodie Adida & Fernanda Bravo, 2019. "Contracts for Healthcare Referral Services: Coordination via Outcome-Based Penalty Contracts," Management Science, INFORMS, vol. 65(3), pages 1322-1341, March.
    5. Singh, Nirvikar, 1985. "Monitoring and Hierarchies: The Marginal Value of Information in a Principal-Agent Model," Journal of Political Economy, University of Chicago Press, vol. 93(3), pages 599-609, June.
    6. Nolan Miller & Paul Resnick & Richard Zeckhauser, 2005. "Eliciting Informative Feedback: The Peer-Prediction Method," Management Science, INFORMS, vol. 51(9), pages 1359-1373, September.
    7. Holmstrom, Bengt & Milgrom, Paul, 1987. "Aggregation and Linearity in the Provision of Intertemporal Incentives," Econometrica, Econometric Society, vol. 55(2), pages 303-328, March.
    8. Nitish Jain & Sameer Hasija & Dana G. Popescu, 2013. "Optimal Contracts for Outsourcing of Repair and Restoration Services," Operations Research, INFORMS, vol. 61(6), pages 1295-1311, December.
    9. Joann F. de Zegher & Dan A. Iancu & Hau L. Lee, 2019. "Designing Contracts and Sourcing Channels to Create Shared Value," Manufacturing & Service Operations Management, INFORMS, vol. 21(2), pages 271-289, May.
    10. Gabriel Carroll, 2015. "Robustness and Linear Contracts," American Economic Review, American Economic Association, vol. 105(2), pages 536-563, February.
    11. Lopomo, Giuseppe & Rigotti, Luca & Shannon, Chris, 2011. "Knightian uncertainty and moral hazard," Journal of Economic Theory, Elsevier, vol. 146(3), pages 1148-1172, May.
    12. Daniel Walton & Gabriel Carroll, 2022. "A General Framework for Robust Contracting Models," Econometrica, Econometric Society, vol. 90(5), pages 2129-2159, September.
    13. Giuseppe Moscarini & Lones Smith, 2002. "The Law of Large Demand for Information," Econometrica, Econometric Society, vol. 70(6), pages 2351-2366, November.
    14. Harris, Milton & Raviv, Artur, 1979. "Optimal incentive contracts with imperfect information," Journal of Economic Theory, Elsevier, vol. 20(2), pages 231-259, April.
    15. Corbett, Charles J. & DeCroix, Gregory A. & Ha, Albert Y., 2005. "Optimal shared-savings contracts in supply chains: Linear contracts and double moral hazard," European Journal of Operational Research, Elsevier, vol. 163(3), pages 653-667, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Rosenthal, Maxwell, 2023. "Robust incentives for risk," Journal of Mathematical Economics, Elsevier, vol. 109(C).
    2. Burkett, Justin & Rosenthal, Maxwell, 2024. "Statistical uncertainty and coarse contracts," Journal of Economic Theory, Elsevier, vol. 220(C).
    3. Carroll, Gabriel & Bolte, Lukas, 2023. "Robust contracting under double moral hazard," Theoretical Economics, Econometric Society, vol. 18(4), November.
    4. Inés Macho-Stadler & David Pérez-Castrillo, 2018. "Moral hazard: Base models and two extensions," Chapters, in: Luis C. Corchón & Marco A. Marini (ed.), Handbook of Game Theory and Industrial Organization, Volume I, chapter 16, pages 453-485, Edward Elgar Publishing.
    5. Paul Dütting & Michal Feldman & Daniel Peretz & Larry Samuelson, 2024. "Ambiguous Contracts," Econometrica, Econometric Society, vol. 92(6), pages 1967-1992, November.
    6. Xianyi Wang & Xiaofang Wang & Hui He, 2021. "Contracts to Coordinate Healthcare Providers in the Telemedicine Referral System," Sustainability, MDPI, vol. 13(18), pages 1-25, September.
    7. George Georgiadis & Balazs Szentes, 2020. "Optimal Monitoring Design," Econometrica, Econometric Society, vol. 88(5), pages 2075-2107, September.
    8. Matsushima, Hitoshi & Noda, Shunya, 2023. "Mechanism design with general ex-ante investments," Journal of Mathematical Economics, Elsevier, vol. 106(C).
    9. Peter Zhang, 2023. "Distributionally Robust Principal-Agent Problems and Optimality of Contracts," Papers 2303.07468, arXiv.org, revised Jan 2024.
    10. Hitoshi Matsushima & Shunya Noda, 2019. "Mechanism Design with General Ex-Ante Investments (Revised version of F415 )," CARF F-Series CARF-F-464, Center for Advanced Research in Finance, Faculty of Economics, The University of Tokyo.
    11. Hitoshi Matsushima & Shunya Noda, 2017. "Mechanism Design in Hidden Action and Hidden Information: Richness and Pure-VCG," CIRJE F-Series CIRJE-F-1057, CIRJE, Faculty of Economics, University of Tokyo.
    12. Hitoshi Matsushima & Shunya Noda, 2016. "Mechanism Design in Hidden Action and Hidden Information: Richness and Pure Groves," CARF F-Series CARF-F-386, Center for Advanced Research in Finance, Faculty of Economics, The University of Tokyo.
    13. Bartsch, Elga, 1996. "Enforcement of environmental liability in the case of uncertain causality and asymmetric information," Kiel Working Papers 755, Kiel Institute for the World Economy (IfW Kiel).
    14. Tal Alon & Paul Dutting & Yingkai Li & Inbal Talgam-Cohen, 2022. "Approximate Optimality of Linear Contracts Under Uncertainty," Papers 2211.06850, arXiv.org, revised Mar 2025.
    15. Paul Duetting & Michal Feldman & Inbal Talgam-Cohen, 2024. "Algorithmic Contract Theory: A Survey," Papers 2412.16384, arXiv.org.
    16. Martin Dumav, 2021. "Moral Hazard, Dynamic Incentives, and Ambiguous Perceptions," Papers 2110.15229, arXiv.org.
    17. Weinschenk, Philipp, 2024. "Incentives and performance under two-dimensional moral hazard," Journal of Economic Behavior & Organization, Elsevier, vol. 225(C), pages 107-115.
    18. Christian Kellner, 2017. "The principal-agent problem with smooth ambiguity," Review of Economic Design, Springer;Society for Economic Design, vol. 21(2), pages 83-119, June.
    19. Urmee Khan & Martin Dumav, 2018. "Moral Hazard, Uncertain Technologies, and Linear Contracts," Working Papers 201806, University of California at Riverside, Department of Economics.
    20. Tinglong Dai & Kinshuk Jerath, 2019. "Salesforce Contracting Under Uncertain Demand and Supply: Double Moral Hazard and Optimality of Smooth Contracts," Marketing Science, INFORMS, vol. 38(5), pages 852-870, September.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2502.06387. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.