IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2405.08030.html
   My bibliography  Save this paper

Distilling Data from Large Language Models: An Application to Research Productivity Measurement

Author

Listed:
  • Maya M. Durvasula
  • Sabri Eyuboglu
  • David M. Ritzwoller

Abstract

We develop a method for assigning high-quality labels to unstructured text, based on fine-tuning an open-source language model with data extracted from a proprietary large language model. We apply this method to construct a census of published clinical trials. With these data, we revisit a literature that contends that pharmaceutical research productivity is declining. Measurements of substantial increases in the quantity of clinical trials are central to this conclusion. In our data, the quantity and composition of trials are stable since 2010. Previous measurements are an artifact of biases driven by shifts in the composition of other forms of research.

Suggested Citation

  • Maya M. Durvasula & Sabri Eyuboglu & David M. Ritzwoller, 2024. "Distilling Data from Large Language Models: An Application to Research Productivity Measurement," Papers 2405.08030, arXiv.org, revised Dec 2024.
    • Handle: RePEc:arx:papers:2405.08030
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2405.08030
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Benjamin F. Jones, 2009. "The Burden of Knowledge and the "Death of the Renaissance Man": Is Innovation Getting Harder?," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 76(1), pages 283-317.
    2. Nicholas Bloom & Charles I. Jones & John Van Reenen & Michael Webb, 2020. "Are Ideas Getting Harder to Find?," American Economic Review, American Economic Association, vol. 110(4), pages 1104-1144, April.
    3. Kenneth F Schulz & Douglas G Altman & David Moher & for the CONSORT Group, 2010. "CONSORT 2010 Statement: Updated Guidelines for Reporting Parallel Group Randomised Trials," PLOS Medicine, Public Library of Science, vol. 7(3), pages 1-7, March.
    4. Chad Syverson, 2017. "Challenges to Mismeasurement Explanations for the US Productivity Slowdown," Journal of Economic Perspectives, American Economic Association, vol. 31(2), pages 165-186, Spring.
    5. Daron Acemoglu & Simon Johnson, 2007. "Disease and Development: The Effect of Life Expectancy on Economic Growth," Journal of Political Economy, University of Chicago Press, vol. 115(6), pages 925-985, December.
    6. Aghion, Philippe & Howitt, Peter, 1992. "A Model of Growth through Creative Destruction," Econometrica, Econometric Society, vol. 60(2), pages 323-351, March.
    7. Josh Angrist & Pierre Azoulay & Glenn Ellison & Ryan Hill & Susan Feng Lu, 2020. "Inside Job or Deep Impact? Extramural Citations and the Influence of Economic Scholarship," Journal of Economic Literature, American Economic Association, vol. 58(1), pages 3-52, March.
    8. Bryan Kelly & Dimitris Papanikolaou & Amit Seru & Matt Taddy, 2021. "Measuring Technological Innovation over the Long Run," American Economic Review: Insights, American Economic Association, vol. 3(3), pages 303-320, September.
    9. DiMasi, Joseph A. & Grabowski, Henry G. & Hansen, Ronald W., 2016. "Innovation in the pharmaceutical industry: New estimates of R&D costs," Journal of Health Economics, Elsevier, vol. 47(C), pages 20-33.
    10. Alwyn Young, 1998. "Growth without Scale Effects," Journal of Political Economy, University of Chicago Press, vol. 106(1), pages 41-63, February.
    11. Douglas Almond & Joseph J. Doyle & Amanda E. Kowalski & Heidi Williams, 2010. "Estimating Marginal Returns to Medical Care: Evidence from At-risk Newborns," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 125(2), pages 591-634.
    12. Philippe Aghion & Ufuk Akcigit & Antonin Bergeaud & Richard Blundell & David Hemous, 2019. "Innovation and Top Income Inequality," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 86(1), pages 1-45.
    13. Michael Park & Erin Leahey & Russell J. Funk, 2023. "Papers and patents are becoming less disruptive over time," Nature, Nature, vol. 613(7942), pages 138-144, January.
    14. Jones, Charles I, 1995. "R&D-Based Models of Economic Growth," Journal of Political Economy, University of Chicago Press, vol. 103(4), pages 759-784, August.
    15. Peretto, Pietro F, 1998. "Technological Change and Population Growth," Journal of Economic Growth, Springer, vol. 3(4), pages 283-311, December.
    16. Lisa Larrimore Ouellette & Bhaven N. Sampat, 2024. "The Feasibility of Using Bayh-Dole March-In Rights to Lower Drug Prices: An Update," NBER Working Papers 32217, National Bureau of Economic Research, Inc.
    17. Daron Acemoglu & David Autor & Christina Patterson, 2024. "Bottlenecks: Sectoral Imbalances and the US Productivity Slowdown," NBER Macroeconomics Annual, University of Chicago Press, vol. 38(1), pages 153-207.
    18. Pakes, Ariel S, 1986. "Patents as Options: Some Estimates of the Value of Holding European Patent Stocks," Econometrica, Econometric Society, vol. 54(4), pages 755-784, July.
    19. Isaiah Andrews & Maximilian Kasy, 2019. "Identification of and Correction for Publication Bias," American Economic Review, American Economic Association, vol. 109(8), pages 2766-2794, August.
    20. Easterlin, Richard A., 1999. "How beneficent is the market? A look at the modern history of mortality," European Review of Economic History, Cambridge University Press, vol. 3(3), pages 257-294, December.
    21. Ian Goldin & Pantelis Koutroumpis & François Lafond & Julian Winkler, 2024. "Why Is Productivity Slowing Down?," Journal of Economic Literature, American Economic Association, vol. 62(1), pages 196-268, March.
    22. Peter Howitt, 1999. "Steady Endogenous Growth with Population and R & D Inputs Growing," Journal of Political Economy, University of Chicago Press, vol. 107(4), pages 715-730, August.
    23. Leila Agha & David Molitor, 2018. "The Local Influence of Pioneer Investigators on Technology Adoption: Evidence from New Cancer Drugs," The Review of Economics and Statistics, MIT Press, vol. 100(1), pages 29-44, March.
    24. Marcella Alsan & Maya Durvasula & Harsh Gupta & Joshua Schwartzstein & Heidi Williams, 2024. "Representation and Extrapolation: Evidence from Clinical Trials," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 139(1), pages 575-635.
    25. Dinopoulos, Elias & Thompson, Peter, 1998. "Schumpeterian Growth without Scale Effects," Journal of Economic Growth, Springer, vol. 3(4), pages 313-335, December.
    26. Kyle Myers & Mark Pauly, 2019. "Endogenous productivity of demand‐induced R&D: evidence from pharmaceuticals," RAND Journal of Economics, RAND Corporation, vol. 50(3), pages 591-614, September.
    27. Danielle Li, 2017. "Expertise versus Bias in Evaluation: Evidence from the NIH," American Economic Journal: Applied Economics, American Economic Association, vol. 9(2), pages 60-92, April.
    28. Kyle Myers, 2020. "The Elasticity of Science," American Economic Journal: Applied Economics, American Economic Association, vol. 12(4), pages 103-134, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Cozzi, Guido, 2023. "Semi-endogenous or fully endogenous growth? A unified theory," Journal of Economic Theory, Elsevier, vol. 213(C).
    2. Steven Bond‐Smith, 2022. "Discretely innovating: The effect of limited market contestability on innovation and growth," Scottish Journal of Political Economy, Scottish Economic Society, vol. 69(3), pages 301-327, July.
    3. T. Gries & R. Grundmann & I. Palnau & M. Redlin, 2017. "Innovations, growth and participation in advanced economies - a review of major concepts and findings," International Economics and Economic Policy, Springer, vol. 14(2), pages 293-351, April.
    4. Cozzi, Guido & Pataracchia, Beatrice & Pfeiffer, Philipp & Marco, Ratto, 2017. "How much Keynes and how much Schumpeter? An Estimated Macromodel of the US Economy," JRC Working Papers in Economics and Finance 2017-01, Joint Research Centre, European Commission.
    5. Turnovsky, S., 2000. "Growth in an Open Economy: some Recent Developments," Papers 5, Warwick - Development Economics Research Centre.
    6. repec:got:cegedp:140 is not listed on IDEAS
    7. Baldanzi, Annarita & Bucci, Alberto & Prettner, Klaus, 2021. "Children’S Health, Human Capital Accumulation, And R&D-Based Economic Growth," Macroeconomic Dynamics, Cambridge University Press, vol. 25(3), pages 651-668, April.
    8. Gehringer, Agnieszka & Prettner, Klaus, 2019. "Longevity And Technological Change," Macroeconomic Dynamics, Cambridge University Press, vol. 23(4), pages 1471-1503, June.
    9. Richard M. H. Suen, 2013. "Research Policy and U.S. Economic Growth," Working papers 2013-18, University of Connecticut, Department of Economics.
    10. Capolupo, Rosa, 2009. "The New Growth Theories and Their Empirics after Twenty Years," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 3, pages 1-72.
    11. Sener, Fuat, 2008. "R&D policies, endogenous growth and scale effects," Journal of Economic Dynamics and Control, Elsevier, vol. 32(12), pages 3895-3916, December.
    12. Gray, Elie & Grimaud, André, 2016. "Using the Salop Circle to Study Scale Effects in Schumpeterian Growth Models: Why Inter-sectoral Knowledge Diffusion Matters," TSE Working Papers 16-676, Toulouse School of Economics (TSE).
    13. Cozzi, Guido, 2021. "Semi-Endogenous or Fully Endogenous Growth? A Simple Unified Theory," MPRA Paper 110681, University Library of Munich, Germany.
    14. Barbara Annicchiarico & Valentina Antonaroli & Alessandra Pelloni, 2022. "Optimal factor taxation in a scale free model of vertical innovation," Economic Inquiry, Western Economic Association International, vol. 60(2), pages 794-830, April.
    15. Dean Scrimgeour, 2015. "Dynamic Scoring in a Romer‐Style Economy," Southern Economic Journal, John Wiley & Sons, vol. 81(3), pages 697-723, January.
    16. Gray, Elie & Grimaud, André, 2014. "The Lindahl equilibrium in Schumpeterian growth models: Knowledge diffusion, social value of innovations and optimal R&D incentives," TSE Working Papers 14-469, Toulouse School of Economics (TSE).
    17. Carmelo Pierpaolo Parello, 2022. "Migration and growth in a Schumpeterian growth model with creative destruction [A model of growth through creative destruction]," Oxford Economic Papers, Oxford University Press, vol. 74(4), pages 1139-1166.
    18. Cozzi, Guido & Galli, Silvia, 2011. "Privatization of Knowledge: Did the U.S. Get It Right? (New Version)," MPRA Paper 29710, University Library of Munich, Germany.
    19. Elie Gray & André Grimaud, 2016. "The Lindahl equilibrium in Schumpeterian growth models," Journal of Evolutionary Economics, Springer, vol. 26(1), pages 101-142, March.
    20. Dean Scrimgeour, 2015. "Dynamic Scoring in a Romer-Style Economy," Southern Economic Journal, Southern Economic Association, vol. 81(3), pages 697-723, January.
    21. Holger Strulik & Klaus Prettner & Alexia Prskawetz, 2013. "The past and future of knowledge-based growth," Journal of Economic Growth, Springer, vol. 18(4), pages 411-437, December.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2405.08030. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.