IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2408.09742.html
   My bibliography  Save this paper

Paired Completion: Flexible Quantification of Issue-framing at Scale with LLMs

Author

Listed:
  • Simon D Angus
  • Lachlan O'Neill

Abstract

Detecting and quantifying issue framing in textual discourse - the perspective one takes to a given topic (e.g. climate science vs. denialism, misogyny vs. gender equality) - is highly valuable to a range of end-users from social and political scientists to program evaluators and policy analysts. However, conceptual framing is notoriously challenging for automated natural language processing (NLP) methods since the words and phrases used by either `side' of an issue are often held in common, with only subtle stylistic flourishes separating their use. Here we develop and rigorously evaluate new detection methods for issue framing and narrative analysis within large text datasets. By introducing a novel application of next-token log probabilities derived from generative large language models (LLMs) we show that issue framing can be reliably and efficiently detected in large corpora with only a few examples of either perspective on a given issue, a method we call `paired completion'. Through 192 independent experiments over three novel, synthetic datasets, we evaluate paired completion against prompt-based LLM methods and labelled methods using traditional NLP and recent LLM contextual embeddings. We additionally conduct a cost-based analysis to mark out the feasible set of performant methods at production-level scales, and a model bias analysis. Together, our work demonstrates a feasible path to scalable, accurate and low-bias issue-framing in large corpora.

Suggested Citation

  • Simon D Angus & Lachlan O'Neill, 2024. "Paired Completion: Flexible Quantification of Issue-framing at Scale with LLMs," Papers 2408.09742, arXiv.org.
  • Handle: RePEc:arx:papers:2408.09742
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2408.09742
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Chong, Dennis & Druckman, James N., 2007. "Framing Public Opinion in Competitive Democracies," American Political Science Review, Cambridge University Press, vol. 101(4), pages 637-655, November.
    2. Jason Barabas & Jennifer Jerit, 2009. "Estimating the Causal Effects of Media Coverage on Policy‐Specific Knowledge," American Journal of Political Science, John Wiley & Sons, vol. 53(1), pages 73-89, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Katerina Linos & Kimberly Twist, 2016. "The Supreme Court, the Media, and Public Opinion: Comparing Experimental and Observational Methods," The Journal of Legal Studies, University of Chicago Press, vol. 45(2), pages 223-254.
    2. Schnellenbach, Jan & Schubert, Christian, 2015. "Behavioral political economy: A survey," European Journal of Political Economy, Elsevier, vol. 40(PB), pages 395-417.
    3. Midha, Joshua, 2022. "The Cycle of Rule: Existential Risks, Continuity Of Governance, And Conflict-Based Preservation," SocArXiv vc7w9, Center for Open Science.
    4. Fung, Timothy K.F. & Choi, Doo Hun & Scheufele, Dietram A. & Shaw, Bret R., 2014. "Public opinion about biofuels: The interplay between party identification and risk/benefit perception," Energy Policy, Elsevier, vol. 73(C), pages 344-355.
    5. Matthew Gentzkow & Jesse M. Shapiro & Matt Taddy, 2019. "Measuring Group Differences in High‐Dimensional Choices: Method and Application to Congressional Speech," Econometrica, Econometric Society, vol. 87(4), pages 1307-1340, July.
    6. Bloemraad, Irene & Voss, Kim & Silva, Fabiana, 2014. "Framing the Immigrant Movement as about Rights, Family, or Economics: Which Appeals Resonate and for Whom?," Institute for Research on Labor and Employment, Working Paper Series qt3b32w33p, Institute of Industrial Relations, UC Berkeley.
    7. Justin Wedeking, 2010. "Supreme Court Litigants and Strategic Framing," American Journal of Political Science, John Wiley & Sons, vol. 54(3), pages 617-631, July.
    8. Matt Guardino & Suzanne Mettler, 2020. "Revealing the “Hidden welfare state†: How policy information influences public attitudes about tax expenditures," Journal of Behavioral Public Administration, Center for Experimental and Behavioral Public Administration, vol. 3(1).
    9. Bhalotra, Sonia & Clots-Figueras, Irma & Iyer, Lakshmi, 2021. "Religion and abortion: The role of politician identity," Journal of Development Economics, Elsevier, vol. 153(C).
    10. Max Grömping & Darren R. Halpin, 2021. "Do think tanks generate media attention on issues they care about? Mediating internal expertise and prevailing governmental agendas," Policy Sciences, Springer;Society of Policy Sciences, vol. 54(4), pages 849-866, December.
    11. Trisha R. Shrum, 2021. "The salience of future impacts and the willingness to pay for climate change mitigation: an experiment in intergenerational framing," Climatic Change, Springer, vol. 165(1), pages 1-20, March.
    12. Rogers, Todd & Nickerson, David W., 2013. "Can Inaccurate Beliefs about Incumbents be Changed? And Can Reframing Change Votes?," Working Paper Series rwp13-018, Harvard University, John F. Kennedy School of Government.
    13. Hawkins, Christopher V. & Chia-Yuan, Yu, 2018. "Voter support for environmental bond referenda," Land Use Policy, Elsevier, vol. 76(C), pages 193-200.
    14. Shapiro, Matthew A. & Bolsen, Toby, 2019. "Korean perceptions of transboundary air pollution and domestic coal development: Two framing experiments," Energy Policy, Elsevier, vol. 126(C), pages 333-342.
    15. Bechtel, Michael & Hainmueller, Jens & Hangartner, Dominik & Helbling, Marc, 2015. "Reality Bites: The Limits of Framing Effects for Salient and Contested Policy Issues," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 3(3), pages 683-695.
    16. Sigurd Hilmo Lundheim & Giuseppe Pellegrini-Masini & Christian A. Klöckner & Stefan Geiss, 2022. "Developing a Theoretical Framework to Explain the Social Acceptability of Wind Energy," Energies, MDPI, vol. 15(14), pages 1-24, July.
    17. Giebler, Heiko & Banducci, Susan & Kritzinger, Sylvia, 2017. "New perspectives on information and electoral competition," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 52(4), pages 429-435.
    18. Grant D. Jacobsen, 2019. "How do different sources of policy analysis affect policy preferences? Experimental evidence from the United States," Policy Sciences, Springer;Society of Policy Sciences, vol. 52(3), pages 315-342, September.
    19. Stefanie Haeffele & Virgil Henry Storr, 2021. "Rhetoric as a Way of Limiting the Range of Acceptable Policy Positions," Journal of Private Enterprise, The Association of Private Enterprise Education, vol. 36(Spring 20), pages 1-16.
    20. McComas, Katherine A. & Schuldt, Jonathon P. & Burge, Colleen A. & Roh, Sungjong, 2015. "Communicating about marine disease: The effects of message frames on policy support," Marine Policy, Elsevier, vol. 57(C), pages 45-52.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2408.09742. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.