IDEAS home Printed from https://ideas.repec.org/a/zbw/espost/261091.html
   My bibliography  Save this article

Examining Sentiment in Complex Texts. A Comparison of Different Computational Approaches

Author

Listed:
  • Munnes, Stefan
  • Harsch, Corinna
  • Knobloch, Marcel
  • Vogel, Johannes S.
  • Hipp, Lena
  • Schilling, Erik

Abstract

Can we rely on computational methods to accurately analyze complex texts? To answer this question, we compared different dictionary and scaling methods used in predicting the sentiment of German literature reviews to the “gold standard” of human-coded sentiments. Literature reviews constitute a challenging text corpus for computational analysis as they not only contain different text levels—for example, a summary of the work and the reviewer's appraisal—but are also characterized by subtle and ambiguous language elements. To take the nuanced sentiments of literature reviews into account, we worked with a metric rather than a dichotomous scale for sentiment analysis. The results of our analyses show that the predicted sentiments of prefabricated dictionaries, which are computationally efficient and require minimal adaption, have a low to medium correlation with the human-coded sentiments (r between 0.32 and 0.39). The accuracy of self-created dictionaries using word embeddings (both pre-trained and self-trained) was considerably lower (r between 0.10 and 0.28). Given the high coding intensity and contingency on seed selection as well as the degree of data pre-processing of word embeddings that we found with our data, we would not recommend them for complex texts without further adaptation. While fully automated approaches appear not to work in accurately predicting text sentiments with complex texts such as ours, we found relatively high correlations with a semiautomated approach (r of around 0.6)—which, however, requires intensive human coding efforts for the training dataset. In addition to illustrating the benefits and limits of computational approaches in analyzing complex text corpora and the potential of metric rather than binary scales of text sentiment, we also provide a practical guide for researchers to select an appropriate method and degree of pre-processing when working with complex texts.

Suggested Citation

  • Munnes, Stefan & Harsch, Corinna & Knobloch, Marcel & Vogel, Johannes S. & Hipp, Lena & Schilling, Erik, 2022. "Examining Sentiment in Complex Texts. A Comparison of Different Computational Approaches," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 5, pages 1-1.
  • Handle: RePEc:zbw:espost:261091
    DOI: 10.3389/fdata.2022.886362
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/261091/1/Full-text-article-Munnes-et-al-Examining-sentiment-in.pdf
    Download Restriction: no

    File URL: https://libkey.io/10.3389/fdata.2022.886362?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Rice, Douglas R. & Zorn, Christopher, 2021. "Corpus-based dictionaries for sentiment analysis of specialized vocabularies," Political Science Research and Methods, Cambridge University Press, vol. 9(1), pages 20-35, January.
    2. Lauderdale, Benjamin E. & Herzog, Alexander, 2016. "Measuring Political Positions from Legislative Speech," Political Analysis, Cambridge University Press, vol. 24(3), pages 374-394, July.
    3. Ng, Weiyi & Leung, Ming D., 2015. "For love or money? Gender differences in how one approaches getting a job," Institute for Research on Labor and Employment, Working Paper Series qt41f0k4gd, Institute of Industrial Relations, UC Berkeley.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Bernhardt, Lea & Dewenter, Ralf & Thomas, Tobias, 2023. "Measuring partisan media bias in US newscasts from 2001 to 2012," European Journal of Political Economy, Elsevier, vol. 78(C).
    2. Hugo Oriola & Matthieu Picault, 2023. "Opportunistic Political Central Bank Coverage: Does media coverage of ECB's Monetary Policy Impacts German Political Parties' Popularity?," EconomiX Working Papers 2023-30, University of Paris Nanterre, EconomiX.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dugoua, Eugenie & Dumas, Marion & Noailly, Joëlle, 2022. "Text as data in environmental economics and policy," LSE Research Online Documents on Economics 115396, London School of Economics and Political Science, LSE Library.
    2. Chitralekha Basu & Carles Boix & Sonia Giurumescu & Paulo Serôdio, 2022. "Democratizing from Within: British Elites and the Expansion of the Franchise," ECONtribute Discussion Papers Series 139, University of Bonn and University of Cologne, Germany.
    3. Ash, Elliott & Gauthier, Germain & Widmer, Philine, 2024. "Relatio: Text Semantics Capture Political and Economic Narratives," Political Analysis, Cambridge University Press, vol. 32(1), pages 115-132, January.
    4. Hausladen, Carina I. & Schubert, Marcel H. & Ash, Elliott, 2020. "Text classification of ideological direction in judicial opinions," International Review of Law and Economics, Elsevier, vol. 62(C).
    5. Josef Schwaiger & Timo Hammerl & Johannsen Florian & Susanne Leist, 2021. "UR: SMART–A tool for analyzing social media content," Information Systems and e-Business Management, Springer, vol. 19(4), pages 1275-1320, December.
    6. Sami Diaf & Jörg Döpke & Ulrich Fritsche & Ida Rockenbach, 2020. "Sharks and minnows in a shoal of words: Measuring latent ideological positions of German economic research institutes based on text mining techniques," Macroeconomics and Finance Series 202001, University of Hamburg, Department of Socioeconomics.
    7. Eitan Sapiro-Gheiler, 2018. ""Read My Lips": Using Automatic Text Analysis to Classify Politicians by Party and Ideology," Papers 1809.00741, arXiv.org.
    8. Mónica D. Oliveira & Inês Mataloto & Panos Kanavos, 2019. "Multi-criteria decision analysis for health technology assessment: addressing methodological challenges to improve the state of the art," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 20(6), pages 891-918, August.
    9. Diaf, Sami & Döpke, Jörg & Fritsche, Ulrich & Rockenbach, Ida, 2022. "Sharks and minnows in a shoal of words: Measuring latent ideological positions based on text mining techniques," European Journal of Political Economy, Elsevier, vol. 75(C).
    10. Christopher Wratil & Sara B Hobolt, 2019. "Public deliberations in the Council of the European Union: Introducing and validating DICEU," European Union Politics, , vol. 20(3), pages 511-531, September.
    11. Caroline Bhattacharya, 2020. "Gatekeeping the Plenary Floor: Discourse Network Analysis as a Novel Approach to Party Control," Politics and Governance, Cogitatio Press, vol. 8(2), pages 229-242.
    12. Peter Grajzl & Peter Murrell, 2024. "From Status to Contract? A Macrohistory from Early-Modern English Caselaw and Print Culture," CESifo Working Paper Series 11246, CESifo.
    13. Rangina Ahmad & Dominik Siemon & Ulrich Gnewuch & Susanne Robra-Bissantz, 2022. "Designing Personality-Adaptive Conversational Agents for Mental Health Care," Information Systems Frontiers, Springer, vol. 24(3), pages 923-943, June.
    14. Salla Simola & Jeremias Nieminen & Janne Tukiainen, 2023. "A century of partisanship in Finnish political speech," Discussion Papers 160, Aboa Centre for Economics.
    15. Jon H. Fiva & Oda Nedregård & Henning Øien, 2021. "Polarization in Parliamentary Speech," CESifo Working Paper Series 8818, CESifo.
    16. Miriam Barnum & James Lo, 2020. "Is the NPT unraveling? Evidence from text analysis of review conference statements," Journal of Peace Research, Peace Research Institute Oslo, vol. 57(6), pages 740-751, November.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:espost:261091. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://edirc.repec.org/data/zbwkide.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.