IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0156527.html
   My bibliography  Save this article

Crowdsourcing the Measurement of Interstate Conflict

Author

Listed:
  • Vito D’Orazio
  • Michael Kenwick
  • Matthew Lane
  • Glenn Palmer
  • David Reitter

Abstract

Much of the data used to measure conflict is extracted from news reports. This is typically accomplished using either expert coders to quantify the relevant information or machine coders to automatically extract data from documents. Although expert coding is costly, it produces quality data. Machine coding is fast and inexpensive, but the data are noisy. To diminish the severity of this tradeoff, we introduce a method for analyzing news documents that uses crowdsourcing, supplemented with computational approaches. The new method is tested on documents about Militarized Interstate Disputes, and its accuracy ranges between about 68 and 76 percent. This is shown to be a considerable improvement over automated coding, and to cost less and be much faster than expert coding.

Suggested Citation

  • Vito D’Orazio & Michael Kenwick & Matthew Lane & Glenn Palmer & David Reitter, 2016. "Crowdsourcing the Measurement of Interstate Conflict," PLOS ONE, Public Library of Science, vol. 11(6), pages 1-21, June.
    • Handle: RePEc:plo:pone00:0156527
    DOI: 10.1371/journal.pone.0156527
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0156527
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0156527&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0156527?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. D'Orazio, Vito & Landis, Steven T. & Palmer, Glenn & Schrodt, Philip, 2014. "Separating the Wheat from the Chaff: Applications of Automated Document Classification Using Support Vector Machines," Political Analysis, Cambridge University Press, vol. 22(2), pages 224-242, April.
    2. repec:cup:judgdm:v:5:y:2010:i:5:p:411-419 is not listed on IDEAS
    3. Seth Cooper & Firas Khatib & Adrien Treuille & Janos Barbero & Jeehyung Lee & Michael Beenen & Andrew Leaver-Fay & David Baker & Zoran Popović & Foldit players, 2010. "Predicting protein structures with a multiplayer online game," Nature, Nature, vol. 466(7307), pages 756-760, August.
    4. King, Gary & Lowe, Will, 2003. "An Automated Information Extraction Tool for International Conflict Data with Performance as Good as Human Coders: A Rare Events Evaluation Design," International Organization, Cambridge University Press, vol. 57(3), pages 617-642, July.
    5. Berinsky, Adam J. & Huber, Gregory A. & Lenz, Gabriel S., 2012. "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk," Political Analysis, Cambridge University Press, vol. 20(3), pages 351-368, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Glenn Palmer & Roseanne W McManus & Vito D’Orazio & Michael R Kenwick & Mikaela Karstens & Chase Bloch & Nick Dietrich & Kayla Kahn & Kellan Ritter & Michael J Soules, 2022. "The MID5 Dataset, 2011–2014: Procedures, coding rules, and description," Conflict Management and Peace Science, Peace Science Society (International), vol. 39(4), pages 470-482, July.
    2. Zhanna Terechshenko, 2020. "Hot under the collar: A latent measure of interstate hostility," Journal of Peace Research, Peace Research Institute Oslo, vol. 57(6), pages 764-776, November.
    3. Juan D Botero & Weisi Guo & Guillem Mosquera & Alan Wilson & Samuel Johnson & Gicela A Aguirre-Garcia & Leonardo A Pachon, 2019. "Gang confrontation: The case of Medellin (Colombia)," PLOS ONE, Public Library of Science, vol. 14(12), pages 1-19, December.
    4. Michael R Kenwick & Beth A Simmons & Richard J McAlexander, 2024. "Infrastructure and authority at the state’s edge: The Border Crossings of the World dataset," Journal of Peace Research, Peace Research Institute Oslo, vol. 61(3), pages 500-510, May.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Robbett, Andrea & Matthews, Peter Hans, 2018. "Partisan bias and expressive voting," Journal of Public Economics, Elsevier, vol. 157(C), pages 107-120.
    2. Desbordes, Rodolphe & Vicard, Vincent, 2009. "Foreign direct investment and bilateral investment treaties: An international political perspective," Journal of Comparative Economics, Elsevier, vol. 37(3), pages 372-386, September.
    3. Christoph Safferling & Aaron Lowen, 2011. "Economics in the Kingdom of Loathing: Analysis of Virtual Market Data," Working Paper Series of the Department of Economics, University of Konstanz 2011-30, Department of Economics, University of Konstanz.
    4. Mattozzi, Andrea & Snowberg, Erik, 2018. "The right type of legislator: A theory of taxation and representation," Journal of Public Economics, Elsevier, vol. 159(C), pages 54-65.
    5. Jasper Grashuis & Theodoros Skevas & Michelle S. Segovia, 2020. "Grocery Shopping Preferences during the COVID-19 Pandemic," Sustainability, MDPI, vol. 12(13), pages 1-10, July.
    6. Jeanette A.M.J. Deetlefs & Mathew Chylinski & Andreas Ortmann, 2015. "MTurk ‘Unscrubbed’: Exploring the good, the ‘Super’, and the unreliable on Amazon’s Mechanical Turk," Discussion Papers 2015-20, School of Economics, The University of New South Wales.
    7. Cantarella, Michele & Strozzi, Chiara, 2019. "Workers in the Crowd: The Labour Market Impact of the Online Platform Economy," IZA Discussion Papers 12327, Institute of Labor Economics (IZA).
    8. John Hulland & Jeff Miller, 2018. "“Keep on Turkin’”?," Journal of the Academy of Marketing Science, Springer, vol. 46(5), pages 789-794, September.
    9. Kyungsik Han, 2018. "How do you perceive this author? Understanding and modeling authors’ communication quality in social media," PLOS ONE, Public Library of Science, vol. 13(2), pages 1-25, February.
    10. Azzam, Tarek & Harman, Elena, 2016. "Crowdsourcing for quantifying transcripts: An exploratory study," Evaluation and Program Planning, Elsevier, vol. 54(C), pages 63-73.
    11. Barton, Jared & Pan, Xiaofei, 2022. "Movin’ on up? A survey experiment on mobility enhancing policies," European Journal of Political Economy, Elsevier, vol. 74(C).
    12. Huet-Vaughn, Emiliano & Robbett, Andrea & Spitzer, Matthew, 2019. "A taste for taxes: Minimizing distortions using political preferences," Journal of Public Economics, Elsevier, vol. 180(C).
    13. Holgersen, Henning & Jia, Zhiyang & Svenkerud, Simen, 2021. "Who and how many can work from home? Evidence from task descriptions," Journal for Labour Market Research, Institut für Arbeitsmarkt- und Berufsforschung (IAB), Nürnberg [Institute for Employment Research, Nuremberg, Germany], vol. 55, pages 1-4.
    14. Gandullia, Luca & Lezzi, Emanuela & Parciasepe, Paolo, 2020. "Replication with MTurk of the experimental design by Gangadharan, Grossman, Jones & Leister (2018): Charitable giving across donor types," Journal of Economic Psychology, Elsevier, vol. 78(C).
    15. Prissé, Benjamin & Jorrat, Diego, 2022. "Lab vs online experiments: No differences," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 100(C).
    16. Min Chung Han, 2021. "Thumbs down on “likes”? The impact of Facebook reactions on online consumers’ nonprofit engagement behavior," International Review on Public and Nonprofit Marketing, Springer;International Association of Public and Non-Profit Marketing, vol. 18(2), pages 255-272, June.
    17. Narae Kim & Jeong-Nam Kim, 2024. "A COVID-19 Paradox of Communication, Ignorance, and Vaccination Intention," SAGE Open, , vol. 14(3), pages 21582440241, September.
    18. Dongshu Liu & Li Shao, 2024. "Nationalist propaganda and support for war in an authoritarian context: Evidence from China," Journal of Peace Research, Peace Research Institute Oslo, vol. 61(6), pages 985-1001, November.
    19. Valerio Capraro & Hélène Barcelo, 2021. "Punishing defectors and rewarding cooperators: Do people discriminate between genders?," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 7(1), pages 19-32, September.
    20. Kovacs, Attila, 2018. "Gender Differences in Equity Crowdfunding," OSF Preprints 5pcmb, Center for Open Science.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0156527. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.