IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0156527.html
   My bibliography  Save this article

Crowdsourcing the Measurement of Interstate Conflict

Author

Listed:
  • Vito D’Orazio
  • Michael Kenwick
  • Matthew Lane
  • Glenn Palmer
  • David Reitter

Abstract

Much of the data used to measure conflict is extracted from news reports. This is typically accomplished using either expert coders to quantify the relevant information or machine coders to automatically extract data from documents. Although expert coding is costly, it produces quality data. Machine coding is fast and inexpensive, but the data are noisy. To diminish the severity of this tradeoff, we introduce a method for analyzing news documents that uses crowdsourcing, supplemented with computational approaches. The new method is tested on documents about Militarized Interstate Disputes, and its accuracy ranges between about 68 and 76 percent. This is shown to be a considerable improvement over automated coding, and to cost less and be much faster than expert coding.

Suggested Citation

  • Vito D’Orazio & Michael Kenwick & Matthew Lane & Glenn Palmer & David Reitter, 2016. "Crowdsourcing the Measurement of Interstate Conflict," PLOS ONE, Public Library of Science, vol. 11(6), pages 1-21, June.
    • Handle: RePEc:plo:pone00:0156527
    DOI: 10.1371/journal.pone.0156527
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0156527
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0156527&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0156527?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Gabriele Paolacci & Jesse Chandler & Panagiotis G. Ipeirotis, 2010. "Running experiments on Amazon Mechanical Turk," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 5(5), pages 411-419, August.
    2. D'Orazio, Vito & Landis, Steven T. & Palmer, Glenn & Schrodt, Philip, 2014. "Separating the Wheat from the Chaff: Applications of Automated Document Classification Using Support Vector Machines," Political Analysis, Cambridge University Press, vol. 22(2), pages 224-242, April.
    3. repec:cup:judgdm:v:5:y:2010:i:5:p:411-419 is not listed on IDEAS
    4. Seth Cooper & Firas Khatib & Adrien Treuille & Janos Barbero & Jeehyung Lee & Michael Beenen & Andrew Leaver-Fay & David Baker & Zoran Popović & Foldit players, 2010. "Predicting protein structures with a multiplayer online game," Nature, Nature, vol. 466(7307), pages 756-760, August.
    5. King, Gary & Lowe, Will, 2003. "An Automated Information Extraction Tool for International Conflict Data with Performance as Good as Human Coders: A Rare Events Evaluation Design," International Organization, Cambridge University Press, vol. 57(3), pages 617-642, July.
    6. Berinsky, Adam J. & Huber, Gregory A. & Lenz, Gabriel S., 2012. "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk," Political Analysis, Cambridge University Press, vol. 20(3), pages 351-368, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Glenn Palmer & Roseanne W McManus & Vito D’Orazio & Michael R Kenwick & Mikaela Karstens & Chase Bloch & Nick Dietrich & Kayla Kahn & Kellan Ritter & Michael J Soules, 2022. "The MID5 Dataset, 2011–2014: Procedures, coding rules, and description," Conflict Management and Peace Science, Peace Science Society (International), vol. 39(4), pages 470-482, July.
    2. Zhanna Terechshenko, 2020. "Hot under the collar: A latent measure of interstate hostility," Journal of Peace Research, Peace Research Institute Oslo, vol. 57(6), pages 764-776, November.
    3. Juan D Botero & Weisi Guo & Guillem Mosquera & Alan Wilson & Samuel Johnson & Gicela A Aguirre-Garcia & Leonardo A Pachon, 2019. "Gang confrontation: The case of Medellin (Colombia)," PLOS ONE, Public Library of Science, vol. 14(12), pages 1-19, December.
    4. Michael R Kenwick & Beth A Simmons & Richard J McAlexander, 2024. "Infrastructure and authority at the state’s edge: The Border Crossings of the World dataset," Journal of Peace Research, Peace Research Institute Oslo, vol. 61(3), pages 500-510, May.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Pan, Jing Yu & Liu, Dahai, 2022. "Mask-wearing intentions on airplanes during COVID-19 – Application of theory of planned behavior model," Transport Policy, Elsevier, vol. 119(C), pages 32-44.
    2. Rebecca R Carter & Analisa DiFeo & Kath Bogie & Guo-Qiang Zhang & Jiayang Sun, 2014. "Crowdsourcing Awareness: Exploration of the Ovarian Cancer Knowledge Gap through Amazon Mechanical Turk," PLOS ONE, Public Library of Science, vol. 9(1), pages 1-10, January.
    3. Michele Cantarella & Chiara Strozzi, 2021. "Workers in the crowd: the labor market impact of the online platform economy [An evaluation of instrumental variable strategies for estimating the effects of catholic schooling]," Industrial and Corporate Change, Oxford University Press and the Associazione ICC, vol. 30(6), pages 1429-1458.
    4. Robbett, Andrea & Matthews, Peter Hans, 2018. "Partisan bias and expressive voting," Journal of Public Economics, Elsevier, vol. 157(C), pages 107-120.
    5. Sandra Wankmüller, 2023. "A comparison of approaches for imbalanced classification problems in the context of retrieving relevant documents for an analysis," Journal of Computational Social Science, Springer, vol. 6(1), pages 91-163, April.
    6. Sam R. Bell & K. Chad Clay & Amanda Murdie, 2019. "Join the Chorus, Avoid the Spotlight: The Effect of Neighborhood and Social Dynamics on Human Rights Organization Shaming," Journal of Conflict Resolution, Peace Science Society (International), vol. 63(1), pages 167-193, January.
    7. Park, JungKun & Ahn, Jiseon & Thavisay, Toulany & Ren, Tianbao, 2019. "Examining the role of anxiety and social influence in multi-benefits of mobile payment service," Journal of Retailing and Consumer Services, Elsevier, vol. 47(C), pages 140-149.
    8. Chunhao Wei & Han Chen & Yee Ming Lee, 2022. "COVID-19 preventive measures and restaurant customers’ intention to dine out: the role of brand trust and perceived risk," Service Business, Springer;Pan-Pacific Business Association, vol. 16(3), pages 581-600, September.
    9. Desbordes, Rodolphe & Vicard, Vincent, 2009. "Foreign direct investment and bilateral investment treaties: An international political perspective," Journal of Comparative Economics, Elsevier, vol. 37(3), pages 372-386, September.
    10. Masha Shunko & Julie Niederhoff & Yaroslav Rosokha, 2018. "Humans Are Not Machines: The Behavioral Impact of Queueing Design on Service Time," Management Science, INFORMS, vol. 64(1), pages 453-473, January.
    11. Christoph Safferling & Aaron Lowen, 2011. "Economics in the Kingdom of Loathing: Analysis of Virtual Market Data," Working Paper Series of the Department of Economics, University of Konstanz 2011-30, Department of Economics, University of Konstanz.
    12. Prpić, John & Shukla, Prashant P. & Kietzmann, Jan H. & McCarthy, Ian P., 2015. "How to work a crowd: Developing crowd capital through crowdsourcing," Business Horizons, Elsevier, vol. 58(1), pages 77-85.
    13. Yoram Halevy & Guy Mayraz, 2024. "Identifying Rule-Based Rationality," The Review of Economics and Statistics, MIT Press, vol. 106(5), pages 1369-1380, September.
    14. Abel Brodeur, Nikolai M. Cook, Anthony Heyes, 2022. "We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell Us about Publication Bias and p-Hacking in Online Experiments," LCERPA Working Papers am0133, Laurier Centre for Economic Research and Policy Analysis.
    15. Lude, Maximilian & Prügl, Reinhard, 2021. "Experimental studies in family business research," Journal of Family Business Strategy, Elsevier, vol. 12(1).
    16. Mattozzi, Andrea & Snowberg, Erik, 2018. "The right type of legislator: A theory of taxation and representation," Journal of Public Economics, Elsevier, vol. 159(C), pages 54-65.
    17. Jasper Grashuis & Theodoros Skevas & Michelle S. Segovia, 2020. "Grocery Shopping Preferences during the COVID-19 Pandemic," Sustainability, MDPI, vol. 12(13), pages 1-10, July.
    18. Jeanette A.M.J. Deetlefs & Mathew Chylinski & Andreas Ortmann, 2015. "MTurk ‘Unscrubbed’: Exploring the good, the ‘Super’, and the unreliable on Amazon’s Mechanical Turk," Discussion Papers 2015-20, School of Economics, The University of New South Wales.
    19. Jun Zhang & Joon Soo Lim, 2021. "Mitigating negative spillover effects in a product-harm crisis: strategies for market leaders versus market challengers," Journal of Brand Management, Palgrave Macmillan, vol. 28(1), pages 77-98, January.
    20. Haas, Nicholas & Hassan, Mazen & Mansour, Sarah & Morton, Rebecca B., 2021. "Polarizing information and support for reform," Journal of Economic Behavior & Organization, Elsevier, vol. 185(C), pages 883-901.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0156527. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.