IDEAS home Printed from https://ideas.repec.org/a/spr/comaot/v29y2023i1d10.1007_s10588-022-09362-3.html
   My bibliography  Save this article

Does big data serve policy? Not without context. An experiment with in silico social science

Author

Listed:
  • Chris Graziul

    (University of Chicago)

  • Alexander Belikov

    (University of Chicago)

  • Ishanu Chattopadyay

    (University of Chicago)

  • Ziwen Chen

    (University of Chicago)

  • Hongbo Fang

    (Carnegie Mellon University)

  • Anuraag Girdhar

    (University of Chicago)

  • Xiaoshuang Jia

    (Sun Yat-sen University)

  • P. M. Krafft

    (University of Oxford)

  • Max Kleiman-Weiner

    (MIT
    Harvard University)

  • Candice Lewis

    (University of Chicago)

  • Chen Liang

    (University of Chicago)

  • John Muchovej

    (MIT
    Harvard University)

  • Alejandro Vientós

    (MIT
    Rutgers University)

  • Meg Young

    (Cornell University)

  • James Evans

    (University of Chicago
    Santa Fe Institute)

Abstract

The DARPA Ground Truth project sought to evaluate social science by constructing four varied simulated social worlds with hidden causality and unleashed teams of scientists to collect data, discover their causal structure, predict their future, and prescribe policies to create desired outcomes. This large-scale, long-term experiment of in silico social science, about which the ground truth of simulated worlds was known, but not by us, reveals the limits of contemporary quantitative social science methodology. First, problem solving without a shared ontology—in which many world characteristics remain existentially uncertain—poses strong limits to quantitative analysis even when scientists share a common task, and suggests how they could become insurmountable without it. Second, data labels biased the associations our analysts made and assumptions they employed, often away from the simulated causal processes those labels signified, suggesting limits on the degree to which analytic concepts developed in one domain may port to others. Third, the current standard for computational social science publication is a demonstration of novel causes, but this limits the relevance of models to solve problems and propose policies that benefit from the simpler and less surprising answers associated with most important causes, or the combination of all causes. Fourth, most singular quantitative methods applied on their own did not help to solve most analytical challenges, and we explored a range of established and emerging methods, including probabilistic programming, deep neural networks, systems of predictive probabilistic finite state machines, and more to achieve plausible solutions. However, despite these limitations common to the current practice of computational social science, we find on the positive side that even imperfect knowledge can be sufficient to identify robust prediction if a more pluralistic approach is applied. Applying competing approaches by distinct subteams, including at one point the vast TopCoder.com global community of problem solvers, enabled discovery of many aspects of the relevant structure underlying worlds that singular methods could not. Together, these lessons suggest how different a policy-oriented computational social science would be than the computational social science we have inherited. Computational social science that serves policy would need to endure more failure, sustain more diversity, maintain more uncertainty, and allow for more complexity than current institutions support.

Suggested Citation

  • Chris Graziul & Alexander Belikov & Ishanu Chattopadyay & Ziwen Chen & Hongbo Fang & Anuraag Girdhar & Xiaoshuang Jia & P. M. Krafft & Max Kleiman-Weiner & Candice Lewis & Chen Liang & John Muchovej &, 2023. "Does big data serve policy? Not without context. An experiment with in silico social science," Computational and Mathematical Organization Theory, Springer, vol. 29(1), pages 188-219, March.
  • Handle: RePEc:spr:comaot:v:29:y:2023:i:1:d:10.1007_s10588-022-09362-3
    DOI: 10.1007/s10588-022-09362-3
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s10588-022-09362-3
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    File URL: https://libkey.io/10.1007/s10588-022-09362-3?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Markusen, James R. & Venables, Anthony J., 1988. "Trade policy with increasing returns and imperfect competition : Contradictory results from competing assumptions," Journal of International Economics, Elsevier, vol. 24(3-4), pages 299-316, May.
    2. Handcock, Mark S. & Hunter, David R. & Butts, Carter T. & Goodreau, Steven M. & Morris, Martina, 2008. "statnet: Software Tools for the Representation, Visualization, Analysis and Simulation of Network Data," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 24(i01).
    3. Roger Pielke & Tom Wigley & Christopher Green, 2008. "Dangerous assumptions," Nature, Nature, vol. 452(7187), pages 531-532, April.
    4. Sascha Holzhauer & Friedrich Krebs & Andreas Ernst, 2013. "Considering baseline homophily when generating spatial social networks for agent-based modelling," Computational and Mathematical Organization Theory, Springer, vol. 19(2), pages 128-150, June.
    5. Sean F. Reardon & Lindsay Fox & Joseph Townsend, 2015. "Neighborhood Income Composition by Household Race and Income, 1990–2009," The ANNALS of the American Academy of Political and Social Science, , vol. 660(1), pages 78-97, July.
    6. Liu, Xi & Gong, Li & Gong, Yongxi & Liu, Yu, 2015. "Revealing travel patterns and city structure with taxi trip data," Journal of Transport Geography, Elsevier, vol. 43(C), pages 78-90.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Frank Asche, 2001. "Testing the effect of an anti-dumping duty: The US salmon market," Empirical Economics, Springer, vol. 26(2), pages 343-355.
    2. Samrachana Adhikari & Beau Dabbs, 2018. "Social Network Analysis in R: A Software Review," Journal of Educational and Behavioral Statistics, , vol. 43(2), pages 225-253, April.
    3. Baccini, Leonardo & Impullitti, Giammario & Malesky, Edmund J., 2019. "Globalization and state capitalism: Assessing Vietnam's accession to the WTO," Journal of International Economics, Elsevier, vol. 119(C), pages 75-92.
    4. Mario Larch, 2007. "The Home Market Effect in Models with Multinational Enterprises," Review of International Economics, Wiley Blackwell, vol. 15(1), pages 62-74, February.
    5. Bolling, H. Christine & Neff, Steven & Handy, Charles R., 1998. "U.S. Foreign Direct Investment in the Western Hemisphere Processed Food Industry," Agricultural Economic Reports 34017, United States Department of Agriculture, Economic Research Service.
    6. Kirtonia, Sajeeb & Sun, Yanshuo, 2022. "Evaluating rail transit's comparative advantages in travel cost and time over taxi with open data in two U.S. cities," Transport Policy, Elsevier, vol. 115(C), pages 75-87.
    7. Zhou, Xiaolu & Wang, Mingshu & Li, Dongying, 2019. "Bike-sharing or taxi? Modeling the choices of travel mode in Chicago using machine learning," Journal of Transport Geography, Elsevier, vol. 79(C), pages 1-1.
    8. Li, Ze-Tao & Nie, Wei-Peng & Cai, Shi-Min & Zhao, Zhi-Dan & Zhou, Tao, 2023. "Exploring the topological characteristics of urban trip networks based on taxi trajectory data," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 609(C).
    9. Bagwell, Kyle & Staiger, Robert W., 2012. "The economics of trade agreements in the linear Cournot delocation model," Journal of International Economics, Elsevier, vol. 88(1), pages 32-46.
    10. Ting Wang & Yong Zhang & Meiye Li & Lei Liu, 2019. "How Do Passengers with Different Using Frequencies Choose between Traditional Taxi Service and Online Car-Hailing Service? A Case Study of Nanjing, China," Sustainability, MDPI, vol. 11(23), pages 1-18, November.
    11. Changhee Kim & Soo Wook Kim & Hee Jay Kang & Seung-Min Song, 2017. "What Makes Urban Transportation Efficient? Evidence from Subway Transfer Stations in Korea," Sustainability, MDPI, vol. 9(11), pages 1-18, November.
    12. Zhang, Shen & Liu, Xin & Tang, Jinjun & Cheng, Shaowu & Qi, Yong & Wang, Yinhai, 2018. "Spatio-temporal modeling of destination choice behavior through the Bayesian hierarchical approach," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 512(C), pages 537-551.
    13. Conconi, P., 2000. "Trade Bloc Formation Under Imperfect Competition," The Warwick Economics Research Paper Series (TWERPS) 571, University of Warwick, Department of Economics.
    14. Yang, Xiping & Fang, Zhixiang & Xu, Yang & Yin, Ling & Li, Junyi & Lu, Shiwei, 2019. "Spatial heterogeneity in spatial interaction of human movements—Insights from large-scale mobile positioning data," Journal of Transport Geography, Elsevier, vol. 78(C), pages 29-40.
    15. Takatoshi Tabuchi & Kristian Behrens & Andrea R. Lamorgese, 2004. "Testing the Home Market Effects in a Multi-country World: The Theory," Econometric Society 2004 Far Eastern Meetings 595, Econometric Society.
    16. Apantri Peungnumsai & Apichon Witayangkurn & Masahiko Nagai & Hiroyuki Miyazaki, 2018. "A Taxi Zoning Analysis Using Large-Scale Probe Data: A Case Study for Metropolitan Bangkok," The Review of Socionetwork Strategies, Springer, vol. 12(1), pages 21-45, June.
    17. Keith Head & Thierry Mayer & John Ries, 2000. "On the Pervasiveness of Home Market Effects," Econometric Society World Congress 2000 Contributed Papers 0862, Econometric Society.
    18. Jonas Friege & Georg Holtz & Emile Chappin, 2016. "Exploring Homeowners’ Insulation Activity," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 19(1), pages 1-4.
    19. Ignatius J. Horstmann & James R. Markusen, 1990. "Endogenous Market Structures in International Trade," NBER Working Papers 3283, National Bureau of Economic Research, Inc.
    20. De Nicola, Giacomo & Fritz, Cornelius & Mehrl, Marius & Kauermann, Göran, 2023. "Dependence matters: Statistical models to identify the drivers of tie formation in economic networks," Journal of Economic Behavior & Organization, Elsevier, vol. 215(C), pages 351-363.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:comaot:v:29:y:2023:i:1:d:10.1007_s10588-022-09362-3. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.