IDEAS home Printed from https://ideas.repec.org/a/spr/comaot/v18y2012i1d10.1007_s10588-011-9097-3.html
   My bibliography  Save this article

Opening the ‘black box’ of simulations: increased transparency and effective communication through the systematic design of experiments

Author

Listed:
  • Iris Lorscheid

    (Hamburg University of Technology)

  • Bernd-Oliver Heine

    (Hamburg University of Technology)

  • Matthias Meyer

    (Hamburg University of Technology)

Abstract

Many still view simulation models as a black box. This paper argues that perceptions could change if the systematic design of experiments (DOE) for simulation research was fully realized. DOE can increase (1) the transparency of simulation model behavior and (2) the effectiveness of reporting simulation results. Based on DOE principles, we develop a systematic procedure to guide the analysis of simulation models as well as concrete templates for sharing the results. A simulation model investigating the performance of learning algorithms in an economic mechanism design context illustrates our suggestions. Overall, the proposed systematic procedure for applying DOE principles complements current initiatives for a more standardized simulation research process.

Suggested Citation

  • Iris Lorscheid & Bernd-Oliver Heine & Matthias Meyer, 2012. "Opening the ‘black box’ of simulations: increased transparency and effective communication through the systematic design of experiments," Computational and Mathematical Organization Theory, Springer, vol. 18(1), pages 22-62, March.
    • Handle: RePEc:spr:comaot:v:18:y:2012:i:1:d:10.1007_s10588-011-9097-3
    DOI: 10.1007/s10588-011-9097-3
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s10588-011-9097-3
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s10588-011-9097-3?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Matteo Richiardi & Roberto Leombruni & Nicole J. Saam & Michele Sonnessa, 2006. "A Common Protocol for Agent-Based Social Simulation," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 9(1), pages 1-15.
    2. Dhananjay K. Gode & Shyam Sunder, 1997. "What Makes Markets Allocationally Efficient?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 112(2), pages 603-630.
    3. T. S. Raghu & P. K. Sen & H. R. Rao, 2003. "Relative Performance of Incentive Mechanisms: Computational Modeling and Simulation of Delegated Investment Decisions," Management Science, INFORMS, vol. 49(2), pages 160-178, February.
    4. Colin Camerer & Teck-Hua Ho, 1999. "Experience-weighted Attraction Learning in Normal Form Games," Econometrica, Econometric Society, vol. 67(4), pages 827-874, July.
    5. J. Gareth Polhill & Dawn C. Parker & Daniel Brown & Volker Grimm, 2008. "Using the ODD Protocol for Describing Three Agent-Based Social Simulation Models of Land-Use Change," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 11(2), pages 1-3.
    6. Regine Pei Tze Oh & Susan M. Sanchez & Thomas W. Lucas & Hong Wan & Mark E. Nissen, 2009. "Efficient experimental design tools for exploring large simulation models," Computational and Mathematical Organization Theory, Springer, vol. 15(3), pages 237-257, September.
    7. R. Myerson, 2010. "Nash Equilibrium and the History of Economic Theory," Voprosy Ekonomiki, NP Voprosy Ekonomiki, issue 6.
    8. Bernd-O. Heine & Matthias Meyer & Oliver Strangfeld, 2005. "Stylised Facts and the Contribution of Simulation to the Economic Analysis of Budgeting," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 8(4), pages 1-4.
    9. Markus Arnold & Eva Ponick & Heike Schenk-Mathes, 2008. "Groves Mechanism vs. Profit Sharing for Corporate Budgeting - An Experimental Analysis with Preplay Communication," European Accounting Review, Taylor & Francis Journals, vol. 17(1), pages 37-63.
    10. Jack P.C. Kleijnen, 2015. "Design and Analysis of Simulation Experiments," International Series in Operations Research and Management Science, Springer, edition 2, number 978-3-319-18087-8, January.
    11. Prajit K. Dutta, 1999. "Strategies and Games: Theory and Practice," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262041693, December.
    12. Jasmina Arifovic & John Ledyard, 2004. "Scaling Up Learning Models in Public Good Games," Journal of Public Economic Theory, Association for Public Economic Theory, vol. 6(2), pages 203-238, May.
    13. Grimm, Volker & Berger, Uta & DeAngelis, Donald L. & Polhill, J. Gary & Giske, Jarl & Railsback, Steven F., 2010. "The ODD protocol: A review and first update," Ecological Modelling, Elsevier, vol. 221(23), pages 2760-2768.
    14. Malerba, Franco, et al, 1999. "'History-Friendly' Models of Industry Evolution: The Computer Industry," Industrial and Corporate Change, Oxford University Press and the Associazione ICC, vol. 8(1), pages 3-40, March.
    15. Groves, Theodore, 1973. "Incentives in Teams," Econometrica, Econometric Society, vol. 41(4), pages 617-631, July.
    16. Theodore Groves & Martin Loeb, 1979. "Incentives in a Divisionalized Firm," Management Science, INFORMS, vol. 25(3), pages 221-230, March.
    17. Sarin, Rajiv & Vahid, Farshid, 1999. "Payoff Assessments without Probabilities: A Simple Dynamic Model of Choice," Games and Economic Behavior, Elsevier, vol. 28(2), pages 294-309, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Friederike Wall, 2023. "Modeling managerial search behavior based on Simon’s concept of satisficing," Computational and Mathematical Organization Theory, Springer, vol. 29(2), pages 265-299, June.
    2. Darío Blanco-Fernández & Stephan Leitner & Alexandra Rausch, 2023. "Interactions between the individual and the group level in organizations: The case of learning and group turnover," Central European Journal of Operations Research, Springer;Slovak Society for Operations Research;Hungarian Operational Research Society;Czech Society for Operations Research;Österr. Gesellschaft für Operations Research (ÖGOR);Slovenian Society Informatika - Section for Operational Research;Croatian Operational Research Society, vol. 31(4), pages 1087-1128, December.
    3. Patrick Afflerbach & Christopher Dun & Henner Gimpel & Dominik Parak & Johannes Seyfried, 2021. "A Simulation-Based Approach to Understanding the Wisdom of Crowds Phenomenon in Aggregating Expert Judgment," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 63(4), pages 329-348, August.
    4. Emanuele Borgonovo & Marco Pangallo & Jan Rivkin & Leonardo Rizzo & Nicolaj Siggelkow, 2022. "Sensitivity analysis of agent-based models: a new protocol," Computational and Mathematical Organization Theory, Springer, vol. 28(1), pages 52-94, March.
    5. Priscilla Avegliano & Jaime Simão Sichman, 2023. "Equation-Based Versus Agent-Based Models: Why Not Embrace Both for an Efficient Parameter Calibration?," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 26(4), pages 1-3.
    6. Iris Lorscheid & Matthias Meyer, 2021. "Toward a better understanding of team decision processes: combining laboratory experiments with agent-based modeling," Journal of Business Economics, Springer, vol. 91(9), pages 1431-1467, November.
    7. Apetrei, Cristina I. & Strelkovskii, Nikita & Khabarov, Nikolay & Javalera Rincón, Valeria, 2024. "Improving the representation of smallholder farmers’ adaptive behaviour in agent-based models: Learning-by-doing and social learning," Ecological Modelling, Elsevier, vol. 489(C).
    8. Dario Blanco-Fernandez & Stephan Leitner & Alexandra Rausch, 2022. "Interactions between the individual and the group level in organizations: The case of learning and autonomous group adaptation," Papers 2203.09162, arXiv.org.
    9. Leitner, Stephan & Wall, Friederike, 2022. "Micro-level dynamics in hidden action situations with limited information," Journal of Economic Behavior & Organization, Elsevier, vol. 204(C), pages 372-393.
    10. Davis, Natalie & Jarvis, Andrew & Polhill, J. Gareth, 2022. "Co-evolution of network structure and consumer inequality in a spatially explicit model of energetic resource acquisition," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 608(P1).
    11. Philippe Collard, 2022. "The “flat peer learning” agent-based model," Journal of Computational Social Science, Springer, vol. 5(1), pages 161-187, May.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Duffy, John, 2006. "Agent-Based Models and Human Subject Experiments," Handbook of Computational Economics, in: Leigh Tesfatsion & Kenneth L. Judd (ed.), Handbook of Computational Economics, edition 1, volume 2, chapter 19, pages 949-1011, Elsevier.
    2. Giorgio Fagiolo & Mattia Guerini & Francesco Lamperti & Alessio Moneta & Andrea Roventini, 2017. "Validation of Agent-Based Models in Economics and Finance," LEM Papers Series 2017/23, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    3. Mikhail Anufriev & Jasmina Arifovic & John Ledyard & Valentyn Panchenko, 2013. "Efficiency of continuous double auctions under individual evolutionary learning with full or limited information," Journal of Evolutionary Economics, Springer, vol. 23(3), pages 539-573, July.
    4. Weidlich, Anke & Veit, Daniel, 2008. "A critical survey of agent-based wholesale electricity market models," Energy Economics, Elsevier, vol. 30(4), pages 1728-1759, July.
    5. Christian Lohmann & Sandro Lombardo, 2014. "Resource allocation within a budgeting game: truthful reporting as the dominant strategy under collusion," Metrika: International Journal for Theoretical and Applied Statistics, Springer, vol. 25(1), pages 33-54, September.
    6. Markus C. Arnold & Eva Ponick, 2006. "Kommunikation im Groves-Mechanismus — Ergebnisse eines Laborexperiments," Schmalenbach Journal of Business Research, Springer, vol. 58(1), pages 89-120, February.
    7. Erhao Xie, 2019. "Monetary Payoff and Utility Function in Adaptive Learning Models," Staff Working Papers 19-50, Bank of Canada.
    8. Merbis, M.D., 1983. "On the compensator (Part III) : Stochastic Nash and team problems," Other publications TiSEM 8bee9de8-996e-47a2-998d-8, Tilburg University, School of Economics and Management.
    9. Lorscheid, Iris & Meyer, Matthias, 2016. "Divide and conquer: Configuring submodels for valid and efficient analyses of complex simulation models," Ecological Modelling, Elsevier, vol. 326(C), pages 152-161.
    10. Ahmed Laatabi & Nicolas Marilleau & Tri Nguyen-Huu & Hassan Hbid & Mohamed Ait Babram, 2018. "ODD+2D: An ODD Based Protocol for Mapping Data to Empirical ABMs," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 21(2), pages 1-9.
    11. Chen, Kebing & Xiao, Tiaojun, 2009. "Demand disruption and coordination of the supply chain with a dominant retailer," European Journal of Operational Research, Elsevier, vol. 197(1), pages 225-234, August.
    12. Flaminio Squazzoni, 2010. "The impact of agent-based models in the social sciences after 15 years of incursions," History of Economic Ideas, Fabrizio Serra Editore, Pisa - Roma, vol. 18(2), pages 197-234.
    13. Xie, Erhao, 2021. "Empirical properties and identification of adaptive learning models in behavioral game theory," Journal of Economic Behavior & Organization, Elsevier, vol. 191(C), pages 798-821.
    14. Huber, Robert & Bakker, Martha & Balmann, Alfons & Berger, Thomas & Bithell, Mike & Brown, Calum & Grêt-Regamey, Adrienne & Xiong, Hang & Le, Quang Bao & Mack, Gabriele & Meyfroidt, Patrick & Millingt, 2018. "Representation of decision-making in European agricultural agent-based models," Agricultural Systems, Elsevier, vol. 167(C), pages 143-160.
    15. Fernando Bernstein & Awi Federgruen, 2003. "Pricing and Replenishment Strategies in a Distribution System with Competing Retailers," Operations Research, INFORMS, vol. 51(3), pages 409-426, June.
    16. Arifovic, Jasmina & Karaivanov, Alexander, 2010. "Learning by doing vs. learning from others in a principal-agent model," Journal of Economic Dynamics and Control, Elsevier, vol. 34(10), pages 1967-1992, October.
    17. Funai, Naoki, 2022. "Reinforcement learning with foregone payoff information in normal form games," Journal of Economic Behavior & Organization, Elsevier, vol. 200(C), pages 638-660.
    18. Hopkins, Ed, 2007. "Adaptive learning models of consumer behavior," Journal of Economic Behavior & Organization, Elsevier, vol. 64(3-4), pages 348-368.
    19. Oldham, Matthew, 2020. "Quantifying the concerns of Dimon and Buffett with data and computation," Journal of Economic Dynamics and Control, Elsevier, vol. 113(C).
    20. Giorgio Fagiolo & Paul Windrum & Alessio Moneta, 2006. "Empirical Validation of Agent Based Models: A Critical Survey," LEM Papers Series 2006/14, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:comaot:v:18:y:2012:i:1:d:10.1007_s10588-011-9097-3. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.