IDEAS home Printed from https://ideas.repec.org/p/iza/izadps/dp8583.html
   My bibliography  Save this paper

Statistical Power of Within and Between-Subjects Designs in Economic Experiments

Author

Listed:
  • Bellemare, Charles

    (Université Laval)

  • Bissonnette, Luc

    (Laval University)

  • Kröger, Sabine

    (Université Laval)

Abstract

This paper discusses the choice of the number of participants for within-subjects (WS) designs and between-subjects (BS) designs based on simulations of statistical power allowing for different numbers of experimental periods. We illustrate the usefulness of the approach in the context of field experiments on gift exchange. Our results suggest that a BS design requires between 4 to 8 times more subjects than a WS design to reach an acceptable level of statistical power. Moreover, the predicted minimal sample sizes required to correctly detect a treatment effect with a probability of 80% greatly exceed sizes currently used in the literature. Our results suggest that adding experimental periods in an experiment can substantially increase the statistical power of a WS design, but have very little effect on the statistical power of the BS design. Finally, we discuss issues relating to numerical computation and present the powerBBK package programmed for STATA. This package allows users to conduct their own analysis of power for the different designs (WS and BS), conditional on user specified experimental parameters (true effect size, sample size, number of periods, noise levels for control and treatment, error distributions), statistical tests (parametric and nonparametric), and estimation methods (linear regression, binary choice models (probit and logit), censored regression models (tobit)).

Suggested Citation

  • Bellemare, Charles & Bissonnette, Luc & Kröger, Sabine, 2014. "Statistical Power of Within and Between-Subjects Designs in Economic Experiments," IZA Discussion Papers 8583, Institute of Labor Economics (IZA).
    • Handle: RePEc:iza:izadps:dp8583
    as

    Download full text from publisher

    File URL: https://docs.iza.org/dp8583.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Jan Stoop, 2014. "From the lab to the field: envelopes, dictators and manners," Experimental Economics, Springer;Economic Science Association, vol. 17(2), pages 304-313, June.
    2. Rutström, E. Elisabet & Wilcox, Nathaniel T., 2009. "Stated beliefs versus inferred beliefs: A methodological inquiry and experimental test," Games and Economic Behavior, Elsevier, vol. 67(2), pages 616-632, November.
    3. Charness, Gary & Gneezy, Uri & Kuhn, Michael A., 2012. "Experimental methods: Between-subject and within-subject design," Journal of Economic Behavior & Organization, Elsevier, vol. 81(1), pages 1-8.
    4. Trautmann, Stefan T. & van de Kuilen, Gijs & Zeckhauser, Richard J., 2013. "Social Class and Un(ethical) Behavior: A Framework, with Evidence from a Large Population Sample," Working Paper Series rwp13-004, Harvard University, John F. Kennedy School of Government.
    5. Andreas Ortman & Le Zhang, 2013. "Exploring the Meaning of Significance in Experimental Economics," Discussion Papers 2013-32, School of Economics, The University of New South Wales.
    6. Smith, Vernon L, et al, 1982. "Competitive Market Institutions: Double Auctions vs. Sealed Bid-Offer Auctions," American Economic Review, American Economic Association, vol. 72(1), pages 58-77, March.
    7. De Long, J Bradford & Lang, Kevin, 1992. "Are All Economic Hypotheses False?," Journal of Political Economy, University of Chicago Press, vol. 100(6), pages 1257-1272, December.
    8. John List & Sally Sadoff & Mathis Wagner, 2011. "So you want to run an experiment, now what? Some simple rules of thumb for optimal experimental design," Experimental Economics, Springer;Economic Science Association, vol. 14(4), pages 439-457, November.
    9. Maarten J. Voors & Eleonora E. M. Nillesen & Philip Verwimp & Erwin H. Bulte & Robert Lensink & Daan P. Van Soest, 2012. "Violent Conflict and Behavior: A Field Experiment in Burundi," American Economic Review, American Economic Association, vol. 102(2), pages 941-964, April.
    10. McKenzie, David, 2012. "Beyond baseline and follow-up: The case for more T in experiments," Journal of Development Economics, Elsevier, vol. 99(2), pages 210-221.
    11. Uri Gneezy & John A List, 2006. "Putting Behavioral Economics to Work: Testing for Gift Exchange in Labor Markets Using Field Experiments," Econometrica, Econometric Society, vol. 74(5), pages 1365-1384, September.
    12. A. H. Feiveson, 2002. "Power by simulation," Stata Journal, StataCorp LP, vol. 2(2), pages 107-124, May.
    13. Peter Bossaerts & Charles Plott & William R. Zame, 2007. "Prices and Portfolio Choices in Financial Markets: Theory, Econometrics, Experiments," Econometrica, Econometric Society, vol. 75(4), pages 993-1038, July.
    14. Bellemare, Charles & Shearer, Bruce, 2009. "Gift giving and worker productivity: Evidence from a firm-level experiment," Games and Economic Behavior, Elsevier, vol. 67(1), pages 233-244, September.
    15. Paul J. Ferraro & Michael K. Price, 2013. "Using Nonpecuniary Strategies to Influence Behavior: Evidence from a Large-Scale Field Experiment," The Review of Economics and Statistics, MIT Press, vol. 95(1), pages 64-73, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Reddinger, J. Lucas & Charness, Gary & Levine, David, 2022. "Prosocial motivation for vaccination," SocArXiv emj6v, Center for Open Science.
    2. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    3. Kathryn N. Vasilaky & J. Michelle Brock, 2020. "Power(ful) guidelines for experimental economists," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 6(2), pages 189-212, December.
    4. Cloos, Janis & Greiff, Matthias & Rusch, Hannes, 2019. "Geographical Concentration and Editorial Favoritism within the Field of Laboratory Experimental Economics," Research Memorandum 029, Maastricht University, Graduate School of Business and Economics (GSBE).
    5. Cloos, Janis & Greiff, Matthias & Rusch, Hannes, 2020. "Geographical Concentration and Editorial Favoritism within the Field of Laboratory Experimental Economics (RM/19/029-revised-)," Research Memorandum 014, Maastricht University, Graduate School of Business and Economics (GSBE).
    6. Braaten, Ragnhild Haugli & Brekke, Kjell Arne & Rogeberg, Ole, 2015. "Buying the right to do wrong – An experimental test of moral objections to trading emission permits," Resource and Energy Economics, Elsevier, vol. 42(C), pages 110-124.
    7. Cloos, Janis & Greiff, Matthias & Rusch, Hannes, 2021. "Editorial favoritism in the field of laboratory experimental economics (RM/20/014-revised-)," Research Memorandum 005, Maastricht University, Graduate School of Business and Economics (GSBE).
    8. Igor Asanov & Christoph Buehren & Panagiota Zacharodimou, 2020. "The power of experiments: How big is your n?," MAGKS Papers on Economics 202032, Philipps-Universität Marburg, Faculty of Business Administration and Economics, Department of Economics (Volkswirtschaftliche Abteilung).
    9. Iavor Bojinov & Ashesh Rambachan & Neil Shephard, 2021. "Panel experiments and dynamic causal effects: A finite population perspective," Quantitative Economics, Econometric Society, vol. 12(4), pages 1171-1196, November.
    10. Lennart Erixon & Louise Johannesson, 2015. "Is the psychology of high profits detrimental to industrial renewal? Experimental evidence for the theory of transformation pressure," Journal of Evolutionary Economics, Springer, vol. 25(2), pages 475-511, April.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    2. Leah H. Palm-Forster & Paul J. Ferraro & Nicholas Janusch & Christian A. Vossler & Kent D. Messer, 2019. "Behavioral and Experimental Agri-Environmental Research: Methodological Challenges, Literature Gaps, and Recommendations," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 73(3), pages 719-742, July.
    3. Charles Bellemare & Luc Bissonnette & Sabine Kröger, 2016. "Simulating power of economic experiments: the powerBBK package," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 2(2), pages 157-168, November.
    4. Brian Albert Monroe, 2020. "The statistical power of individual-level risk preference estimation," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 6(2), pages 168-188, December.
    5. Brendon McConnell & Marcos Vera-Hernandez, 2015. "Going beyond simple sample size calculations: a practitioner's guide," IFS Working Papers W15/17, Institute for Fiscal Studies.
    6. Zacharias Maniadis & Fabio Tufano & John A. List, 2017. "To Replicate or Not To Replicate? Exploring Reproducibility in Economics through the Lens of a Model and a Pilot Study," Economic Journal, Royal Economic Society, vol. 127(605), pages 209-235, October.
    7. Alain Cohn & Ernst Fehr & Lorenz Goette, 2015. "Fair Wages and Effort Provision: Combining Evidence from a Choice Experiment and a Field Experiment," Management Science, INFORMS, vol. 61(8), pages 1777-1794, August.
    8. Kathryn N. Vasilaky & J. Michelle Brock, 2020. "Power(ful) guidelines for experimental economists," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 6(2), pages 189-212, December.
    9. da Silveira, Jaylson Jair & Lima, Gilberto Tadeu, 2021. "Wage inequality as a source of endogenous macroeconomic fluctuations," Structural Change and Economic Dynamics, Elsevier, vol. 56(C), pages 35-52.
    10. Faralla, Valeria & Borà, Guido & Innocenti, Alessandro & Novarese, Marco, 2020. "Promises in group decision making," Research in Economics, Elsevier, vol. 74(1), pages 1-11.
    11. Omar Al-Ubaydli & John A. List, 2019. "How natural field experiments have enhanced our understanding of unemployment," Nature Human Behaviour, Nature, vol. 3(1), pages 33-39, January.
    12. ITO Koichiro & IDA Takanori & TANAKA Makoto, 2015. "The Persistence of Moral Suasion and Economic Incentives: Field experimental evidence from energy demand," Discussion papers 15014, Research Institute of Economy, Trade and Industry (RIETI).
    13. Bartoš, Vojtěch, 2021. "Seasonal scarcity and sharing norms," Journal of Economic Behavior & Organization, Elsevier, vol. 185(C), pages 303-316.
    14. Pedro Carneiro & Sokbae Lee & Daniel Wilhelm, 2020. "Optimal data collection for randomized control trials," The Econometrics Journal, Royal Economic Society, vol. 23(1), pages 1-31.
    15. Greiner, Ben & Ockenfels, Axel & Werner, Peter, 2011. "Wage transparency and performance: A real-effort experiment," Economics Letters, Elsevier, vol. 111(3), pages 236-238, June.
    16. Maurizio Canavari & Andreas C. Drichoutis & Jayson L. Lusk & Rodolfo M. Nayga, Jr., 2018. "How to run an experimental auction: A review of recent advances," Working Papers 2018-5, Agricultural University of Athens, Department Of Agricultural Economics.
    17. Brit Grosskopf & Graeme Pearce, 2020. "Do You Mind Me Paying Less? Measuring Other-Regarding Preferences in the Market for Taxis," Management Science, INFORMS, vol. 66(11), pages 5059-5074, November.
    18. Constança Esteves-Sorenson, 2018. "Gift Exchange in the Workplace: Addressing the Conflicting Evidence with a Careful Test," Management Science, INFORMS, vol. 64(9), pages 4365-4388, September.
    19. Gerard J. Van den Berg & Iris Kesternich & Gerrit Müller & Bettina Siflinger, 2019. "Reciprocity and the Interaction between the Unemployed and the Caseworker," CESifo Working Paper Series 7947, CESifo.
    20. Sautua, Santiago I., 2023. "Disentangling the influences of positive reciprocity and mood on gift exchange at work," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 102(C).

    More about this item

    Keywords

    within-subjects design; between-subjects design; sample sizes; statistical power; experiments;
    All these keywords.

    JEL classification:

    • C8 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs
    • C9 - Mathematical and Quantitative Methods - - Design of Experiments
    • D03 - Microeconomics - - General - - - Behavioral Microeconomics: Underlying Principles

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:iza:izadps:dp8583. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Holger Hinte (email available below). General contact details of provider: https://edirc.repec.org/data/izaaade.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.