IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/26389.html
   My bibliography  Save this paper

Design and Analysis of Cluster-Randomized Field Experiments in Panel Data Settings

Author

Listed:
  • Bharat K. Chandar
  • Ali Hortaçsu
  • John A. List
  • Ian Muir
  • Jeffrey M. Wooldridge

Abstract

Field experiments conducted with the village, city, state, region, or even country as the unit of randomization are becoming commonplace in the social sciences. While convenient, subsequent data analysis may be complicated by the constraint on the number of clusters in treatment and control. Through a battery of Monte Carlo simulations, we examine best practices for estimating unit-level treatment effects in cluster-randomized field experiments, particularly in settings that generate short panel data. In most settings we consider, unit-level estimation with unit fixed effects and cluster-level estimation weighted by the number of units per cluster tend to be robust to potentially problematic features in the data while giving greater statistical power. Using insights from our analysis, we evaluate the effect of a unique field experiment: a nationwide tipping field experiment across markets on the Uber app. Beyond the import of showing how tipping affects aggregate market outcomes, we provide several insights on aspects of generating and analyzing cluster-randomized experimental data when there are constraints on the number of experimental units in treatment and control.

Suggested Citation

  • Bharat K. Chandar & Ali Hortaçsu & John A. List & Ian Muir & Jeffrey M. Wooldridge, 2019. "Design and Analysis of Cluster-Randomized Field Experiments in Panel Data Settings," NBER Working Papers 26389, National Bureau of Economic Research, Inc.
    • Handle: RePEc:nbr:nberwo:26389
    Note: IO PE
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w26389.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Conlin, Michael & Lynn, Michael & O'Donoghue, Ted, 2003. "The norm of restaurant tipping," Journal of Economic Behavior & Organization, Elsevier, vol. 52(3), pages 297-321, November.
    2. Bharat Chandar & Uri Gneezy & John List & Ian Muir, 2019. "The Drivers of Social Preferences: Evidence from a Nationwide Tipping Field Experiment," Natural Field Experiments 00680, The Field Experiments Website.
    3. A. Colin Cameron & Douglas L. Miller, 2015. "A Practitioner’s Guide to Cluster-Robust Inference," Journal of Human Resources, University of Wisconsin Press, vol. 50(2), pages 317-372.
    4. Susan Athey & Guido Imbens, 2016. "The Econometrics of Randomized Experiments," Papers 1607.00698, arXiv.org.
    5. Hansen, Christian B., 2007. "Generalized least squares inference in panel and multilevel models with serial correlation and fixed effects," Journal of Econometrics, Elsevier, vol. 140(2), pages 670-694, October.
    6. Alberto Abadie & Susan Athey & Guido W. Imbens & Jeffrey M. Wooldridge, 2014. "Finite Population Causal Standard Errors," NBER Working Papers 20325, National Bureau of Economic Research, Inc.
    7. Cody Cook & Rebecca Diamond & Jonathan V Hall & John A List & Paul Oyer, 2021. "The Gender Earnings Gap in the Gig Economy: Evidence from over a Million Rideshare Drivers [Measuring the Gig Economy: Current Knowledge and Open Issues]," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 88(5), pages 2210-2238.
    8. Moulton, Brent R, 1990. "An Illustration of a Pitfall in Estimating the Effects of Aggregate Variables on Micro Unit," The Review of Economics and Statistics, MIT Press, vol. 72(2), pages 334-338, May.
    9. Lynn, Michael, 2016. "Why are we more likely to tip some service occupations than others? Theory, evidence, and implications," Journal of Economic Psychology, Elsevier, vol. 54(C), pages 134-150.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Basil Halperin & Benjamin Ho & John A List & Ian Muir, 2022. "Toward An Understanding of the Economics of Apologies: Evidence from a Large-Scale Natural Field Experiment," The Economic Journal, Royal Economic Society, vol. 132(641), pages 273-298.
    2. Bharat Chandar & Uri Gneezy & John A. List & Ian Muir, 2019. "The Drivers of Social Preferences: Evidence from a Nationwide Tipping Field Experiment," NBER Working Papers 26380, National Bureau of Economic Research, Inc.
    3. Cody Cook & Rebecca Diamond & Jonathan V Hall & John A List & Paul Oyer, 2021. "The Gender Earnings Gap in the Gig Economy: Evidence from over a Million Rideshare Drivers [Measuring the Gig Economy: Current Knowledge and Open Issues]," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 88(5), pages 2210-2238.
    4. Jason J Sandvik & Richard E Saouma & Nathan T Seegert & Christopher T Stanton, 2020. "Workplace Knowledge Flows," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 135(3), pages 1635-1680.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Bharat Chandar & Uri Gneezy & John A. List & Ian Muir, 2019. "The Drivers of Social Preferences: Evidence from a Nationwide Tipping Field Experiment," NBER Working Papers 26380, National Bureau of Economic Research, Inc.
    2. Frank, David G. & Lynn, Michael, 2020. "Shattering the Illusion of the Self-Earned Tip: The Effect of a Restaurant Magician on Co-Workers’ Tips," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 87(C).
    3. Cross, Jeffrey & Zhang, Guangli, 2024. "Focal points for giving," Journal of Economic Psychology, Elsevier, vol. 102(C).
    4. Michael Pollmann, 2020. "Causal Inference for Spatial Treatments," Papers 2011.00373, arXiv.org, revised Jan 2023.
    5. Hagemann, Andreas, 2019. "Placebo inference on treatment effects when the number of clusters is small," Journal of Econometrics, Elsevier, vol. 213(1), pages 190-209.
    6. Michele Campolieti & Chris Riddell, 2020. "Does Mediation-Arbitration Reduce Arbitration Rates? Evidence from a Natural Experiment," ILR Review, Cornell University, ILR School, vol. 73(1), pages 211-235, January.
    7. Kunze, Lars & Suppa, Nicolai, 2017. "Bowling alone or bowling at all? The effect of unemployment on social participation," Journal of Economic Behavior & Organization, Elsevier, vol. 133(C), pages 213-235.
    8. Hirschauer, Norbert & Grüner, Sven & Mußhoff, Oliver & Becker, Claudia & Jantsch, Antje, 2020. "Can p-values be meaningfully interpreted without random sampling?," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 14, pages 71-91.
    9. Laurent Davezies & Xavier D'Haultfoeuille & Yannick Guyonvarch, 2018. "Asymptotic results under multiway clustering," Papers 1807.07925, arXiv.org, revised Aug 2018.
    10. Yijuan Chen & Juergen Meinecke, 2012. "Do Healthcare Report Cards Cause Providers To Select Patients And Raise Quality Of Care?," Health Economics, John Wiley & Sons, Ltd., vol. 21(S1), pages 33-55, June.
    11. Alberto Abadie & Susan Athey & Guido W Imbens & Jeffrey M Wooldridge, 2023. "When Should You Adjust Standard Errors for Clustering?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 138(1), pages 1-35.
    12. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    13. Dale Belman & Paul Wolfson & Kritkorn Nawakitphaitoon, 2015. "Who Is Affected by the Minimum Wage?," Industrial Relations: A Journal of Economy and Society, Wiley Blackwell, vol. 54(4), pages 582-621, October.
    14. Hansen, Bruce E. & Lee, Seojeong, 2019. "Asymptotic theory for clustered samples," Journal of Econometrics, Elsevier, vol. 210(2), pages 268-290.
    15. Katja Görlitz & Sylvi Rzepka, 2017. "Regional training supply and employees’ training participation," The Annals of Regional Science, Springer;Western Regional Science Association, vol. 59(1), pages 281-296, July.
    16. Chenhong Peng & Lue Fang & Julia Shu-Huah Wang & Yik Wa Law & Yi Zhang & Paul S. F. Yip, 2019. "Determinants of Poverty and Their Variation Across the Poverty Spectrum: Evidence from Hong Kong, a High-Income Society with a High Poverty Level," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 144(1), pages 219-250, July.
    17. Bürker, Matthias & Mammi, Irene & Minerva, G. Alfredo, 2021. "Civic capital and service outsourcing: Evidence from Italy," European Economic Review, Elsevier, vol. 138(C).
    18. Pereira, João & Ramos, Raul & Martins, Pedro S., 2024. "Wage cyclicality and labour market institutions," GLO Discussion Paper Series 1469, Global Labor Organization (GLO).
    19. Vikström, Johan, 2009. "Cluster sample inference using sensitivity analysis: the case with few groups," Working Paper Series 2009:15, IFAU - Institute for Evaluation of Labour Market and Education Policy.
    20. Dick Durevall & Annika Lindskog, 2016. "Adult Mortality, AIDS, and Fertility in Rural Malawi," The Developing Economies, Institute of Developing Economies, vol. 54(3), pages 215-242, September.

    More about this item

    JEL classification:

    • C23 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Models with Panel Data; Spatio-temporal Models
    • C33 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Models with Panel Data; Spatio-temporal Models
    • C5 - Mathematical and Quantitative Methods - - Econometric Modeling
    • C9 - Mathematical and Quantitative Methods - - Design of Experiments
    • C91 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Individual Behavior
    • C92 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Group Behavior
    • C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments
    • D47 - Microeconomics - - Market Structure, Pricing, and Design - - - Market Design

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:26389. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.