IDEAS home Printed from https://ideas.repec.org/p/ehl/lserod/67498.html
   My bibliography  Save this paper

Cluster sampling bias in government-sponsored evaluations: a correlational study of employment and welfare pilots in England

Author

Listed:
  • Vaganay, Arnaud

Abstract

For pilot or experimental employment programme results to apply beyond their test bed, researchers must select ‘clusters’ (i.e. the job centres delivering the new intervention) that are reasonably representative of the whole territory. More specifically, this requirement must account for conditions that could artificially inflate the effect of a programme, such as the fluidity of the local labour market or the performance of the local job centre. Failure to achieve representativeness results in Cluster Sampling Bias (CSB). This paper makes three contributions to the literature. Theoretically, it approaches the notion of CSB as a human behaviour. It offers a comprehensive theory, whereby researchers with limited resources and conflicting priorities tend to oversample ‘effect-enhancing’ clusters when piloting a new intervention. Methodologically, it advocates for a ‘narrow and deep’ scope, as opposed to the ‘wide and shallow’ scope, which has prevailed so far. The PILOT-2 dataset was developed to test this idea. Empirically, it provides evidence on the prevalence of CSB. In conditions similar to the PILOT-2 case study, investigators (1) do not sample clusters with a view to maximise generalisability; (2) do not oversample ‘effect-enhancing’ clusters; (3) consistently oversample some clusters, including those with higher-than-average client caseloads; and (4) report their sampling decisions in an inconsistent and generally poor manner. In conclusion, although CSB is prevalent, it is still unclear whether it is intentional and meant to mislead stakeholders about the expected effect of the intervention or due to higher-level constraints or other considerations.

Suggested Citation

  • Vaganay, Arnaud, 2016. "Cluster sampling bias in government-sponsored evaluations: a correlational study of employment and welfare pilots in England," LSE Research Online Documents on Economics 67498, London School of Economics and Political Science, LSE Library.
  • Handle: RePEc:ehl:lserod:67498
    as

    Download full text from publisher

    File URL: http://eprints.lse.ac.uk/67498/
    File Function: Open access version.
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Robert B. Olsen & Larry L. Orr & Stephen H. Bell & Elizabeth A. Stuart, 2013. "External Validity in Policy Evaluations That Choose Sites Purposively," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 32(1), pages 107-121, January.
    2. Philip Gleason & Melissa Clark & Christina Clark Tuttle & Emily Dwoyer, "undated". "The Evaluation of Charter School Impacts," Mathematica Policy Research Reports 3066da11915a4b04a77b38848, Mathematica Policy Research.
    3. repec:mpr:mprres:6676 is not listed on IDEAS
    4. repec:mpr:mprres:7293 is not listed on IDEAS
    5. repec:mpr:mprres:6720 is not listed on IDEAS
    6. Philip Gleason & Melissa Clark & Christina Clark Tuttle & Emily Dwoyer, 2010. "The Evaluation of Charter School Impacts (Presentation)," Mathematica Policy Research Reports 770e250b2ef343a3b1ec8c932, Mathematica Policy Research.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Peter Z. Schochet, "undated". "Statistical Theory for the RCT-YES Software: Design-Based Causal Inference for RCTs," Mathematica Policy Research Reports a0c005c003c242308a92c02dc, Mathematica Policy Research.
    2. Susan Dynarski & Daniel Hubbard & Brian Jacob & Silvia Robles, 2018. "Estimating the Effects of a Large For-Profit Charter School Operator," NBER Working Papers 24428, National Bureau of Economic Research, Inc.
    3. Eyles, Andrew & Machin, Stephen & McNally, Sandra, 2017. "Unexpected school reform: Academisation of primary schools in England," Journal of Public Economics, Elsevier, vol. 155(C), pages 108-121.
    4. Carlson, Deven & Lavertu, Stéphane, 2016. "Charter school closure and student achievement: Evidence from Ohio," Journal of Urban Economics, Elsevier, vol. 95(C), pages 31-48.
    5. Christina Clark Tuttle & Philip Gleason & Virginia Knechtel & Ira Nichols-Barrer & Kevin Booker & Gregory Chojnacki & Thomas Coen & Lisbeth Goble, "undated". "Understanding the Effect of KIPP as it Scales: Volume I, Impacts on Achievement and Other Outcomes," Mathematica Policy Research Reports 7d8e94c5e77a4a9c8bf09000d, Mathematica Policy Research.
    6. Ron Zimmer & Brian Gill & Jonathon Attridge & Kaitlin Obenauf, 2014. "Charter School Authorizers and Student Achievement," Education Finance and Policy, MIT Press, vol. 9(1), pages 59-85, January.
    7. Vilsa E. Curto & Roland G. Fryer Jr., 2014. "The Potential of Urban Boarding Schools for the Poor: Evidence from SEED," Journal of Labor Economics, University of Chicago Press, vol. 32(1), pages 65-93.
    8. Roland G. Fryer, Jr, 2016. "The Production of Human Capital in Developed Countries: Evidence from 196 Randomized Field Experiments," NBER Working Papers 22130, National Bureau of Economic Research, Inc.
    9. Lisa Dragoset & Susanne James-Burdumy & Kristin Hallgren & Irma Perez-Johnson & Mariesa Herrmann & Christina Tuttle & Megan Hague Angus & Rebecca Herman & Matthew Murray & Courtney Tanenbaum & Cheryl , 2015. "Usage of Practices Promoted by School Improvement Grants," Mathematica Policy Research Reports 8e99f01663504ef5b9f8357f6, Mathematica Policy Research.
    10. Spiro Maroulis, 2016. "Interpreting School Choice Treatment Effects: Results and Implications from Computational Experiments," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 19(1), pages 1-7.
    11. Joe Regan-Stansfield, 2016. "Do good primary schools perform even better as academies?," Working Papers 141167564, Lancaster University Management School, Economics Department.
    12. Matthew Johnson & Alicia Demers & Cleo Jacobs Johnson & Claudia Gentile, "undated". "Ewing Marion Kauffman School Evaluation Impact Report Year 4," Mathematica Policy Research Reports 5175e15711054006b860f263e, Mathematica Policy Research.
    13. Matthew Johnson & Eric Lundquist & Cleo Jacobs Johnson & Claudia Gentile, "undated". "Kauffman School Evaluation Long-Term Outcomes Report: Year 2," Mathematica Policy Research Reports 08feceb41e2a451e88fd9b926, Mathematica Policy Research.
    14. Joshua Furgeson & Moira McCullough & Clare Wolfendale & Brian Gill, "undated". "The Equity Project Charter School: Impacts on Student Achievement," Mathematica Policy Research Reports 9ed165ddb03646a496128da4d, Mathematica Policy Research.
    15. Sarah R. Cohodes, 2016. "Teaching to the Student: Charter School Effectiveness in Spite of Perverse Incentives," Education Finance and Policy, MIT Press, vol. 11(1), pages 1-42, Winter.
    16. Joshua M. Cowen & Marcus A. Winters, 2013. "Do Charters Retain Teachers Differently? Evidence from Elementary Schools in Florida," Education Finance and Policy, MIT Press, vol. 8(1), pages 14-42, January.
    17. Virginia Knechtel & Thomas Coen & Pia Caronongan & Nickie Fung & Lisbeth Goble, "undated". "Pre-Kindergarten Impacts Over Time: An Analysis of KIPP Charter Schools," Mathematica Policy Research Reports 252c4c955f4c48d699d7acc89, Mathematica Policy Research.
    18. Fortson, Kenneth & Gleason, Philip & Kopa, Emma & Verbitsky-Savitz, Natalya, 2015. "Horseshoes, hand grenades, and treatment effects? Reassessing whether nonexperimental estimators are biased," Economics of Education Review, Elsevier, vol. 44(C), pages 100-113.
    19. Lisa Dragoset & Jaime Thomas & Mariesa Herrmann & John Deke & Susanne James-Burdumy & Cheryl Graczewski & Andrea Boyle & Rachel Upton & Courtney Tanenbaum & Jessica Giffin, "undated". "School Improvement Grants: Implementation and Effectiveness (Final Report)," Mathematica Policy Research Reports 76bce3f4bb0944f29a481fae0, Mathematica Policy Research.
    20. Matthew Johnson & Alicia Demers, "undated". "Ewing Marion Kauffman School Year 6 Impacts," Mathematica Policy Research Reports bd1f54f237124b218114062d5, Mathematica Policy Research.

    More about this item

    JEL classification:

    • R14 - Urban, Rural, Regional, Real Estate, and Transportation Economics - - General Regional Economics - - - Land Use Patterns
    • J01 - Labor and Demographic Economics - - General - - - Labor Economics: General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ehl:lserod:67498. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: LSERO Manager (email available below). General contact details of provider: https://edirc.repec.org/data/lsepsuk.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.