IDEAS home Printed from https://ideas.repec.org/p/zbw/i4rdps/136.html
   My bibliography  Save this paper

Manipulation Tests in Regression Discontinuity Design: The Need for Equivalence Testing

Author

Listed:
  • Fitzgerald, Jack

Abstract

Researchers utilizing regression discontinuity design (RDD) commonly test for running variable (RV) manipulation around a cutoff, but incorrectly assert that insignificant manipulation test statistics are evidence of negligible manipulation. I introduce simple frequentist equivalence testing procedures that can provide statistically significant evidence that RV manipulation around a cutoff is practically equal to zero. I then demonstrate the necessity of these procedures, leveraging replication data from 36 RDD publications to conduct 45 equivalence-based RV manipulation tests. Over 44% of RV density discontinuities at the cutoff cannot be significantly bounded beneath a 50% upward jump. Bounding equivalence-based manipulation test failure rates beneath 5% requires arguing that a 350% upward density jump is practically equal to zero. Meta-analytic estimates reveal that average RV manipulation around the cutoff is equivalent to a 26% upward density jump. These results imply that many published RDD estimates may be confounded by discontinuities in potential outcomes due to RV manipulation that remains undetectable by existing tests. I provide research guidelines and commands in Stata and R to help researchers conduct more credible equivalencebased manipulation testing in future RDD research.

Suggested Citation

  • Fitzgerald, Jack, 2024. "Manipulation Tests in Regression Discontinuity Design: The Need for Equivalence Testing," I4R Discussion Paper Series 136, The Institute for Replication (I4R).
    • Handle: RePEc:zbw:i4rdps:136
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/300277/1/I4R-DP136.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. François Gerard & Miikka Rokkanen & Christoph Rothe, 2020. "Bounds on treatment effects in regression discontinuity designs with a manipulated running variable," Quantitative Economics, Econometric Society, vol. 11(3), pages 839-870, July.
    2. Erin Hartman & F. Daniel Hidalgo, 2018. "An Equivalence Approach to Balance and Placebo Tests," American Journal of Political Science, John Wiley & Sons, vol. 62(4), pages 1000-1013, October.
    3. Jun Ma & Hugo Jales & Zhengfei Yu, 2020. "Minimum Contrast Empirical Likelihood Inference of Discontinuity in Density," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 38(4), pages 934-950, October.
    4. Stommes, Drew & Aronow, P. M. & Sävje, Fredrik, 2023. "On the Reliability of Published Findings Using the Regression Discontinuity Design in Political Science," I4R Discussion Paper Series 22, The Institute for Replication (I4R).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Matias D. Cattaneo & Rocío Titiunik, 2022. "Regression Discontinuity Designs," Annual Review of Economics, Annual Reviews, vol. 14(1), pages 821-851, August.
    2. Behrens, Christiaan & de Jong, Gerben & van Ommeren, Jos, 2024. "From silver to platinum: The impact of frequent flier tier levels on air travellers’ behaviour," Transportation Research Part B: Methodological, Elsevier, vol. 186(C).
    3. Ivan A Canay & Vishal Kamat, 2018. "Approximate Permutation Tests and Induced Order Statistics in the Regression Discontinuity Design," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 85(3), pages 1577-1608.
    4. Omesh Kini & Mo Shen & Jaideep Shenoy & Venkat Subramaniam, 2022. "Labor Unions and Product Quality Failures," Management Science, INFORMS, vol. 68(7), pages 5403-5440, July.
    5. Andres Gonzalez-Lira & Michael S. Walker & Rodrigo Carril, 2022. "Competition under Incomplete Contracts and the Design of Procurement Policies," Working Papers 1327, Barcelona School of Economics.
    6. Wu, Guoyong & Gao, Yue & Feng, Yanchao, 2023. "Assessing the environmental effects of the supporting policies for mineral resource-exhausted cities in China," Resources Policy, Elsevier, vol. 85(PB).
    7. repec:irs:cepswp:2024-01 is not listed on IDEAS
    8. Guo, Chuanyi & Wang, Xuening & Meng, Chen, 2023. "Does the early bird catch the worm? Evidence and interpretation on the long-term impact of school entry age in China," China Economic Review, Elsevier, vol. 77(C).
    9. Takuya Ishihara & Masayuki Sawada, 2020. "Manipulation-Robust Regression Discontinuity Designs," Papers 2009.07551, arXiv.org, revised Sep 2024.
    10. Carozzi, Felipe & Cipullo, Davide & Repetto, Luca, 2024. "Powers that be? Political alignment, government formation, and government stability," Journal of Public Economics, Elsevier, vol. 230(C).
    11. Matteo Bobba & Tim Ederer & Gianmarco León-Ciliotta & Christopher A. Neilson & Marco Nieddu, 2021. "Teacher compensation and structural inequality: Evidence from centralized teacher school choice in Perú," Economics Working Papers 1788, Department of Economics and Business, Universitat Pompeu Fabra.
    12. Johannes Spinnewijn, 2020. "The Trade‐Off between Insurance and Incentives in Differentiated Unemployment Policies," Fiscal Studies, John Wiley & Sons, vol. 41(1), pages 101-127, March.
    13. Reif, Simon & Wichert, Sebastian & Wuppermann, Amelie, 2018. "Is it good to be too light? Birth weight thresholds in hospital reimbursement systems," Journal of Health Economics, Elsevier, vol. 59(C), pages 1-25.
    14. Mr. Alberto Behar & Ramin Hassan, 2022. "The Current Account Income Balance: External Adjustment Channel or Vulnerability Amplifier?," IMF Working Papers 2022/106, International Monetary Fund.
    15. Cheng, Lingguo & Lu, Yunfeng, 2024. "Does retirement make people more risk averse?," Journal of Economic Behavior & Organization, Elsevier, vol. 224(C), pages 135-155.
    16. François Gerard & Miikka Rokkanen & Christoph Rothe, 2020. "Bounds on treatment effects in regression discontinuity designs with a manipulated running variable," Quantitative Economics, Econometric Society, vol. 11(3), pages 839-870, July.
    17. Gregory Eady & Tom Paskhalis & Jan Zilinsky & Richard Bonneau & Jonathan Nagler & Joshua A. Tucker, 2023. "Exposure to the Russian Internet Research Agency foreign influence campaign on Twitter in the 2016 US election and its relationship to attitudes and voting behavior," Nature Communications, Nature, vol. 14(1), pages 1-11, December.
    18. Weiss, Amanda, 2024. "How Much Should We Trust Modern Difference-in-Differences Estimates?," OSF Preprints bqmws, Center for Open Science.
    19. Judit Krekó, 2019. "Effect of employment tax incentives: the case of disability quota in Hungary," CEU Working Papers 2019_1, Department of Economics, Central European University.
    20. Emmanuel José Vázquez, 2022. "La escolarización y las ganancias de aprendizaje según el nivel de integración socioeconómica de los estudiantes," Asociación Argentina de Economía Política: Working Papers 4606, Asociación Argentina de Economía Política.
    21. Joshua D. Angrist, 2022. "Empirical Strategies in Economics: Illuminating the Path From Cause to Effect," Econometrica, Econometric Society, vol. 90(6), pages 2509-2539, November.

    More about this item

    Keywords

    McCrary density test; rddensity; DCdensity; Hartman test;
    All these keywords.

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General
    • C87 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Econometric Software
    • P00 - Political Economy and Comparative Economic Systems - - General - - - General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:i4rdps:136. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://www.i4replication.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.