IDEAS home Printed from https://ideas.repec.org/a/eee/csdana/v54y2010i10p2296-2305.html
   My bibliography  Save this article

Confidence intervals for dependent data: Equating non-overlap with statistical significance

Author

Listed:
  • Afshartous, David
  • Preston, Richard A.

Abstract

We revisit the problem of determining confidence interval widths for the comparison of means. For the independent two-sample (two-sided) case, Goldstein and Healy (1995) draw attention to the fact that comparisons based on 95% error bars are not very effective in assessing the statistical significance of the difference in means and derive the correct confidence interval for such a comparison. We provide an extension to Goldstein and Healy (1995) to account for the correlation structure and unequal variances. We use the results to develop rules of thumb for evaluating differences, in an exploratory manner, like Moses (1987) and Cumming (2009), from the independent case. We illustrate the method for the simple comparison of two means in a real data set, provide R code that may be easily implemented in practice, and discuss the extension of the method to other applied problems.

Suggested Citation

  • Afshartous, David & Preston, Richard A., 2010. "Confidence intervals for dependent data: Equating non-overlap with statistical significance," Computational Statistics & Data Analysis, Elsevier, vol. 54(10), pages 2296-2305, October.
    • Handle: RePEc:eee:csdana:v:54:y:2010:i:10:p:2296-2305
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0167-9473(10)00156-8
    Download Restriction: Full text for ScienceDirect subscribers only.
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Joseph P. Romano & Michael Wolf, 2005. "Stepwise Multiple Testing as Formalized Data Snooping," Econometrica, Econometric Society, vol. 73(4), pages 1237-1282, July.
    2. David Afshartous & Michael Wolf, 2007. "Avoiding ‘data snooping’ in multilevel and mixed effects models," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 170(4), pages 1035-1059, October.
    3. Schenker N. & Gentleman J. F., 2001. "On Judging the Significance of Differences by Examining the Overlap Between Confidence Intervals," The American Statistician, American Statistical Association, vol. 55, pages 182-186, August.
    4. Unknown, 2007. "FAPRI U.S Baseline Briefing Book," FAPRI-MU Report Series 44776, Food and Agricultural Policy Research Institute (FAPRI).
    5. Harvey Goldstein & Michael J. R. Healy, 1995. "The Graphical Presentation of a Collection of Means," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 158(1), pages 175-177, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Columbu, Silvia & Porcu, Mariano & Sulis, Isabella, 2021. "University choice and the attractiveness of the study area: Insights on the differences amongst degree programmes in Italy based on generalised mixed-effect models," Socio-Economic Planning Sciences, Elsevier, vol. 74(C).
    2. Tim Goedemé & Karel Van den Bosch & Lina Salanauskaite & Gerlinde Verbist, 2013. "Testing the Statistical Significance of Microsimulation Results: A Plea," International Journal of Microsimulation, International Microsimulation Association, vol. 6(3), pages 50-77.
    3. Roman P. Kuster & Wilhelmus J. A. Grooten & Victoria Blom & Daniel Baumgartner & Maria Hagströmer & Örjan Ekblom, 2021. "How Accurate and Precise Can We Measure the Posture and the Energy Expenditure Component of Sedentary Behaviour with One Sensor?," IJERPH, MDPI, vol. 18(11), pages 1-13, May.
    4. Stefan J. Reichelstein & Anshuman Sahoo, 2015. "Cost- and Price Dynamics of Solar PV Modules," CESifo Working Paper Series 5674, CESifo.
    5. Tim Goedemé & Tess Penne & Tine Hufkens & Alexandros Karakitsios & Anikó Bernát & Bori Simonovits & Elena Carillo Alvarez & Eleni Kanavitsa & Irene Cussó Parcerisas & Jordi Riera Romaní & Lauri Mäkine, 2017. "What Does It Mean To Live on the Poverty Threshold? Lessons From Reference Budgets," Working Papers 1707, Herman Deleeck Centre for Social Policy, University of Antwerp.
    6. Verbist, Gerlinde & Goedemé, Tim & Van den Bosch, Karel & Salanauskaite, Lina, 2013. "Testing the statistical significance of microsimulation results: often easier than you think. A technical note," EUROMOD Working Papers EM18/13, EUROMOD at the Institute for Social and Economic Research.
    7. Kimihiro Noguchi & Fernando Marmolejo-Ramos, 2016. "Assessing Equality of Means Using the Overlap of Range-Preserving Confidence Intervals," The American Statistician, Taylor & Francis Journals, vol. 70(4), pages 325-334, October.
    8. Brini, Elisa & Borgen, Solveig Topstad & Borgen, Nicolai T., 2023. "Avoiding the eyeballing fallacy: Visualizing statistical differences between estimates using the pheatplot command," SocArXiv fghcd, Center for Open Science.
    9. Giorgio E. Montanari & Marco Doretti, 2019. "Ranking Nursing Homes’ Performances Through a Latent Markov Model with Fixed and Random Effects," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 146(1), pages 307-326, November.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. George Leckie & Harvey Goldstein, 2009. "The limitations of using school league tables to inform school choice," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 172(4), pages 835-851, October.
    2. David Afshartous & Michael Wolf, 2007. "Avoiding ‘data snooping’ in multilevel and mixed effects models," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 170(4), pages 1035-1059, October.
    3. Tom Tregenza & Petri T Niemelä & Rolando Rodríguez-Muñoz & Paul E Hopwood, 2022. "Environment and mate attractiveness in a wild insect [Female pied flycatchers choose territory quality and not male characteristics]," Behavioral Ecology, International Society for Behavioral Ecology, vol. 33(5), pages 999-1006.
    4. David M. Ritzwoller & Joseph P. Romano, 2019. "Uncertainty in the Hot Hand Fallacy: Detecting Streaky Alternatives to Random Bernoulli Sequences," Papers 1908.01406, arXiv.org, revised Apr 2021.
    5. Bruno Ferman & Cristine Pinto & Vitor Possebom, 2020. "Cherry Picking with Synthetic Controls," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 510-532, March.
    6. Sant’Anna, Pedro H.C. & Zhao, Jun, 2020. "Doubly robust difference-in-differences estimators," Journal of Econometrics, Elsevier, vol. 219(1), pages 101-122.
    7. Orazio Attanasio & Sarah Cattan & Emla Fitzsimons & Costas Meghir & Marta Rubio-Codina, 2020. "Estimating the Production Function for Human Capital: Results from a Randomized Controlled Trial in Colombia," American Economic Review, American Economic Association, vol. 110(1), pages 48-85, January.
    8. Bhalotra, Sonia & Clarke, Damian & Mühlrad, Hanna & Palme, Mårten, 2021. "Health and Labor Market Impacts of Twin Birth : Evidence from a Swedish IVF Policy Mandate," The Warwick Economics Research Paper Series (TWERPS) 1391, University of Warwick, Department of Economics.
    9. Hermes, Henning & Mierisch, Fabian & Peter, Frauke & Wiederhold, Simon & Lergetporer, Philipp, 2023. "Discrimination on the Child Care Market: A Nationwide Field Experiment," IZA Discussion Papers 16082, Institute of Labor Economics (IZA).
    10. Orazio Attanasio & Helen Baker-Henningham & Raquel Bernal & Costas Meghir & Diana Pineda & Marta Rubio-Codina, 2022. "Early Stimulation and Nutrition: The Impacts of a Scalable Intervention," Journal of the European Economic Association, European Economic Association, vol. 20(4), pages 1395-1432.
    11. Pierfrancesco Rolla & Patricia Justino, 2022. "The social consequences of organized crime in Italy," WIDER Working Paper Series wp-2022-106, World Institute for Development Economic Research (UNU-WIDER).
    12. Chicu, Mark & Masten, Matthew A., 2013. "A specification test for discrete choice models," Economics Letters, Elsevier, vol. 121(2), pages 336-339.
    13. Billings, Stephen B. & Johnson, Erik B., 2012. "A non-parametric test for industrial specialization," Journal of Urban Economics, Elsevier, vol. 71(3), pages 312-331.
    14. Hızıroğlu Aygün, Aysun & Kırdar, Murat Güray & Koyuncu, Murat & Stoeffler, Quentin, 2024. "Keeping refugee children in school and out of work: Evidence from the world's largest humanitarian cash transfer program," Journal of Development Economics, Elsevier, vol. 168(C).
    15. Cygan-Rehm, Kamila & Karbownik, Krzysztof, 2022. "The effects of incentivizing early prenatal care on infant health," Journal of Health Economics, Elsevier, vol. 83(C).
    16. Smeekes, Stephan & Taylor, A.M. Robert, 2012. "Bootstrap Union Tests For Unit Roots In The Presence Of Nonstationary Volatility," Econometric Theory, Cambridge University Press, vol. 28(2), pages 422-456, April.
    17. Domínguez-Torreiro, Marcos & Soliño, Mario, 2011. "Provided and perceived status quo in choice experiments: Implications for valuing the outputs of multifunctional rural areas," Ecological Economics, Elsevier, vol. 70(12), pages 2523-2531.
    18. Hanck, Christoph, 2011. "Now, whose schools are really better (or weaker) than Germany's? A multiple testing approach," Economic Modelling, Elsevier, vol. 28(4), pages 1739-1746, July.
    19. Berger, Eva M. & Fehr, Ernst & Hermes, Henning & Schunk, Daniel & Winkel, Kirsten, 2020. "The Impact of Working Memory Training on Children's Cognitive and Noncognitive Skills," IZA Discussion Papers 13338, Institute of Labor Economics (IZA).
    20. Grønborg, Niels S. & Lunde, Asger & Timmermann, Allan & Wermers, Russ, 2021. "Picking funds with confidence," Journal of Financial Economics, Elsevier, vol. 139(1), pages 1-28.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:csdana:v:54:y:2010:i:10:p:2296-2305. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/csda .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.