IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v24y1999i1p42-69.html
   My bibliography  Save this article

Controlling Error in Multiple Comparisons, with Examples from State-to-State Differences in Educational Achievement

Author

Listed:
  • Valerie S. L. Williams
  • Lyle V. Jones
  • John W. Tukey

Abstract

Three alternative procedures to adjust significance levels for multiplicity are the traditional Bonferroni technique, a sequential Bonferroni technique developed by Hochberg (1988) , and a sequential approach for controlling the false discovery rate proposed by Benjamini and Hochberg (1995). These procedures are illustrated and compared using examples from the National Assessment of Educational Progress (NAEP). A prominent advantage of the Benjamini and Hochberg (B-H) procedure, as demonstrated in these examples, is the greater invariance of statistical significance for given comparisons over alternative family sizes. Simulation studies show that all three procedures maintain a false discovery rate bounded above, often grossly, by α (or α/2). For both uncorrelated and pairwise families of comparisons, the B-H technique is shown to have greater power than the Hochberg or Bonferroni procedures, and its power remains relatively stable as the number of comparisons becomes large, giving it an increasing advantage when many comparisons are involved. We recommend that results from NAEP State Assessments be reported using the B-H technique rather than the Bonferroni procedure.

Suggested Citation

  • Valerie S. L. Williams & Lyle V. Jones & John W. Tukey, 1999. "Controlling Error in Multiple Comparisons, with Examples from State-to-State Differences in Educational Achievement," Journal of Educational and Behavioral Statistics, , vol. 24(1), pages 42-69, March.
    • Handle: RePEc:sae:jedbes:v:24:y:1999:i:1:p:42-69
    DOI: 10.3102/10769986024001042
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/10769986024001042
    Download Restriction: no
    File URL: https://libkey.io/10.3102/10769986024001042?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. David Afshartous & Michael Wolf, 2007. "Avoiding ‘data snooping’ in multilevel and mixed effects models," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 170(4), pages 1035-1059, October.
    2. Omotilewa, Oluwatoba J. & Ricker-Gilbert, Jacob & Ainembabazi, John Herbert & Shively, Gerald E., 2018. "Does improved storage technology promote modern input use and food security? Evidence from a randomized trial in Uganda," Journal of Development Economics, Elsevier, vol. 135(C), pages 176-198.
    3. Tangney, June P. & Folk, Johanna B. & Graham, David M. & Stuewig, Jeffrey B. & Blalock, Daniel V. & Salatino, Andrew & Blasko, Brandy L. & Moore, Kelly E., 2016. "Changes in inmates' substance use and dependence from pre-incarceration to one year post-release," Journal of Criminal Justice, Elsevier, vol. 46(C), pages 228-238.
    4. A. Stewart Fotheringham & Taylor M. Oshan, 2016. "Geographically weighted regression and multicollinearity: dispelling the myth," Journal of Geographical Systems, Springer, vol. 18(4), pages 303-329, October.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:24:y:1999:i:1:p:42-69. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.