IDEAS home Printed from https://ideas.repec.org/a/eee/csdana/v55y2011i11p3041-3048.html
   My bibliography  Save this article

Model-averaged confidence intervals for factorial experiments

Author

Listed:
  • Fletcher, David
  • Dillingham, Peter W.

Abstract

We consider the coverage rate of model-averaged confidence intervals for the treatment means in a factorial experiment, when we use a normal linear model in the analysis. Model-averaging provides a useful compromise between using the full model (containing all main effects and interactions) and a "best model" obtained by some model-selection process. Use of the full model guarantees perfect coverage, whereas use of a best model is known to lead to narrow intervals with poor coverage. Model-averaging allows us to achieve good coverage using intervals that are also narrower than those from the full model. We compare four information criteria that might be used for model-averaging in this setting: AIC, AICc, and BIC. In this setting, if the full model is "truth", all the criteria will have perfect coverage rates asymptotically. We use simulation to assess the coverage rates and interval widths likely to be achieved by a confidence interval with a nominal coverage of 95%. Our results suggest that AIC performs best in terms of coverage rate; across a wide range of scenarios and replication levels, it consistently provides coverage rates within 1.5% points of the nominal level, while also leading to reductions in interval-width of up to 30%, compared to the full model. AICc performed worst overall, with a coverage rate that was up to 5.2% points too low. We recommend that model-averaging become standard practise when summarising the results of a factorial experiment in terms of the treatment means, and that AIC be used to perform the model-averaging.

Suggested Citation

  • Fletcher, David & Dillingham, Peter W., 2011. "Model-averaged confidence intervals for factorial experiments," Computational Statistics & Data Analysis, Elsevier, vol. 55(11), pages 3041-3048, November.
    • Handle: RePEc:eee:csdana:v:55:y:2011:i:11:p:3041-3048
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0167947311001800
    Download Restriction: Full text for ScienceDirect subscribers only.
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Paul Lukacs & Kenneth Burnham & David Anderson, 2010. "Model selection bias and Freedman’s paradox," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 62(1), pages 117-125, February.
    2. Claeskens,Gerda & Hjort,Nils Lid, 2008. "Model Selection and Model Averaging," Cambridge Books, Cambridge University Press, number 9780521852258, October.
    3. Chris Chatfield, 1995. "Model Uncertainty, Data Mining and Statistical Inference," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 158(3), pages 419-444, May.
    4. Hjort N.L. & Claeskens G., 2003. "Frequentist Model Average Estimators," Journal of the American Statistical Association, American Statistical Association, vol. 98, pages 879-899, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Schomaker, Michael & Heumann, Christian, 2014. "Model selection and model averaging after multiple imputation," Computational Statistics & Data Analysis, Elsevier, vol. 71(C), pages 758-770.
    2. Paul Kabaila & A. H. Welsh & Waruni Abeysekera, 2016. "Model-Averaged Confidence Intervals," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 43(1), pages 35-48, March.
    3. Michael Schomaker & Christian Heumann, 2020. "When and when not to use optimal model averaging," Statistical Papers, Springer, vol. 61(5), pages 2221-2240, October.
    4. Turek, Daniel & Fletcher, David, 2012. "Model-averaged Wald confidence intervals," Computational Statistics & Data Analysis, Elsevier, vol. 56(9), pages 2809-2815.
    5. Shaobo Jin & Sebastian Ankargren, 2019. "Frequentist Model Averaging in Structural Equation Modelling," Psychometrika, Springer;The Psychometric Society, vol. 84(1), pages 84-104, March.
    6. Jiaxu Zeng & David Fletcher & Peter W Dillingham & Christopher E Cornwall, 2019. "Studentized bootstrap model-averaged tail area intervals," PLOS ONE, Public Library of Science, vol. 14(3), pages 1-16, March.
    7. Shaobo Jin, 2022. "Frequentist Model Averaging in Structure Equation Model With Ordinal Data," Psychometrika, Springer;The Psychometric Society, vol. 87(3), pages 1130-1145, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tumala, Mohammed M & Olubusoye, Olusanya E & Yaaba, Baba N & Yaya, OlaOluwa S & Akanbi, Olawale B, 2017. "Forecasting Nigerian Inflation using Model Averaging methods: Modelling Frameworks to Central Banks," MPRA Paper 88754, University Library of Munich, Germany, revised Feb 2018.
    2. Jing Zhou & Gerda Claeskens & Jelena Bradic, 2020. "Detangling robustness in high dimensions: composite versus model-averaged estimation," Papers 2006.07457, arXiv.org.
    3. Phillip Heiler & Jana Mareckova, 2019. "Shrinkage for Categorical Regressors," Papers 1901.01898, arXiv.org.
    4. John Copas & Shinto Eguchi, 2020. "Strong model dependence in statistical analysis: goodness of fit is not enough for model choice," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 72(2), pages 329-352, April.
    5. Yuting Wei & Qihua Wang & Wei Liu, 2021. "Model averaging for linear models with responses missing at random," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 73(3), pages 535-553, June.
    6. Katrin Wölfel & Christoph S. Weber, 2017. "Searching for the Fed’s reaction function," Empirical Economics, Springer, vol. 52(1), pages 191-227, February.
    7. Kira Alhorn & Holger Dette & Kirsten Schorning, 2021. "Optimal Designs for Model Averaging in non-nested Models," Sankhya A: The Indian Journal of Statistics, Springer;Indian Statistical Institute, vol. 83(2), pages 745-778, August.
    8. Hjort, Nils Lid & McKeague, Ian W. & Van Keilegom, Ingrid, 2017. "Hybrid combinations of parametric and empirical likelihoods," LIDAM Discussion Papers ISBA 2017021, Université catholique de Louvain, Institute of Statistics, Biostatistics and Actuarial Sciences (ISBA).
    9. Paul Kabaila & A. H. Welsh & Waruni Abeysekera, 2016. "Model-Averaged Confidence Intervals," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 43(1), pages 35-48, March.
    10. Giuseppe De Luca & Jan Magnus & Franco Peracchi, 2022. "Asymptotic properties of the weighted average least squares (WALS) estimator," Tinbergen Institute Discussion Papers 22-022/III, Tinbergen Institute.
    11. Martins, Luis F. & Gabriel, Vasco J., 2014. "Linear instrumental variables model averaging estimation," Computational Statistics & Data Analysis, Elsevier, vol. 71(C), pages 709-724.
    12. Xinyu Zhang & Alan T. K. Wan & Sherry Z. Zhou, 2011. "Focused Information Criteria, Model Selection, and Model Averaging in a Tobit Model With a Nonzero Threshold," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 30(1), pages 132-142, June.
    13. Giuseppe Luca & Jan R. Magnus & Franco Peracchi, 2023. "Weighted-Average Least Squares (WALS): Confidence and Prediction Intervals," Computational Economics, Springer;Society for Computational Economics, vol. 61(4), pages 1637-1664, April.
    14. Raffaella Giacomini & Toru Kitagawa & Alessio Volpicella, 2022. "Uncertain identification," Quantitative Economics, Econometric Society, vol. 13(1), pages 95-123, January.
    15. Jiaxu Zeng & David Fletcher & Peter W Dillingham & Christopher E Cornwall, 2019. "Studentized bootstrap model-averaged tail area intervals," PLOS ONE, Public Library of Science, vol. 14(3), pages 1-16, March.
    16. Romain Duval & Davide Furceri & Jakob Miethe, 2021. "Robust political economy correlates of major product and labor market reforms in advanced economies: Evidence from BAMLE for logit models," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 36(1), pages 98-124, January.
    17. DiTraglia, Francis J., 2016. "Using invalid instruments on purpose: Focused moment selection and averaging for GMM," Journal of Econometrics, Elsevier, vol. 195(2), pages 187-208.
    18. Magnus, Jan R. & Wan, Alan T.K. & Zhang, Xinyu, 2011. "Weighted average least squares estimation with nonspherical disturbances and an application to the Hong Kong housing market," Computational Statistics & Data Analysis, Elsevier, vol. 55(3), pages 1331-1341, March.
    19. Faguang Wen & Jiming Jiang & Yihui Luan, 2024. "Model Selection Path and Construction of Model Confidence Set under High-Dimensional Variables," Mathematics, MDPI, vol. 12(5), pages 1-21, February.
    20. Ruoyao Shi, 2021. "An Averaging Estimator for Two Step M Estimation in Semiparametric Models," Working Papers 202105, University of California at Riverside, Department of Economics.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:csdana:v:55:y:2011:i:11:p:3041-3048. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/csda .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.