Assessing Inter-rater Reliability With Heterogeneous Variance Components Models: Flexible Approach Accounting for Contextual Variables
Author
Abstract
Suggested Citation
DOI: 10.3102/10769986221150517
Download full text from publisher
References listed on IDEAS
- Tiago M. Fragoso & Wesley Bertoli & Francisco Louzada, 2018. "Bayesian Model Averaging: A Systematic Review and Conceptual Classification," International Statistical Review, International Statistical Institute, vol. 86(1), pages 1-28, April.
- Goldhaber, Dan & Grout, Cyrus & Wolff, Malcolm & Martinková, Patrícia, 2021. "Evidence on the Dimensionality and Reliability of Professional References’ Ratings of Teacher Applicants," Economics of Education Review, Elsevier, vol. 83(C).
- Patrícia Martinková & Dan Goldhaber & Elena Erosheva, 2018. "Disparities in ratings of internal and external applicants: A case for model-based inter-rater reliability," PLOS ONE, Public Library of Science, vol. 13(10), pages 1-17, October.
- Rüdiger Mutz & Lutz Bornmann & Hans-Dieter Daniel, 2012. "Heterogeneity of Inter-Rater Reliabilities of Grant Peer Reviews and Its Determinants: A General Estimating Equations Approach," PLOS ONE, Public Library of Science, vol. 7(10), pages 1-10, October.
- Jeffrey N. Rouder & Richard D. Morey, 2019. "Teaching Bayes’ Theorem: Strength of Evidence as Predictive Accuracy," The American Statistician, Taylor & Francis Journals, vol. 73(2), pages 186-190, April.
Most related items
These are the items that most often cite the same works as this one and are cited by the same works as this one.- Zhou, W. & O’Neill, E. & Moncaster, A. & Reiner, D. & Guthrie, P., 2019.
"Applying Bayesian Model Averaging to Characterise Urban Residential Stock Turnover Dynamics,"
Cambridge Working Papers in Economics
1986, Faculty of Economics, University of Cambridge.
- Wei Zhou & Eoghan O'Neill & Alice Moncaster & David M Reiner & Peter Guthrie, 2019. "Applying Bayesian Model Averaging to Characterise Urban Residential Stock Turnover Dynamics," Working Papers EPRG1933, Energy Policy Research Group, Cambridge Judge Business School, University of Cambridge.
- Roland Brown & Yingling Fan & Kirti Das & Julian Wolfson, 2021. "Iterated multisource exchangeability models for individualized inference with an application to mobile sensor data," Biometrics, The International Biometric Society, vol. 77(2), pages 401-412, June.
- He, Ni & Yongqiao, Wang & Tao, Jiang & Zhaoyu, Chen, 2022. "Self-Adaptive bagging approach to credit rating," Technological Forecasting and Social Change, Elsevier, vol. 175(C).
- Patrícia Martinková & Dan Goldhaber & Elena Erosheva, 2018. "Disparities in ratings of internal and external applicants: A case for model-based inter-rater reliability," PLOS ONE, Public Library of Science, vol. 13(10), pages 1-17, October.
- Abdul Salam & Marco Grzegorczyk, 2023. "Model averaging for sparse seemingly unrelated regression using Bayesian networks among the errors," Computational Statistics, Springer, vol. 38(2), pages 779-808, June.
- Hyemin Han, 2024. "Bayesian Model Averaging and Regularized Regression as Methods for Data-Driven Model Exploration, with Practical Considerations," Stats, MDPI, vol. 7(3), pages 1-13, July.
- Emanuel Kopp, 2018. "Determinants of U.S. Business Investment," IMF Working Papers 2018/139, International Monetary Fund.
- Liao, Jun & Zou, Guohua, 2020. "Corrected Mallows criterion for model averaging," Computational Statistics & Data Analysis, Elsevier, vol. 144(C).
- Mark F. J. Steel, 2020.
"Model Averaging and Its Use in Economics,"
Journal of Economic Literature, American Economic Association, vol. 58(3), pages 644-719, September.
- Steel, Mark F. J., 2017. "Model Averaging and its Use in Economics," MPRA Paper 81568, University Library of Munich, Germany.
- Steel, Mark F. J., 2017. "Model Averaging and its Use in Economics," MPRA Paper 90110, University Library of Munich, Germany, revised 16 Nov 2018.
- Goldhaber, Dan & Grout, Cyrus & Wolff, Malcolm & Martinková, Patrícia, 2021. "Evidence on the Dimensionality and Reliability of Professional References’ Ratings of Teacher Applicants," Economics of Education Review, Elsevier, vol. 83(C).
- David G Pina & Darko Hren & Ana Marušić, 2015. "Peer Review Evaluation Process of Marie Curie Actions under EU’s Seventh Framework Programme for Research," PLOS ONE, Public Library of Science, vol. 10(6), pages 1-15, June.
- Lutz Bornmann, 2015. "Interrater reliability and convergent validity of F1000Prime peer review," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(12), pages 2415-2426, December.
- Díaz, Juan D. & Hansen, Erwin & Cabrera, Gabriel, 2021. "Economic drivers of commodity volatility: The case of copper," Resources Policy, Elsevier, vol. 73(C).
- Karmelavičius, Jaunius & Mikaliūnaitė-Jouvanceau, Ieva & Petrokaitė, Austėja Petrokaitė, 2022.
"Housing and credit misalignments in a two-market disequilibrium framework,"
ESRB Working Paper Series
135, European Systemic Risk Board.
- Jaunius Karmelavičius & Ieva Mikaliūnaitė-Jouvanceau & Austėja Petrokaitė, 2022. "Housing and credit misalignments in a two-market disequilibrium framework," Bank of Lithuania Occasional Paper Series 42, Bank of Lithuania.
- Berrisch, Jonathan & Ziel, Florian, 2023. "CRPS learning," Journal of Econometrics, Elsevier, vol. 237(2).
- Mihai MUTASCU & Nicolae-Bogdan IANC & ALBERT LESSOUA, 2021.
"Public debt and inequality in Sub-Saharan Africa: the case of EMCCA and WAEMU countries,"
LEO Working Papers / DR LEO
2909, Orleans Economics Laboratory / Laboratoire d'Economie d'Orleans (LEO), University of Orleans.
- Mihai MUTASCU & Nicolae-Bogdan IANC & Albert LESSOUA, 2021. "Public debt and inequality in Sub-Saharan Africa: the case of EMCCA and WAEMU countries," LEO Working Papers / DR LEO 2911, Orleans Economics Laboratory / Laboratoire d'Economie d'Orleans (LEO), University of Orleans.
- Mathyn Vervaart & Eline Aas & Karl P. Claxton & Mark Strong & Nicky J. Welton & Torbjørn Wisløff & Anna Heath, 2023. "General-Purpose Methods for Simulating Survival Data for Expected Value of Sample Information Calculations," Medical Decision Making, , vol. 43(5), pages 595-609, July.
- Ngandu Balekelayi & Solomon Tesfamariam, 2020. "Geoadditive Quantile Regression Model for Sewer Pipes Deterioration Using Boosting Optimization Algorithm," Sustainability, MDPI, vol. 12(20), pages 1-24, October.
- Marcin Błażejowski & Jacek Kwiatkowski & Paweł Kufel, 2020. "BACE and BMA Variable Selection and Forecasting for UK Money Demand and Inflation with Gretl," Econometrics, MDPI, vol. 8(2), pages 1-29, May.
- Dimitris Korobilis & Kenichi Shimizu, 2022.
"Bayesian Approaches to Shrinkage and Sparse Estimation,"
Foundations and Trends(R) in Econometrics, now publishers, vol. 11(4), pages 230-354, June.
- Dimitris Korobilis & Kenichi Shimizu, 2021. "Bayesian Approaches to Shrinkage and Sparse Estimation," Working Papers 2021_19, Business School - Economics, University of Glasgow.
- Dimitris Korobilis & Kenichi Shimizu, 2021. "Bayesian Approaches to Shrinkage and Sparse Estimation," Papers 2112.11751, arXiv.org.
- Dimitris Korobilis & Kenichi Shimizu, 2022. "Bayesian Approaches to Shrinkage and Sparse Estimation," Working Paper series 22-02, Rimini Centre for Economic Analysis.
- Korobilis, Dimitris & Shimizu, Kenichi, 2021. "Bayesian Approaches to Shrinkage and Sparse Estimation," MPRA Paper 111631, University Library of Munich, Germany.
More about this item
Keywords
Bayesian inference; inter-rater reliability; mixed-effect models; heterogeneous variance components; grant peer review;All these keywords.
Statistics
Access and download statisticsCorrections
All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:48:y:2023:i:3:p:349-383. See general information about how to correct material in RePEc.
If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.
If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .
If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .
Please note that corrections may take a couple of weeks to filter through the various RePEc services.