IDEAS home Printed from https://ideas.repec.org/a/eee/csdana/v182y2023ics0167947323000348.html
   My bibliography  Save this article

How to test the missing data mechanism in a hidden Markov model

Author

Listed:
  • Chassan, Malika
  • Concordet, Didier

Abstract

A Hidden Markov Model with missing data in the outcome variable is considered. The initial and transition probabilities of the Markov chain and the emission probability of the HMM are allowed to depend on fully observed covariables. Tests for the ignorable and for the MCAR mechanisms are proposed. These tests do not require grouping the individuals by their missing pattern, making them easier to apply in practice. They are based on the estimates of the conditional probabilities of emitting a missing data given the latent state of the Markov chain and some observed covariables. When the ignorable mechanism holds, the conditional probabilities of emitting a missing value are the same for a given value of the observed variables. On the contrary, when the MCAR mechanism holds, these probabilities are all the same. A practical implementation of these tests based on simulations is proposed, along with a presentation of their performances. A real example from piglet farming illustrates their use.

Suggested Citation

  • Chassan, Malika & Concordet, Didier, 2023. "How to test the missing data mechanism in a hidden Markov model," Computational Statistics & Data Analysis, Elsevier, vol. 182(C).
    • Handle: RePEc:eee:csdana:v:182:y:2023:i:c:s0167947323000348
    DOI: 10.1016/j.csda.2023.107723
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0167947323000348
    Download Restriction: Full text for ScienceDirect subscribers only.
    File URL: https://libkey.io/10.1016/j.csda.2023.107723?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Christoph Breunig, 2019. "Testing Missing at Random Using Instrumental Variables," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 37(2), pages 223-234, April.
    2. Mortaza Jamshidian & Siavash Jalal, 2010. "Tests of Homoscedasticity, Normality, and Missing Completely at Random for Incomplete Multivariate Data," Psychometrika, Springer;The Psychometric Society, vol. 75(4), pages 649-674, December.
    3. Jamshidian, Mortaza & Jalal, Siavash & Jansen, Camden, 2014. "MissMech: An R Package for Testing Homoscedasticity, Multivariate Normality, and Missing Completely at Random (MCAR)," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 56(i06).
    4. Kevin Kim & Peter Bentler, 2002. "Tests of homogeneity of means and covariance matrices for multivariate incomplete data," Psychometrika, Springer;The Psychometric Society, vol. 67(4), pages 609-623, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ke-Hai Yuan & Mortaza Jamshidian & Yutaka Kano, 2018. "Missing Data Mechanisms and Homogeneity of Means and Variances–Covariances," Psychometrika, Springer;The Psychometric Society, vol. 83(2), pages 425-442, June.
    2. Hairu Wang & Zhiping Lu & Yukun Liu, 2023. "Score test for missing at random or not under logistic missingness models," Biometrics, The International Biometric Society, vol. 79(2), pages 1268-1279, June.
    3. Jun Li & Yao Yu, 2015. "A Nonparametric Test of Missing Completely at Random for Incomplete Multivariate Data," Psychometrika, Springer;The Psychometric Society, vol. 80(3), pages 707-726, September.
    4. Jamshidian, Mortaza & Jalal, Siavash & Jansen, Camden, 2014. "MissMech: An R Package for Testing Homoscedasticity, Multivariate Normality, and Missing Completely at Random (MCAR)," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 56(i06).
    5. Frahm, Gabriel & Nordhausen, Klaus & Oja, Hannu, 2020. "M-estimation with incomplete and dependent multivariate data," Journal of Multivariate Analysis, Elsevier, vol. 176(C).
    6. Raquel Lourenço Carvalhal Monteiro & Valdecy Pereira & Helder Gomes Costa, 2020. "Dependence Analysis Between Childhood Social Indicators and Human Development Index Through Canonical Correlation Analysis," Child Indicators Research, Springer;The International Society of Child Indicators (ISCI), vol. 13(1), pages 337-362, February.
    7. Forthmann, Boris & Jendryczko, David & Scharfen, Jana & Kleinkorres, Ruben & Benedek, Mathias & Holling, Heinz, 2019. "Creative ideation, broad retrieval ability, and processing speed: A confirmatory study of nested cognitive abilities," Intelligence, Elsevier, vol. 75(C), pages 59-72.
    8. Nobumichi Shutoh & Takahiro Nishiyama & Masashi Hyodo, 2017. "Bartlett correction to the likelihood ratio test for MCAR with two-step monotone sample," Statistica Neerlandica, Netherlands Society for Statistics and Operations Research, vol. 71(3), pages 184-199, August.
    9. Ying Fang & Ming Lin & Shengfang Tang & Zongwu Cai, 2021. "Testing Conditional Independence in Macroeconomic Policy Evaluation for Time Series Data," WORKING PAPERS SERIES IN THEORETICAL AND APPLIED ECONOMICS 202118, University of Kansas, Department of Economics, revised Sep 2021.
    10. Wei Liu & Zhiwei Zhang & Lei Nie & Guoxing Soon, 2017. "A Case Study in Personalized Medicine: Rilpivirine Versus Efavirenz for Treatment-Naive HIV Patients," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(520), pages 1381-1392, October.
    11. Boris Forthmann & Mark A. Runco, 2020. "An Empirical Test of the Inter-Relationships between Various Bibliometric Creative Scholarship Indicators," Publications, MDPI, vol. 8(2), pages 1-16, June.
    12. Fang, Ying & Tang, Shengfang & Cai, Zongwu & Lin, Ming, 2020. "An alternative test for conditional unconfoundedness using auxiliary variables," Economics Letters, Elsevier, vol. 194(C).
    13. Yuan, Ke-Hai, 2009. "Normal distribution based pseudo ML for missing data: With applications to mean and covariance structure analysis," Journal of Multivariate Analysis, Elsevier, vol. 100(9), pages 1900-1918, October.
    14. Ali, Saif & Arora, Gaurav, 2021. "Well-level Missingness Mechanisms in Administrative Groundwater Monitoring Data for Uttar Pradesh (UP), India, 2009-2018," 2021 Annual Meeting, August 1-3, Austin, Texas 314038, Agricultural and Applied Economics Association.
    15. Van Fossen, Jenna A. & Ropp, John W. & Darcy, Kathleen & Hamm, Joseph A., 2022. "Comfort with and willingness to participate in COVID-19 contact tracing: The role of risk perceptions, trust, and political ideology," Social Science & Medicine, Elsevier, vol. 306(C).
    16. Mortaza Jamshidian & Siavash Jalal, 2010. "Tests of Homoscedasticity, Normality, and Missing Completely at Random for Incomplete Multivariate Data," Psychometrika, Springer;The Psychometric Society, vol. 75(4), pages 649-674, December.
    17. Serguei Rouzinov & André Berchtold, 2022. "Regression-Based Approach to Test Missing Data Mechanisms," Data, MDPI, vol. 7(2), pages 1-28, January.
    18. Richard M. Golden & Steven S. Henley & Halbert White & T. Michael Kashner, 2019. "Consequences of Model Misspecification for Maximum Likelihood Estimation with Missing Data," Econometrics, MDPI, vol. 7(3), pages 1-27, September.
    19. Breunig, Christoph & Haan, Peter, 2021. "Nonparametric regression with selectively missing covariates," Journal of Econometrics, Elsevier, vol. 223(1), pages 28-52.
    20. Wei Luo, 2022. "On efficient dimension reduction with respect to the interaction between two response variables," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 84(2), pages 269-294, April.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:csdana:v:182:y:2023:i:c:s0167947323000348. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/csda .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.