IDEAS home Printed from https://ideas.repec.org/a/vrs/stintr/v21y2020i4p123-143n11.html
   My bibliography  Save this article

High dimensional, robust, unsupervised record linkage

Author

Listed:
  • Bera Sabyasachi

    (University of Minnesota, ; Minnesota, ; United States)

  • Chatterjee Snigdhansu

    (University of Minnesota, ; Minnesota, ; United States)

Abstract

We develop a technique for record linkage on high dimensional data, where the two datasets may not have any common variable, and there may be no training set available. Our methodology is based on sparse, high dimensional principal components. Since large and high dimensional datasets are often prone to outliers and aberrant observations, we propose a technique for estimating robust, high dimensional principal components. We present theoretical results validating the robust, high dimensional principal component estimation steps, and justifying their use for record linkage. Some numeric results and remarks are also presented.

Suggested Citation

  • Bera Sabyasachi & Chatterjee Snigdhansu, 2020. "High dimensional, robust, unsupervised record linkage," Statistics in Transition New Series, Statistics Poland, vol. 21(4), pages 123-143, August.
    • Handle: RePEc:vrs:stintr:v:21:y:2020:i:4:p:123-143:n:11
    DOI: 10.21307/stattrans-2020-034
    as

    Download full text from publisher

    File URL: https://doi.org/10.21307/stattrans-2020-034
    Download Restriction: no
    File URL: https://libkey.io/10.21307/stattrans-2020-034?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. P. Lahiri & Michael D. Larsen, 2005. "Regression Analysis With Linked Data," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 222-230, March.
    2. Mauricio Sadinle & Stephen E. Fienberg, 2013. "A Generalized Fellegi--Sunter Framework for Multiple Record Linkage With Application to Homicide Record Systems," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 108(502), pages 385-397, June.
    3. Larsen M. D & Rubin D. B, 2001. "Iterative Automated Record Linkage Using Mixture Models," Journal of the American Statistical Association, American Statistical Association, vol. 96, pages 32-41, March.
    4. Taskinen, Sara & Koch, Inge & Oja, Hannu, 2012. "Robustifying principal component analysis with spatial sign vectors," Statistics & Probability Letters, Elsevier, vol. 82(4), pages 765-774.
    5. Rebecca C. Steorts & Rob Hall & Stephen E. Fienberg, 2016. "A Bayesian Approach to Graphical Record Linkage and Deduplication," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 111(516), pages 1660-1672, October.
    6. Ying Han & Partha Lahiri, 2019. "Statistical Analysis with Linked Data," International Statistical Review, International Statistical Institute, vol. 87(S1), pages 139-157, May.
    7. Mauricio Sadinle, 2017. "Bayesian Estimation of Bipartite Matchings for Record Linkage," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(518), pages 600-612, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sabyasachi Bera & Snigdhansu Chatterjee, 2020. "High dimensional, robust, unsupervised record linkage," Statistics in Transition New Series, Polish Statistical Association, vol. 21(4), pages 123-143, August.
    2. Thomas Stringham, 2022. "Fast Bayesian Record Linkage With Record-Specific Disagreement Parameters," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 40(4), pages 1509-1522, October.
    3. Vo, Thanh Huan & Chauvet, Guillaume & Happe, André & Oger, Emmanuel & Paquelet, Stéphane & Garès, Valérie, 2023. "Extending the Fellegi-Sunter record linkage model for mixed-type data with application to the French national health data system," Computational Statistics & Data Analysis, Elsevier, vol. 179(C).
    4. Betancourt, Brenda & Sosa, Juan & Rodríguez, Abel, 2022. "A prior for record linkage based on allelic partitions," Computational Statistics & Data Analysis, Elsevier, vol. 172(C).
    5. Daniel H. Weinberg & John M. Abowd & Robert F. Belli & Noel Cressie & David C. Folch & Scott H. Holan & Margaret C. Levenstein & Kristen M. Olson & Jerome P. Reiter & Matthew D. Shapiro & Jolene Smyth, 2017. "Effects of a Government-Academic Partnership: Has the NSF-Census Bureau Research Network Helped Improve the U.S. Statistical System?," Working Papers 17-59r, Center for Economic Studies, U.S. Census Bureau.
    6. Duncan Smith, 2020. "Re‐identification in the Absence of Common Variables for Matching," International Statistical Review, International Statistical Institute, vol. 88(2), pages 354-379, August.
    7. Afshin Fallah & Mohsen Mohammadzadeh, 2010. "Bayesian regression analysis with linked data using mixture normal distributions," Statistical Papers, Springer, vol. 51(2), pages 421-430, June.
    8. Han Ying, 2020. "Discussion of “Small area estimation: its evolution in five decades”, by Malay Ghosh," Statistics in Transition New Series, Statistics Poland, vol. 21(4), pages 30-34, August.
    9. Ying Han, 2020. "Discussion of "Small area estimation: its evolution in five decades", by Malay Ghosh," Statistics in Transition New Series, Polish Statistical Association, vol. 21(4), pages 30-34, August.
    10. Angelo Moretti & Natalie Shlomo, 2023. "Improving Probabilistic Record Linkage Using Statistical Prediction Models," International Statistical Review, International Statistical Institute, vol. 91(3), pages 368-394, December.
    11. Michael Scholz & Markus Franz & Oliver Hinz, 2016. "The Ambiguous Identifier Clustering Technique," Electronic Markets, Springer;IIM University of St. Gallen, vol. 26(2), pages 143-156, May.
    12. John M. Abowd & Joelle Abramowitz & Margaret C. Levenstein & Kristin McCue & Dhiren Patki & Trivellore Raghunathan & Ann M. Rodgers & Matthew D. Shapiro & Nada Wasi & Dawn Zinsser, 2021. "Finding Needles in Haystacks: Multiple-Imputation Record Linkage Using Machine Learning," Working Papers 21-35, Center for Economic Studies, U.S. Census Bureau.
    13. D. H. Judson, 2007. "Information integration for constructing social statistics: history, theory and ideas towards a research programme," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 170(2), pages 483-501, March.
    14. Li‐Chun Zhang & Tiziana Tuoto, 2021. "Linkage‐data linear regression," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 184(2), pages 522-547, April.
    15. Al-Kandari Noriah M. & Lahiri Partha, 2016. "Prediction of a Function of Misclassified Binary Data," Statistics in Transition New Series, Statistics Poland, vol. 17(3), pages 429-447, September.
    16. Guangxing Wang & Sisheng Liu & Fang Han & Chong‐Zhi Di, 2023. "Robust functional principal component analysis via a functional pairwise spatial sign operator," Biometrics, The International Biometric Society, vol. 79(2), pages 1239-1253, June.
    17. Josef Schürle, 2005. "A method for consideration of conditional dependencies in the Fellegi and Sunter model of record linkage," Statistical Papers, Springer, vol. 46(3), pages 433-449, July.
    18. Bakker Bart F.M. & Heijden Peter G.M. van der & Scholtus Sander, 2015. "Preface," Journal of Official Statistics, Sciendo, vol. 31(3), pages 349-355, September.
    19. Dasylva Abel, 2018. "Design-Based Estimation with Record-Linked Administrative Files and a Clerical Review Sample," Journal of Official Statistics, Sciendo, vol. 34(1), pages 41-54, March.
    20. Kristian Lum & Megan Emily Price & David Banks, 2013. "Rejoinder," The American Statistician, Taylor & Francis Journals, vol. 67(4), pages 205-206, November.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:vrs:stintr:v:21:y:2020:i:4:p:123-143:n:11. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Peter Golla (email available below). General contact details of provider: https://stat.gov.pl/en/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.