IDEAS home Printed from https://ideas.repec.org/p/ehl/lserod/120493.html
   My bibliography  Save this paper

Epistemic parity: reproducibility as an evaluation metric for differential privacy

Author

Listed:
  • Rosenblatt, Lucas
  • Herman, Bernease
  • Holovenko, Anastasia
  • Lee, Wonkwon
  • Loftus, Joshua
  • McKinnie, Elizabeth
  • Rumezhak, Taras
  • Stadnik, Andrii
  • Howe, Bill
  • Stoyanovich, Julia

Abstract

Differential privacy (DP) data synthesizers are increasingly proposed to afford public release of sensitive information, offering theoretical guarantees for privacy (and, in some cases, utility), but limited empirical evidence of utility in practical settings. Utility is typically measured as the error on representative proxy tasks, such as descriptive statistics, multivariate correlations, the accuracy of trained classifiers, or performance over a query workload. The ability for these results to generalize to practitioners' experience has been questioned in a number of settings, including the U.S. Census. In this paper, we propose an evaluation methodology for synthetic data that avoids assumptions about the representativeness of proxy tasks, instead measuring the likelihood that published conclusions would change had the authors used synthetic data, a condition we call epistemic parity. Our methodology consists of reproducing empirical conclusions of peer-reviewed papers on real, publicly available data, then re-running these experiments a second time on DP synthetic data and comparing the results. We instantiate our methodology over a benchmark of recent peer-reviewed papers that analyze public datasets in the ICPSR social science repository. We model quantitative claims computationally to automate the experimental workflow, and model qualitative claims by reproducing visualizations and comparing the results manually. We then generate DP synthetic datasets using multiple state-of-the-art mechanisms, and estimate the likelihood that these conclusions will hold. We find that, for reasonable privacy regimes, state-of-the-art DP synthesizers are able to achieve high epistemic parity for several papers in our benchmark. However, some papers, and particularly some specific findings, are difficult to reproduce for any of the synthesizers. Given these results, we advocate for a new class of mechanisms that can reorder the priorities for DP data synthesis: favor stronger guarantees for utility (as measured by epistemic parity) and offer privacy protection with a focus on application-specific threat models and risk-assessment.

Suggested Citation

  • Rosenblatt, Lucas & Herman, Bernease & Holovenko, Anastasia & Lee, Wonkwon & Loftus, Joshua & McKinnie, Elizabeth & Rumezhak, Taras & Stadnik, Andrii & Howe, Bill & Stoyanovich, Julia, 2023. "Epistemic parity: reproducibility as an evaluation metric for differential privacy," LSE Research Online Documents on Economics 120493, London School of Economics and Political Science, LSE Library.
  • Handle: RePEc:ehl:lserod:120493
    as

    Download full text from publisher

    File URL: http://eprints.lse.ac.uk/120493/
    File Function: Open access version.
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Shervin Assari & Mohsen Bazargan, 2019. "Baseline Obesity Increases 25-Year Risk of Mortality due to Cerebrovascular Disease: Role of Race," IJERPH, MDPI, vol. 16(19), pages 1-15, October.
    2. Monya Baker, 2016. "1,500 scientists lift the lid on reproducibility," Nature, Nature, vol. 533(7604), pages 452-454, May.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dennis Bontempi & Leonard Nuernberg & Suraj Pai & Deepa Krishnaswamy & Vamsi Thiriveedhi & Ahmed Hosny & Raymond H. Mak & Keyvan Farahani & Ron Kikinis & Andrey Fedorov & Hugo J. W. L. Aerts, 2024. "End-to-end reproducible AI pipelines in radiology using the cloud," Nature Communications, Nature, vol. 15(1), pages 1-9, December.
    2. Fernando Hoces de la Guardia & Sean Grant & Edward Miguel, 2021. "A framework for open policy analysis," Science and Public Policy, Oxford University Press, vol. 48(2), pages 154-163.
    3. Antonella Lanati & Marinella Marzano & Caterina Manzari & Bruno Fosso & Graziano Pesole & Francesca De Leo, 2019. "Management at the service of research: ReOmicS, a quality management system for omics sciences," Palgrave Communications, Palgrave Macmillan, vol. 5(1), pages 1-13, December.
    4. Joel Ferguson & Rebecca Littman & Garret Christensen & Elizabeth Levy Paluck & Nicholas Swanson & Zenan Wang & Edward Miguel & David Birke & John-Henry Pezzuto, 2023. "Survey of open science practices and attitudes in the social sciences," Nature Communications, Nature, vol. 14(1), pages 1-13, December.
    5. Bor Luen Tang, 2023. "Some Insights into the Factors Influencing Continuous Citation of Retracted Scientific Papers," Publications, MDPI, vol. 11(4), pages 1-14, October.
    6. Inga Patarčić & Jadranka Stojanovski, 2022. "Adoption of Transparency and Openness Promotion (TOP) Guidelines across Journals," Publications, MDPI, vol. 10(4), pages 1-10, November.
    7. Susanne Wieschowski & Svenja Biernot & Susanne Deutsch & Silke Glage & André Bleich & René Tolba & Daniel Strech, 2019. "Publication rates in animal research. Extent and characteristics of published and non-published animal studies followed up at two German university medical centres," PLOS ONE, Public Library of Science, vol. 14(11), pages 1-8, November.
    8. Shinichi Nakagawa & Edward R. Ivimey-Cook & Matthew J. Grainger & Rose E. O’Dea & Samantha Burke & Szymon M. Drobniak & Elliot Gould & Erin L. Macartney & April Robin Martinig & Kyle Morrison & Matthi, 2023. "Method Reporting with Initials for Transparency (MeRIT) promotes more granularity and accountability for author contributions," Nature Communications, Nature, vol. 14(1), pages 1-5, December.
    9. Paul J. Ferraro & J. Dustin Tracy, 2022. "A reassessment of the potential for loss-framed incentive contracts to increase productivity: a meta-analysis and a real-effort experiment," Experimental Economics, Springer;Economic Science Association, vol. 25(5), pages 1441-1466, November.
    10. Brian M. Schilder & Alan E. Murphy & Nathan G. Skene, 2024. "rworkflows: automating reproducible practices for the R community," Nature Communications, Nature, vol. 15(1), pages 1-10, December.
    11. Tim Hulsen, 2020. "Sharing Is Caring—Data Sharing Initiatives in Healthcare," IJERPH, MDPI, vol. 17(9), pages 1-12, April.
    12. Michael Meiser & Ingo Zinnikus, 2024. "A Survey on the Use of Synthetic Data for Enhancing Key Aspects of Trustworthy AI in the Energy Domain: Challenges and Opportunities," Energies, MDPI, vol. 17(9), pages 1-29, April.
    13. Chris H. J. Hartgerink & Marino Van Zelst, 2018. "“As-You-Go” Instead of “After-the-Fact”: A Network Approach to Scholarly Communication and Evaluation," Publications, MDPI, vol. 6(2), pages 1-10, April.
    14. Lingjing Jiang & Niina Haiminen & Anna‐Paola Carrieri & Shi Huang & Yoshiki Vázquez‐Baeza & Laxmi Parida & Ho‐Cheol Kim & Austin D. Swafford & Rob Knight & Loki Natarajan, 2022. "Utilizing stability criteria in choosing feature selection methods yields reproducible results in microbiome data," Biometrics, The International Biometric Society, vol. 78(3), pages 1155-1167, September.
    15. Joshua Borycz & Robert Olendorf & Alison Specht & Bruce Grant & Kevin Crowston & Carol Tenopir & Suzie Allard & Natalie M. Rice & Rachael Hu & Robert J. Sandusky, 2023. "Perceived benefits of open data are improving but scientists still lack resources, skills, and rewards," Palgrave Communications, Palgrave Macmillan, vol. 10(1), pages 1-12, December.
    16. Völker, Richard & Hirschauer, Norbert & Lind, Fabienne & Gruener, Sven, 2024. "Search term validation in agricultural economics: conceptual background and application," OSF Preprints v68r7, Center for Open Science.
    17. Paul-Martin Luc & Simon Bauer & Julia Kowal, 2022. "Reproducible Production of Lithium-Ion Coin Cells," Energies, MDPI, vol. 15(21), pages 1-16, October.
    18. Cantone, Giulio Giacomo, 2023. "The multiversal methodology as a remedy of the replication crisis," MetaArXiv kuhmz, Center for Open Science.
    19. Ron S. Kenett & Abraham Rubinstein, 2021. "Generalizing research findings for enhanced reproducibility: an approach based on verbal alternative representations," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(5), pages 4137-4151, May.
    20. Naudé, Wim, 2024. "Is the Scholarly Field of Entrepreneurship at Its End?," IZA Discussion Papers 16916, Institute of Labor Economics (IZA).

    More about this item

    Keywords

    NSF Awards Nos. 1916505; 1922658; 1934405; NSF Graduate Research Fellowship Grant No. DGE-2039655;
    All these keywords.

    JEL classification:

    • C1 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ehl:lserod:120493. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: LSERO Manager (email available below). General contact details of provider: https://edirc.repec.org/data/lsepsuk.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.