IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2405.04816.html
   My bibliography  Save this paper

Testing the Fairness-Accuracy Improvability of Algorithms

Author

Listed:
  • Eric Auerbach
  • Annie Liang
  • Kyohei Okumura
  • Max Tabord-Meehan

Abstract

Many organizations use algorithms that have a disparate impact, i.e., the benefits or harms of the algorithm fall disproportionately on certain social groups. Addressing an algorithm's disparate impact can be challenging, however, because it is often unclear whether it is possible to reduce this impact without sacrificing other objectives of the organization, such as accuracy or profit. Establishing the improvability of algorithms with respect to multiple criteria is of both conceptual and practical interest: in many settings, disparate impact that would otherwise be prohibited under US federal law is permissible if it is necessary to achieve a legitimate business interest. The question is how a policy-maker can formally substantiate, or refute, this "necessity" defense. In this paper, we provide an econometric framework for testing the hypothesis that it is possible to improve on the fairness of an algorithm without compromising on other pre-specified objectives. Our proposed test is simple to implement and can be applied under any exogenous constraint on the algorithm space. We establish the large-sample validity and consistency of our test, and microfound the test's robustness to manipulation based on a game between a policymaker and the analyst. Finally, we apply our approach to evaluate a healthcare algorithm originally considered by Obermeyer et al. (2019), and quantify the extent to which the algorithm's disparate impact can be reduced without compromising the accuracy of its predictions.

Suggested Citation

  • Eric Auerbach & Annie Liang & Kyohei Okumura & Max Tabord-Meehan, 2024. "Testing the Fairness-Accuracy Improvability of Algorithms," Papers 2405.04816, arXiv.org, revised Jan 2025.
  • Handle: RePEc:arx:papers:2405.04816
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2405.04816
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Florios, Kostas & Skouras, Spyros, 2008. "Exact computation of max weighted score estimators," Journal of Econometrics, Elsevier, vol. 146(1), pages 86-91, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chen, Le-Yu & Lee, Sokbae, 2018. "Best subset binary prediction," Journal of Econometrics, Elsevier, vol. 206(1), pages 39-56.
    2. Youngki Shin & Zvezdomir Todorov, 2021. "Exact computation of maximum rank correlation estimator," The Econometrics Journal, Royal Economic Society, vol. 24(3), pages 589-607.
    3. Chen, Le-Yu & Oparina, Ekaterina & Powdthavee, Nattavudh & Srisuma, Sorawoot, 2022. "Robust Ranking of Happiness Outcomes: A Median Regression Perspective," Journal of Economic Behavior & Organization, Elsevier, vol. 200(C), pages 672-686.
    4. D. F. Benoit & D. Van Den Poel, 2010. "Binary quantile regression: A Bayesian approach based on the asymmetric Laplace density," Working Papers of Faculty of Economics and Business Administration, Ghent University, Belgium 10/662, Ghent University, Faculty of Economics and Business Administration.
    5. Dries Benoit & Rahim Alhamzawi & Keming Yu, 2013. "Bayesian lasso binary quantile regression," Computational Statistics, Springer, vol. 28(6), pages 2861-2873, December.
    6. Le‐Yu Chen & Sokbae Lee, 2018. "Exact computation of GMM estimators for instrumental variable quantile regression models," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 33(4), pages 553-567, June.
    7. Andrii Babii & Eric Ghysels & Jonas Striaukas, 2024. "Econometrics of machine learning methods in economic forecasting," Chapters, in: Michael P. Clements & Ana Beatriz Galvão (ed.), Handbook of Research Methods and Applications in Macroeconomic Forecasting, chapter 10, pages 246-273, Edward Elgar Publishing.
    8. Florios, Kostas, 2018. "A hyperplanes intersection simulated annealing algorithm for maximum score estimation," Econometrics and Statistics, Elsevier, vol. 8(C), pages 37-55.
    9. Max Tabord-Meehan, 2023. "Stratification Trees for Adaptive Randomisation in Randomised Controlled Trials," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 90(5), pages 2646-2673.
    10. Chen, Songnian & Zhang, Hanghui, 2015. "Binary quantile regression with local polynomial smoothing," Journal of Econometrics, Elsevier, vol. 189(1), pages 24-40.
    11. Le-Yu Chen & Sokbae (Simon) Lee & Myung Jae Sung, 2013. "Maximum score estimation of preference parameters for a binary choice model under uncertainty," CeMMAP working papers CWP14/13, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    12. Chen, Le-Yu & Oparina, Ekaterina & Powdthavee, Nattavudh & Srisuma, Sorawoot, 2019. "Have Econometric Analyses of Happiness Data Been Futile? A Simple Truth about Happiness Scales," IZA Discussion Papers 12152, Institute of Labor Economics (IZA).
    13. Toru Kitagawa & Aleksey Tetenov, 2018. "Who Should Be Treated? Empirical Welfare Maximization Methods for Treatment Choice," Econometrica, Econometric Society, vol. 86(2), pages 591-616, March.
    14. Le‐Yu Chen & Sokbae Lee & Myung Jae Sung, 2014. "Maximum score estimation with nonparametrically generated regressors," Econometrics Journal, Royal Economic Society, vol. 17(3), pages 271-300, October.
    15. Davide Viviano & Jelena Bradic, 2020. "Fair Policy Targeting," Papers 2005.12395, arXiv.org, revised Jun 2022.
    16. Stéphane Bonhomme & Martin Weidner, 2020. "Posterior average effects," CeMMAP working papers CWP49/20, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    17. Stéphane Bonhomme & Martin Weidner, 2019. "Posterior average effects," CeMMAP working papers CWP43/19, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    18. Christopher D. Walker, 2024. "A Bayesian Perspective on the Maximum Score Problem," Papers 2410.17153, arXiv.org.
    19. Le-Yu Chen & Sokbae Lee, 2018. "High Dimensional Classification through $\ell_0$-Penalized Empirical Risk Minimization," Papers 1811.09540, arXiv.org.
    20. Toru Kitagawa & Aleksey Tetenov, 2015. "Who should be treated? Empirical welfare maximization methods for treatment choice," CeMMAP working papers 10/15, Institute for Fiscal Studies.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2405.04816. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.