IDEAS home Printed from https://ideas.repec.org/p/zbw/zewdip/22001.html
   My bibliography  Save this paper

Towards accountability in machine learning applications: A system-testing approach

Author

Listed:
  • Wan, Wayne Xinwei
  • Lindenthal, Thies

Abstract

A rapidly expanding universe of technology-focused startups is trying to change and improve the way real estate markets operate. The undisputed predictive power of machine learning (ML) models often plays a crucial role in the 'disruption' of traditional processes. However, an accountability gap prevails: How do the models arrive at their predictions? Do they do what we hope they do - or are corners cut? Training ML models is a software development process at heart. We suggest to follow a dedicated software testing framework and to verify that the ML model performs as intended. Illustratively, we augment two ML image classifiers with a system testing procedure based on local interpretable model-agnostic explanation (LIME) techniques. Analyzing the classifications sheds light on some of the factors that determine the behavior of the systems.

Suggested Citation

  • Wan, Wayne Xinwei & Lindenthal, Thies, 2022. "Towards accountability in machine learning applications: A system-testing approach," ZEW Discussion Papers 22-001, ZEW - Leibniz Centre for European Economic Research.
  • Handle: RePEc:zbw:zewdip:22001
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/250385/1/1789605806.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Lily Shen & Stephen L. Ross, 2019. "Information Value of Property Description: A Machine Learning Approach," Working papers 2019-20, University of Connecticut, Department of Economics, revised Sep 2020.
    2. Marc K. Francke & Alex M. Minne, 2017. "Land, Structure and Depreciation," Real Estate Economics, American Real Estate and Urban Economics Association, vol. 45(2), pages 415-451, April.
    3. Sendhil Mullainathan & Ziad Obermeyer, 2017. "Does Machine Learning Automate Moral Hazard and Error?," American Economic Review, American Economic Association, vol. 107(5), pages 476-480, May.
    4. Nikhil Naik & Ramesh Raskar & César A. Hidalgo, 2016. "Cities Are Physical Too: Using Computer Vision to Measure the Quality and Impact of Urban Appearance," American Economic Review, American Economic Association, vol. 106(5), pages 128-132, May.
    5. Gabriel Ahlfeldt & Alexandra Mastro, 2012. "Valuing Iconic Design: Frank Lloyd Wright Architecture in Oak Park, Illinois," Housing Studies, Taylor & Francis Journals, vol. 27(8), pages 1079-1099, November.
    6. Coulson, N. Edward & McMillen, Daniel P., 2008. "Estimating time, age and vintage effects in housing prices," Journal of Housing Economics, Elsevier, vol. 17(2), pages 138-151, June.
    7. Edwin Buitelaar & Frans Schilder, 2017. "The Economics of Style: Measuring the Price Effect of Neo†Traditional Architecture in Housing," Real Estate Economics, American Real Estate and Urban Economics Association, vol. 45(1), pages 7-27, February.
    8. Aubry, Mathieu & Kräussl, Roman & Manso, Gustavo & Spaenjers, Christophe, 2019. "Machine learning, human experts, and the valuation of real assets," CFS Working Paper Series 635, Center for Financial Studies (CFS).
    9. Gabriel M. Ahlfeldt & Kristoffer Moeller & Sevrin Waights & Nicolai Wendland, 2017. "Game of Zones: The Political Economy of Conservation Areas," Economic Journal, Royal Economic Society, vol. 127(605), pages 421-445, October.
    10. Edward L. Glaeser & Michael Scott Kincaid & Nikhil Naik, 2018. "Computer Vision and Real Estate: Do Looks Matter and Do Incentives Determine Looks," NBER Working Papers 25174, National Bureau of Economic Research, Inc.
    11. Nghiep Nguyen & Al Cripps, 2001. "Predicting Housing Value: A Comparison of Multiple Regression Analysis and Artificial Neural Networks," Journal of Real Estate Research, American Real Estate Society, vol. 22(3), pages 313-336.
    12. Erik B Johnson & Alan Tidwell & Sriram V Villupuram, 2020. "Valuing Curb Appeal," The Journal of Real Estate Finance and Economics, Springer, vol. 60(1), pages 111-133, February.
    13. Duncan Simester & Artem Timoshenko & Spyros I. Zoumpoulis, 2020. "Targeting Prospective Customers: Robustness of Machine-Learning Methods to Typical Data Challenges," Management Science, INFORMS, vol. 66(6), pages 2495-2522, June.
    14. David Koch & Miroslav Despotovic & Sascha Leiber & Muntaha Sakeena & Mario Döller & Matthias Zeppelzauer, 2019. "Real Estate Image Analysis: A Literature Review," Journal of Real Estate Literature, Taylor & Francis Journals, vol. 27(2), pages 269-300, December.
    15. Julia Freybote & Lauren Simon & Lauren Beitelspacher, 2016. "Understanding the contribution of curb appeal to retail real estate values," Journal of Property Research, Taylor & Francis Journals, vol. 33(2), pages 147-161, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wayne Xinwei Wan & Thies Lindenthal, 2023. "Testing machine learning systems in real estate," Real Estate Economics, American Real Estate and Urban Economics Association, vol. 51(3), pages 754-778, May.
    2. Ka Shing Cheung & Chung Yim Yiu, 2022. "The economics of architectural aesthetics: Identifying price effect of urban ambiences by different house cohorts," Environment and Planning B, , vol. 49(6), pages 1741-1756, July.
    3. Alan Sage & Mike Langen & Alex van de Minne, 2023. "Where is the opportunity in opportunity zones?," Real Estate Economics, American Real Estate and Urban Economics Association, vol. 51(2), pages 338-371, March.
    4. Yoshida, Jiro, 2020. "The economic depreciation of real estate: Cross-sectional variations and their return implications," Pacific-Basin Finance Journal, Elsevier, vol. 61(C).
    5. Rolheiser, Lyndsey & van Dijk, Dorinth & van de Minne, Alex, 2020. "Housing vintage and price dynamics," Regional Science and Urban Economics, Elsevier, vol. 84(C).
    6. Erik B Johnson & Alan Tidwell & Sriram V Villupuram, 2020. "Valuing Curb Appeal," The Journal of Real Estate Finance and Economics, Springer, vol. 60(1), pages 111-133, February.
    7. Guan‐Yuan Wang, 2023. "The effect of environment on housing prices: Evidence from the Google Street View," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 42(2), pages 288-311, March.
    8. Lyndsey Rolheiser, 2021. "Old, small and unwanted: Post-war housing and neighbourhood socioeconomic status," Urban Studies, Urban Studies Journal Limited, vol. 58(14), pages 2952-2970, November.
    9. Lopez, Luis A. & Yoshida, Jiro, 2022. "Estimating housing rent depreciation for inflation adjustments," Regional Science and Urban Economics, Elsevier, vol. 95(C).
    10. Skripkiūnas Tomas & Navickas Valentinas, 2023. "Architectural Factors Influencing a Housing Market Value: A Theoretical Framework," Real Estate Management and Valuation, Sciendo, vol. 31(1), pages 25-35, March.
    11. Yi Huang & Geoffrey Hewings, 2021. "More Reliable Land Price Index: Is There a Slope Effect?," Land, MDPI, vol. 10(3), pages 1-24, March.
    12. Lyndsey Rolheiser & Dorinth van Dijk & Alex van de Minne, 2018. "Does Housing Vintage Matter? Exploring the Historic City Center of Amsterdam," DNB Working Papers 617, Netherlands Central Bank, Research Department.
    13. Tsai, I-Chun & Wang, Wen-Kai, 2022. "The value of land redevelopment in different types of properties: Considering the effect of hold-out problems on the development probability," Land Use Policy, Elsevier, vol. 119(C).
    14. Liao, Wen-Chi & Jing, Kecen & Lee, Chaun Ying Rachel, 2022. "Economic return of architecture awards: Testing homebuyers’ motives for paying more," Regional Science and Urban Economics, Elsevier, vol. 93(C).
    15. Persson, Petra & Qiu, Xinyao & Rossin-Slater, Maya, 2021. "Family Spillover Effects of Marginal Diagnoses: The Case of ADHD," IZA Discussion Papers 14020, Institute of Labor Economics (IZA).
    16. Saravanan Thirumuruganathan & Soon-gyo Jung & Dianne Ramirez Robillos & Joni Salminen & Bernard J. Jansen, 2021. "Forecasting the nearly unforecastable: why aren’t airline bookings adhering to the prediction algorithm?," Electronic Commerce Research, Springer, vol. 21(1), pages 73-100, March.
    17. Scott Duke Kominers & Alexander Teytelboym & Vincent P Crawford, 2017. "An invitation to market design," Oxford Review of Economic Policy, Oxford University Press and Oxford Review of Economic Policy Limited, vol. 33(4), pages 541-571.
    18. Kitova, Olga & Dyakonova, Ludmila & Savinova, Victoria, 2020. "Prediction of Socio-Economic Indicators of the Megapolis Development on the Basis of the Intellectual Forecasting Information System “SHM Horizon”," MPRA Paper 104234, University Library of Munich, Germany, revised 19 Nov 2020.
    19. Koster, Hans R.A. & van Ommeren, Jos & Volkhausen, Nicolas, 2021. "Short-term rentals and the housing market: Quasi-experimental evidence from Airbnb in Los Angeles," Journal of Urban Economics, Elsevier, vol. 124(C).
    20. Daníelsson, Jón & Macrae, Robert & Uthemann, Andreas, 2022. "Artificial intelligence and systemic risk," Journal of Banking & Finance, Elsevier, vol. 140(C).

    More about this item

    Keywords

    machine learning; accountability gap; computer vision; real estate; urban studies;
    All these keywords.

    JEL classification:

    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection
    • R30 - Urban, Rural, Regional, Real Estate, and Transportation Economics - - Real Estate Markets, Spatial Production Analysis, and Firm Location - - - General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:zewdip:22001. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://edirc.repec.org/data/zemande.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.