IDEAS home Printed from https://ideas.repec.org/p/msh/ebswps/2018-16.html
   My bibliography  Save this paper

On normalization and algorithm selection for unsupervised outlier detection

Author

Listed:
  • Sevvandi Kandanaarachchi
  • Mario A Munoz
  • Rob J Hyndman
  • Kate Smith-Miles

Abstract

This paper demonstrates that the performance of various outlier detection methods depends sensitively on both the data normalization schemes employed, as well as characteristics of the datasets. Recasting the challenge of understanding these dependencies as an algorithm selection problem, we perform the first instance space analysis of outlier detection methods. Such analysis enables the strengths and weaknesses of unsupervised outlier detection methods to be visualized and insights gained into which method and normalization scheme should be selected to obtain the most likely best performance for a given dataset.

Suggested Citation

  • Sevvandi Kandanaarachchi & Mario A Munoz & Rob J Hyndman & Kate Smith-Miles, 2018. "On normalization and algorithm selection for unsupervised outlier detection," Monash Econometrics and Business Statistics Working Papers 16/18, Monash University, Department of Econometrics and Business Statistics.
    • Handle: RePEc:msh:ebswps:2018-16
    as

    Download full text from publisher

    File URL: https://www.monash.edu/business/ebs/research/publications/ebs/wp16-2018.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. David H. Wolpert & William G. Macready, 1995. "No Free Lunch Theorems for Search," Working Papers 95-02-010, Santa Fe Institute.
    2. Markus Goldstein & Seiichi Uchida, 2016. "A Comparative Evaluation of Unsupervised Anomaly Detection Algorithms for Multivariate Data," PLOS ONE, Public Library of Science, vol. 11(4), pages 1-31, April.
    3. Kang, Yanfei & Hyndman, Rob J. & Smith-Miles, Kate, 2017. "Visualising forecasting algorithm performance using time series instance spaces," International Journal of Forecasting, Elsevier, vol. 33(2), pages 345-358.
    4. Billor, Nedret & Hadi, Ali S. & Velleman, Paul F., 2000. "BACON: blocked adaptive computationally efficient outlier nominators," Computational Statistics & Data Analysis, Elsevier, vol. 34(3), pages 279-298, September.
    5. Priyanga Dilini Talagala & Rob J Hyndman & Kate Smith-Miles & Sevvandi Kandanaarachchi & Mario A Munoz, 2018. "Anomaly detection in streaming nonstationary temporal data," Monash Econometrics and Business Statistics Working Papers 4/18, Monash University, Department of Econometrics and Business Statistics.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Mohamad Alissa & Kevin Sim & Emma Hart, 2023. "Automated Algorithm Selection: from Feature-Based to Feature-Free Approaches," Journal of Heuristics, Springer, vol. 29(1), pages 1-38, February.
    2. Sevvandi Kandanaarachchi & Rob J Hyndman, 2019. "Dimension Reduction For Outlier Detection Using DOBIN," Monash Econometrics and Business Statistics Working Papers 17/19, Monash University, Department of Econometrics and Business Statistics.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Priyanga Dilini Talagala & Rob J Hyndman & Catherine Leigh & Kerrie Mengersen & Kate Smith-Miles, 2019. "A Feature-Based Framework for Detecting Technical Outliers in Water-Quality Data from In Situ Sensors," Monash Econometrics and Business Statistics Working Papers 1/19, Monash University, Department of Econometrics and Business Statistics.
    2. Priyanga Dilini Talagala & Rob J Hyndman & Kate Smith-Miles, 2019. "Anomaly Detection in High Dimensional Data," Monash Econometrics and Business Statistics Working Papers 20/19, Monash University, Department of Econometrics and Business Statistics.
    3. Sevvandi Kandanaarachchi & Rob J Hyndman, 2019. "Dimension Reduction For Outlier Detection Using DOBIN," Monash Econometrics and Business Statistics Working Papers 17/19, Monash University, Department of Econometrics and Business Statistics.
    4. Catherine Fuss & Angelos Theodorakopoulos, 2018. "Compositional Changes in Aggregate Productivity in an Era of Globalisation and Financial Crisis," Working Papers of VIVES - Research Centre for Regional Economics 627696, KU Leuven, Faculty of Economics and Business (FEB), VIVES - Research Centre for Regional Economics.
    5. Yin, Sihua & Yang, Haidong & Xu, Kangkang & Zhu, Chengjiu & Zhang, Shaqing & Liu, Guosheng, 2022. "Dynamic real–time abnormal energy consumption detection and energy efficiency optimization analysis considering uncertainty," Applied Energy, Elsevier, vol. 307(C).
    6. Charlotte Fabri & Sam Vermeulen & Steven Van Passel & Sergei Schaub, 2024. "Crop diversification and the effect of weather shocks on Italian farmers' income and income risk," Journal of Agricultural Economics, Wiley Blackwell, vol. 75(3), pages 955-980, September.
    7. Hong Choon Ong & Ekele Alih, 2015. "A Control Chart Based on Cluster-Regression Adjustment for Retrospective Monitoring of Individual Characteristics," PLOS ONE, Public Library of Science, vol. 10(4), pages 1-30, April.
    8. Adele Ravagnani & Fabrizio Lillo & Paola Deriu & Piero Mazzarisi & Francesca Medda & Antonio Russo, 2024. "Dimensionality reduction techniques to support insider trading detection," Papers 2403.00707, arXiv.org, revised May 2024.
    9. Davide Nicola Continanza & Andrea del Monaco & Marco di Lucido & Daniele Figoli & Pasquale Maddaloni & Filippo Quarta & Giuseppe Turturiello, 2023. "Stacking machine learning models for anomaly detection: comparing AnaCredit to other banking data sets," IFC Bulletins chapters, in: Bank for International Settlements (ed.), Data science in central banking: applications and tools, volume 59, Bank for International Settlements.
    10. Stefani, Gianluca & Gadanakis, Yiorgos & Lombardi, Ginevra Virginia & Tiberti, Marco, 2017. "The impact of financial leverage on farms capacity to react in market shocks," 2017 International Congress, August 28-September 1, 2017, Parma, Italy 261156, European Association of Agricultural Economists.
    11. Peter Nemec, 2024. "Contesting the public works domain: examining the factors affecting presence and success of SMES in public procurement," Empirical Economics, Springer, vol. 67(5), pages 2135-2173, November.
    12. Batalla-Bejerano, Joan & Costa-Campi, Maria Teresa & Trujillo-Baute, Elisa, 2016. "Collateral effects of liberalisation: Metering, losses, load profiles and cost settlement in Spain’s electricity system," Energy Policy, Elsevier, vol. 94(C), pages 421-431.
    13. Thiyanga S. Talagala & Feng Li & Yanfei Kang, 2019. "Feature-based Forecast-Model Performance Prediction," Monash Econometrics and Business Statistics Working Papers 21/19, Monash University, Department of Econometrics and Business Statistics.
    14. A.A.M. Nurunnabi & M. Nasser & A.H.M.R. Imon, 2016. "Identification and classification of multiple outliers, high leverage points and influential observations in linear regression," Journal of Applied Statistics, Taylor & Francis Journals, vol. 43(3), pages 509-525, March.
    15. Jui-Sheng Chou & Dinh-Nhat Truong & Chih-Fong Tsai, 2021. "Solving Regression Problems with Intelligent Machine Learner for Engineering Informatics," Mathematics, MDPI, vol. 9(6), pages 1-25, March.
    16. Jiří Schwarz & Martin Pospíšil, 2018. "Bankruptcy, Investment, and Financial Constraints: Evidence from the Czech Republic," Eastern European Economics, Taylor & Francis Journals, vol. 56(2), pages 99-121, March.
    17. Huber, Jakob & Stuckenschmidt, Heiner, 2020. "Daily retail demand forecasting using machine learning with emphasis on calendric special days," International Journal of Forecasting, Elsevier, vol. 36(4), pages 1420-1438.
    18. José Ignacio Giménez-Nadal & José Alberto Molina & Jorge Velilla, 2024. "Intermediate activities while commuting," Review of Economics of the Household, Springer, vol. 22(3), pages 1185-1220, September.
    19. Thiyanga S Talagala & Rob J Hyndman & George Athanasopoulos, 2018. "Meta-learning how to forecast time series," Monash Econometrics and Business Statistics Working Papers 6/18, Monash University, Department of Econometrics and Business Statistics.
    20. You-Shyang Chen & Arun Kumar Sangaiah & Yu-Pei Lin, 2024. "Hyperautomation on fuzzy data dredging on four advanced industrial forecasting models to support sustainable business management," Annals of Operations Research, Springer, vol. 342(1), pages 215-264, November.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:msh:ebswps:2018-16. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Professor Xibin Zhang (email available below). General contact details of provider: https://edirc.repec.org/data/dxmonau.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.