IDEAS home Printed from https://ideas.repec.org/a/eee/phsmap/v509y2018icp169-180.html
   My bibliography  Save this article

Optimal threshold for Pareto tail modelling in the presence of outliers

Author

Listed:
  • Safari, Muhammad Aslam Mohd
  • Masseran, Nurulkamal
  • Ibrahim, Kamarulzaman

Abstract

The Pareto distribution is widely applied in many areas of studies such as economics and sciences. An important issues related to Pareto tail modelling is to determine the optimal threshold of the Pareto distribution. One of the methods used for determining the optimal threshold of Pareto distribution is by choosing the threshold that minimizes the goodness-of-fit statistics found based on empirical distribution function (EDF). This study involves determination of the shape parameter of the Pareto distribution using the maximum likelihood method and robust method based on the probability integral transform statistics. In addition, given the particular estimates of the shape parameter, comparison of the performance of several EDF statistics, namely, Kolmogorov–Smirnov, Kuiper, Anderson–Darling, Cramer–von Misses and Watson statistics in determining the optimal threshold in the presence of outliers is studied based on Monte Carlo simulation. Since the EDF statistics are found smallest for Kolmogorov–Smirnov or Kuiper statistics, these two EDF statistics outperformed other EDF statistics considered. The findings are illustrated using a sample of household income data of the Malaysian population. The optimal threshold found can be used to classify the high income earners in Malaysia since Pareto distribution is one of the most frequently used model to describe the upper tail of income distribution.

Suggested Citation

  • Safari, Muhammad Aslam Mohd & Masseran, Nurulkamal & Ibrahim, Kamarulzaman, 2018. "Optimal threshold for Pareto tail modelling in the presence of outliers," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 509(C), pages 169-180.
    • Handle: RePEc:eee:phsmap:v:509:y:2018:i:c:p:169-180
    DOI: 10.1016/j.physa.2018.06.007
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0378437118307313
    Download Restriction: Full text for ScienceDirect subscribers only. Journal offers the option of making the article available online on Science direct for a fee of $3,000
    File URL: https://libkey.io/10.1016/j.physa.2018.06.007?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Luckstead, Jeff & Devadoss, Stephen & Danforth, Diana, 2017. "The size distributions of all Indian cities," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 474(C), pages 237-249.
    2. Coronel-Brizio, H.F. & Hernández-Montoya, A.R., 2005. "On fitting the Pareto–Levy distribution to stock market index data: Selecting a suitable cutoff value," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 354(C), pages 437-449.
    3. Danielsson, J. & de Haan, L. & Peng, L. & de Vries, C. G., 2001. "Using a Bootstrap Method to Choose the Sample Fraction in Tail Index Estimation," Journal of Multivariate Analysis, Elsevier, vol. 76(2), pages 226-248, February.
    4. Oancea, Bogdan & Andrei, Tudorel & Pirjol, Dan, 2017. "Income inequality in Romania: The exponential-Pareto distribution," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 469(C), pages 486-498.
    5. Cirillo, Pasquale, 2013. "Are your data really Pareto distributed?," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 392(23), pages 5947-5962.
    6. Vladimir Filimonov & Didier Sornette, 2014. "Power law scaling and "Dragon-Kings" in distributions of intraday financial drawdowns," Papers 1407.5037, arXiv.org, revised Apr 2015.
    7. D. Sornette, "undated". "Dragon-Kings, Black Swans and the Prediction of Crises," Working Papers CCSS-09-005, ETH Zurich, Chair of Systems Design.
    8. Mark Finkelstein & Howard G. Tucker & Jerry Alan Veeh, 2006. "Pareto Tail Index Estimation Revisited," North American Actuarial Journal, Taylor & Francis Journals, vol. 10(1), pages 1-10.
    9. Devadoss, Stephen & Luckstead, Jeff & Danforth, Diana & Akhundjanov, Sherzod, 2016. "The power law distribution for lower tail cities in India," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 442(C), pages 193-196.
    10. Reed, William J., 2001. "The Pareto, Zipf and other power laws," Economics Letters, Elsevier, vol. 74(1), pages 15-19, December.
    11. Cowell, Frank A. & Flachaire, Emmanuel, 2007. "Income distribution and inequality measurement: The problem of extreme values," Journal of Econometrics, Elsevier, vol. 141(2), pages 1044-1072, December.
    12. Cirillo, Pasquale & Hüsler, Jürg, 2009. "On the upper tail of Italian firms’ size distribution," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 388(8), pages 1546-1554.
    13. Jagielski, Maciej & Czyżewski, Kordian & Kutner, Ryszard & Stanley, H. Eugene, 2017. "Income and wealth distribution of the richest Norwegian individuals: An inequality analysis," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 474(C), pages 330-333.
    14. Frank A. Cowell, 2008. "Income Distribution and Inequality," Chapters, in: John B. Davis & Wilfred Dolfsma (ed.), The Elgar Companion to Social Economics, chapter 13, Edward Elgar Publishing.
    15. Soo, Kwok Tong, 2005. "Zipf's Law for cities: a cross-country investigation," Regional Science and Urban Economics, Elsevier, vol. 35(3), pages 239-263, May.
    16. Luckstead, Jeff & Devadoss, Stephen, 2017. "Pareto tails and lognormal body of US cities size distribution," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 465(C), pages 573-578.
    17. Maciej Jagielski & Ryszard Kutner, 2013. "Modeling of income distribution in the European Union with the Fokker-Planck equation," Papers 1301.2076, arXiv.org.
    18. Politi, Mauro & Scalas, Enrico, 2008. "Fitting the empirical distribution of intertrade durations," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 387(8), pages 2025-2034.
    19. Soriano-Hernández, P. & del Castillo-Mussot, M. & Córdoba-Rodríguez, O. & Mansilla-Corona, R., 2017. "Non-stationary individual and household income of poor, rich and middle classes in Mexico," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 465(C), pages 403-413.
    20. Rytgaard, Mette, 1990. "Estimation in the Pareto Distribution," ASTIN Bulletin, Cambridge University Press, vol. 20(2), pages 201-216, November.
    21. Jagielski, Maciej & Kutner, Ryszard, 2013. "Modelling of income distribution in the European Union with the Fokker–Planck equation," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 392(9), pages 2130-2138.
    22. V. I. Yukalov & D. Sornette, 2012. "Statistical Outliers and Dragon-Kings as Bose-Condensed Droplets," Papers 1205.1364, arXiv.org.
    23. Clementi, F. & Di Matteo, T. & Gallegati, M., 2006. "The power-law tail exponent of income distributions," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 370(1), pages 49-53.
    24. Andreas Alfons & Matthias Templ & Peter Filzmoser, 2013. "Robust estimation of economic indicators from survey samples based on Pareto tail modelling," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 62(2), pages 271-286, March.
    25. Didier Sornette & Guy Ouillon, "undated". "Dragon-kings: Mechanisms, statistical methods and empirical evidence," Working Papers ETH-RC-12-004, ETH Zurich, Chair of Systems Design.
    26. Xavier Gabaix, 2009. "Power Laws in Economics and Finance," Annual Review of Economics, Annual Reviews, vol. 1(1), pages 255-294, May.
    27. Vladimir FILIMONOV & Didier SORNETTE, 2014. "Power Law Scaling and 'Dragon-Kings' in Distributions of Intraday Financial Drawdowns," Swiss Finance Institute Research Paper Series 14-48, Swiss Finance Institute, revised Apr 2015.
    28. Frank Cowell & Maria-Pia Victoria-Feser, 2007. "Robust stochastic dominance: A semi-parametric approach," The Journal of Economic Inequality, Springer;Society for the Study of Economic Inequality, vol. 5(1), pages 21-37, April.
    29. Balakrishnan, P.V. (Sundar) & Miller, James M. & Shankar, S. Gowri, 2008. "Power law and evolutionary trends in stock markets," Economics Letters, Elsevier, vol. 98(2), pages 194-200, February.
    30. Chami Figueira, F. & Moura, N.J. & Ribeiro, M.B., 2011. "The Gompertz–Pareto income distribution," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 390(4), pages 689-698.
    31. Tomson Ogwang, 2011. "Power laws in top wealth distributions: evidence from Canada," Empirical Economics, Springer, vol. 41(2), pages 473-486, October.
    32. Filimonov, Vladimir & Sornette, Didier, 2015. "Power law scaling and “Dragon-Kings” in distributions of intraday financial drawdowns," Chaos, Solitons & Fractals, Elsevier, vol. 74(C), pages 27-45.
    33. Didier SORNETTE, 2009. "Dragon-Kings, Black Swans and the Prediction of Crises," Swiss Finance Institute Research Paper Series 09-36, Swiss Finance Institute.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Safari, Muhammad Aslam Mohd & Masseran, Nurulkamal & Ibrahim, Kamarulzaman & AL-Dhurafi, Nasr Ahmed, 2020. "The power-law distribution for the income of poor households," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 557(C).
    2. Muhammad Hilmi Abdul Majid & Kamarulzaman Ibrahim & Nurulkamal Masseran, 2023. "Three-Part Composite Pareto Modelling for Income Distribution in Malaysia," Mathematics, MDPI, vol. 11(13), pages 1-15, June.
    3. Safari, Muhammad Aslam Mohd & Masseran, Nurulkamal & Ibrahim, Kamarulzaman, 2018. "A robust semi-parametric approach for measuring income inequality in Malaysia," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 512(C), pages 1-13.
    4. Peipei Ma & Guosheng Li, 2023. "Comparison and Analysis of Detection Methods for Typhoon-Storm Surges Based on Tide-Gauge Data—Taking Coasts of China as Examples," IJERPH, MDPI, vol. 20(4), pages 1-21, February.
    5. Maria Marino & Benedetto Rocchi & Simone Severini, 2021. "Conditional Income Disparity between Farm and Non‐farm Households in the European Union: A Longitudinal Analysis," Journal of Agricultural Economics, Wiley Blackwell, vol. 72(2), pages 589-606, June.
    6. Safari, Muhammad Aslam Mohd & Masseran, Nurulkamal & Ibrahim, Kamarulzaman & Hussain, Saiful Izzuan, 2021. "Measuring income inequality: A robust semi-parametric approach," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 562(C).
    7. Safari, Muhammad Aslam Mohd & Masseran, Nurulkamal & Ibrahim, Kamarulzaman & Hussain, Saiful Izzuan, 2019. "A robust and efficient estimator for the tail index of inverse Pareto distribution," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 517(C), pages 431-439.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Safari, Muhammad Aslam Mohd & Masseran, Nurulkamal & Ibrahim, Kamarulzaman & Hussain, Saiful Izzuan, 2019. "A robust and efficient estimator for the tail index of inverse Pareto distribution," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 517(C), pages 431-439.
    2. Safari, Muhammad Aslam Mohd & Masseran, Nurulkamal & Ibrahim, Kamarulzaman, 2018. "A robust semi-parametric approach for measuring income inequality in Malaysia," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 512(C), pages 1-13.
    3. Safari, Muhammad Aslam Mohd & Masseran, Nurulkamal & Ibrahim, Kamarulzaman & Hussain, Saiful Izzuan, 2021. "Measuring income inequality: A robust semi-parametric approach," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 562(C).
    4. Safari, Muhammad Aslam Mohd & Masseran, Nurulkamal & Ibrahim, Kamarulzaman & AL-Dhurafi, Nasr Ahmed, 2020. "The power-law distribution for the income of poor households," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 557(C).
    5. Michał Brzeziński, 2013. "Robust estimation of the Pareto index: A Monte Carlo Analysis," Working Papers 2013-32, Faculty of Economic Sciences, University of Warsaw.
    6. Igor Fedotenkov, 2020. "A Review of More than One Hundred Pareto-Tail Index Estimators," Statistica, Department of Statistics, University of Bologna, vol. 80(3), pages 245-299.
    7. Arthur Charpentier & Emmanuel Flachaire, 2022. "Pareto models for top incomes and wealth," The Journal of Economic Inequality, Springer;Society for the Study of Economic Inequality, vol. 20(1), pages 1-25, March.
    8. Stephen P. Jenkins, 2017. "Pareto Models, Top Incomes and Recent Trends in UK Income Inequality," Economica, London School of Economics and Political Science, vol. 84(334), pages 261-289, April.
    9. Bluhm, Richard & Krause, Melanie, 2022. "Top lights: Bright cities and their contribution to economic development," Journal of Development Economics, Elsevier, vol. 157(C).
    10. Tomaschitz, Roman, 2020. "Multiply broken power-law densities as survival functions: An alternative to Pareto and lognormal fits," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 541(C).
    11. Vladimir Hlasny, 2021. "Parametric representation of the top of income distributions: Options, historical evidence, and model selection," Journal of Economic Surveys, Wiley Blackwell, vol. 35(4), pages 1217-1256, September.
    12. Muhammad Hilmi Abdul Majid & Kamarulzaman Ibrahim & Nurulkamal Masseran, 2023. "Three-Part Composite Pareto Modelling for Income Distribution in Malaysia," Mathematics, MDPI, vol. 11(13), pages 1-15, June.
    13. Frank A. Cowell & Philippe Kerm, 2015. "Wealth Inequality: A Survey," Journal of Economic Surveys, Wiley Blackwell, vol. 29(4), pages 671-710, September.
    14. Bluhm, Richard & Krause, Melanie, 2022. "Top lights: Bright cities and their contribution to economic development," Journal of Development Economics, Elsevier, vol. 157(C).
    15. Akhundjanov, Sherzod B. & Devadoss, Stephen & Luckstead, Jeff, 2017. "Size distribution of national CO2 emissions," Energy Economics, Elsevier, vol. 66(C), pages 182-193.
    16. Glette-Iversen, Ingrid & Aven, Terje, 2021. "On the meaning of and relationship between dragon-kings, black swans and related concepts," Reliability Engineering and System Safety, Elsevier, vol. 211(C).
    17. Arthur Charpentier & Emmanuel Flachaire, 2019. "Pareto Models for Top Incomes," Université Paris1 Panthéon-Sorbonne (Post-Print and Working Papers) hal-02145024, HAL.
    18. Li, Chengyou & Yu, Yangcheng & Li, Qinghai, 2021. "Top-income data and income inequality correction in China," Economic Modelling, Elsevier, vol. 97(C), pages 210-219.
    19. Díaz, Juan D. & Gutiérrez Cubillos, Pablo & Tapia Griñen, Pablo, 2021. "The exponential Pareto model with hidden income processes: Evidence from Chile," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 561(C).
    20. Rafael González-Val, 2019. "US city-size distribution and space," Spatial Economic Analysis, Taylor & Francis Journals, vol. 14(3), pages 283-300, July.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:phsmap:v:509:y:2018:i:c:p:169-180. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/physica-a-statistical-mechpplications/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.