IDEAS home Printed from https://ideas.repec.org/a/spr/stmapp/v27y2018i1d10.1007_s10260-017-0394-y.html
   My bibliography  Save this article

Dynamic factor analysis for short panels: estimating performance trajectories for water utilities

Author

Listed:
  • Nikolaos Zirogiannis

    (Indiana University Bloomington)

  • Yorghos Tripodis

    (Boston University)

Abstract

We develop a novel estimation algorithm for a dynamic factor model (DFM) applied to panel data with a short time dimension and a large cross sectional dimension. Current DFMs usually require panels with a minimum of 20 years of quarterly data (80 time observations per panel). In contrast, the application we consider includes panels with a median of 8 annual observations. As a result, the time dimension in our paper is substantially shorter than previous work in the DFM literature. This difference increases the computational challenges of the estimation process which we address by developing the “Two-Cycle Conditional Expectation - Maximization” (2CCEM) algorithm which is a variant of the EM algorithm and its extensions. We analyze the conditions under which our model is identified and provide simulation results demonstrating consistency of our 2CCEM estimator. We apply the DFM to a dataset of 802 water and sanitation utilities from 43 countries and use the 2CCEM algorithm in order to estimate dynamic performance trajectories for each utility.

Suggested Citation

  • Nikolaos Zirogiannis & Yorghos Tripodis, 2018. "Dynamic factor analysis for short panels: estimating performance trajectories for water utilities," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 27(1), pages 131-150, March.
    • Handle: RePEc:spr:stmapp:v:27:y:2018:i:1:d:10.1007_s10260-017-0394-y
    DOI: 10.1007/s10260-017-0394-y
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s10260-017-0394-y
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s10260-017-0394-y?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Catherine Doz & Domenico Giannone & Lucrezia Reichlin, 2012. "A Quasi–Maximum Likelihood Approach for Large, Approximate Dynamic Factor Models," The Review of Economics and Statistics, MIT Press, vol. 94(4), pages 1014-1024, November.
    2. R. H. Shumway & D. S. Stoffer, 1982. "An Approach To Time Series Smoothing And Forecasting Using The Em Algorithm," Journal of Time Series Analysis, Wiley Blackwell, vol. 3(4), pages 253-264, July.
    3. Charnes, A. & Cooper, W. W. & Rhodes, E., 1978. "Measuring the efficiency of decision making units," European Journal of Operational Research, Elsevier, vol. 2(6), pages 429-444, November.
    4. Stock, James H. & Watson, Mark, 2011. "Dynamic Factor Models," Scholarly Articles 28469541, Harvard University Department of Economics.
    5. Siem Jan Koopman & Neil Shephard & Jurgen A. Doornik, 1999. "Statistical algorithms for models in state space using SsfPack 2.2," Econometrics Journal, Royal Economic Society, vol. 2(1), pages 107-160.
    6. Henghsiu Tsai & K. S. Chan, 2005. "Temporal Aggregation of Stationary And Nonstationary Discrete‐Time Processes," Journal of Time Series Analysis, Wiley Blackwell, vol. 26(4), pages 613-624, July.
    7. Forni, Mario & Hallin, Marc & Lippi, Marco & Reichlin, Lucrezia, 2005. "The Generalized Dynamic Factor Model: One-Sided Estimation and Forecasting," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 830-840, September.
    8. Mario Forni & Marc Hallin & Marco Lippi & Lucrezia Reichlin, 2000. "The Generalized Dynamic-Factor Model: Identification And Estimation," The Review of Economics and Statistics, MIT Press, vol. 82(4), pages 540-554, November.
    9. Thanassoulis, Emmanuel, 2000. "The use of data envelopment analysis in the regulation of UK water utilities: Water distribution," European Journal of Operational Research, Elsevier, vol. 126(2), pages 436-453, October.
    10. Caroline van den Berg & Alexander Danilenko, 2011. "The IBNET Water Supply and Sanitation Performance Blue Book," World Bank Publications - Books, The World Bank Group, number 2545.
    11. Anwandter, Lars & Ozuna, Teofilo Jr., 2002. "Can public sector reforms improve the efficiency of public water utilities?," Environment and Development Economics, Cambridge University Press, vol. 7(4), pages 687-700, October.
    12. James H. Stock & Mark W. Watson, 1989. "New Indexes of Coincident and Leading Economic Indicators," NBER Chapters, in: NBER Macroeconomics Annual 1989, Volume 4, pages 351-409, National Bureau of Economic Research, Inc.
    13. Durbin, James & Koopman, Siem Jan, 2012. "Time Series Analysis by State Space Methods," OUP Catalogue, Oxford University Press, edition 2, number 9780199641178.
    14. Peter Molenaar & Jan Gooijer & Bernhard Schmitz, 1992. "Dynamic factor analysis of nonstationary multivariate time series," Psychometrika, Springer;The Psychometric Society, vol. 57(3), pages 333-349, September.
    15. Luiz Koodi Hotta, 1989. "Identification Of Unobserved Components Models," Journal of Time Series Analysis, Wiley Blackwell, vol. 10(3), pages 259-270, May.
    16. Boivin, Jean & Ng, Serena, 2006. "Are more data always better for factor analysis?," Journal of Econometrics, Elsevier, vol. 132(1), pages 169-194, May.
    17. Abbott, Malcolm & Cohen, Bruce, 2009. "Productivity and efficiency in the water industry," Utilities Policy, Elsevier, vol. 17(3-4), pages 233-244, September.
    18. Roberts, W.J.J., 2014. "Factor analysis parameter estimation from incomplete data," Computational Statistics & Data Analysis, Elsevier, vol. 70(C), pages 61-66.
    19. Jushan Bai, 2003. "Inferential Theory for Factor Models of Large Dimensions," Econometrica, Econometric Society, vol. 71(1), pages 135-171, January.
    20. Peter Molenaar, 1985. "A dynamic factor model for the analysis of multivariate time series," Psychometrika, Springer;The Psychometric Society, vol. 50(2), pages 181-202, June.
    21. Donald Rubin & Dorothy Thayer, 1982. "EM algorithms for ML factor analysis," Psychometrika, Springer;The Psychometric Society, vol. 47(1), pages 69-76, March.
    22. Zhao, Jianhua & Shi, Lei, 2014. "Automated learning of factor analysis with complete and incomplete data," Computational Statistics & Data Analysis, Elsevier, vol. 72(C), pages 205-218.
    23. Bai, Jushan & Ng, Serena, 2008. "Large Dimensional Factor Analysis," Foundations and Trends(R) in Econometrics, now publishers, vol. 3(2), pages 89-163, June.
    24. Blasques, F. & Koopman, S.J. & Mallee, M. & Zhang, Z., 2016. "Weighted maximum likelihood for dynamic factor analysis and forecasting with mixed frequency data," Journal of Econometrics, Elsevier, vol. 193(2), pages 405-417.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Malin Song & Qianjiao Xie, 2021. "Evaluation of Urban Competitiveness of the Huaihe River Eco-Economic Belt Based on Dynamic Factor Analysis," Computational Economics, Springer;Society for Computational Economics, vol. 58(3), pages 615-639, October.
    2. Nikolaos Zirogiannis & Kerry Krutilla & Yorghos Tripodis & Kathryn Fledderman, 2019. "Human Development Over Time: An Empirical Comparison of a Dynamic Index and the Standard HDI," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 142(2), pages 773-798, April.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zirogiannis, Nikolaos & Tripodis, Yorghos, 2014. "Dynamic Factor Analysis for Short Panels: Estimating Performance Trajectories for Water Utilities," 2014 Annual Meeting, July 27-29, 2014, Minneapolis, Minnesota 170592, Agricultural and Applied Economics Association.
    2. Nikolaos Zirogiannis & Yorghos Tripodis, 2013. "A Generalized Dynamic Factor Model for Panel Data: Estimation with a Two-Cycle Conditional Expectation-Maximization Algorithm," Working Papers 2013-1, University of Massachusetts Amherst, Department of Resource Economics.
    3. Zirogiannis, Nikolaos & Tripodis, Yorghos, 2013. "A Generalized Dynamic Factor Model for Panel Data: Estimation with a Two-Cycle Conditional Expectation-Maximization Algorithm," Working Paper Series 142752, University of Massachusetts, Amherst, Department of Resource Economics.
    4. Stock, J.H. & Watson, M.W., 2016. "Dynamic Factor Models, Factor-Augmented Vector Autoregressions, and Structural Vector Autoregressions in Macroeconomics," Handbook of Macroeconomics, in: J. B. Taylor & Harald Uhlig (ed.), Handbook of Macroeconomics, edition 1, volume 2, chapter 0, pages 415-525, Elsevier.
    5. Poncela, Pilar & Ruiz, Esther & Miranda, Karen, 2021. "Factor extraction using Kalman filter and smoothing: This is not just another survey," International Journal of Forecasting, Elsevier, vol. 37(4), pages 1399-1425.
    6. Catherine Doz & Peter Fuleky, 2019. "Dynamic Factor Models," Working Papers 2019-4, University of Hawaii Economic Research Organization, University of Hawaii at Manoa.
    7. Karim Barhoumi & Olivier Darné & Laurent Ferrara, 2014. "Dynamic factor models: A review of the literature," OECD Journal: Journal of Business Cycle Measurement and Analysis, OECD Publishing, Centre for International Research on Economic Tendency Surveys, vol. 2013(2), pages 73-107.
    8. Pilar Poncela & Esther Ruiz, 2016. "Small- Versus Big-Data Factor Extraction in Dynamic Factor Models: An Empirical Assessment," Advances in Econometrics, in: Dynamic Factor Models, volume 35, pages 401-434, Emerald Group Publishing Limited.
    9. Kihwan Kim & Norman Swanson, 2013. "Diffusion Index Model Specification and Estimation Using Mixed Frequency Datasets," Departmental Working Papers 201315, Rutgers University, Department of Economics.
    10. Stock, James H. & Watson, Mark, 2011. "Dynamic Factor Models," Scholarly Articles 28469541, Harvard University Department of Economics.
    11. Matteo Barigozzi & Matteo Luciani, 2019. "Quasi Maximum Likelihood Estimation and Inference of Large Approximate Dynamic Factor Models via the EM algorithm," Papers 1910.03821, arXiv.org, revised Sep 2024.
    12. Rua, António, 2017. "A wavelet-based multivariate multiscale approach for forecasting," International Journal of Forecasting, Elsevier, vol. 33(3), pages 581-590.
    13. Tóth, Peter, 2014. "Malý dynamický faktorový model na krátkodobé prognózovanie slovenského HDP [A Small Dynamic Factor Model for the Short-Term Forecasting of Slovak GDP]," MPRA Paper 63713, University Library of Munich, Germany.
    14. Matteo Barigozzi, 2023. "Quasi Maximum Likelihood Estimation of High-Dimensional Factor Models: A Critical Review," Papers 2303.11777, arXiv.org, revised May 2024.
    15. James Sampi, 2016. "High Dimensional Factor Models: An Empirical Bayes Approach," Working Papers 75, Peruvian Economic Association.
    16. Fiorentini, Gabriele & Galesi, Alessandro & Sentana, Enrique, 2018. "A spectral EM algorithm for dynamic factor models," Journal of Econometrics, Elsevier, vol. 205(1), pages 249-279.
    17. Poncela, Pilar, 2021. "Dynamic factor models: does the specification matter?," DES - Working Papers. Statistics and Econometrics. WS 32210, Universidad Carlos III de Madrid. Departamento de Estadística.
    18. Luciani, Matteo, 2014. "Forecasting with approximate dynamic factor models: The role of non-pervasive shocks," International Journal of Forecasting, Elsevier, vol. 30(1), pages 20-29.
    19. Bańbura, Marta & Giannone, Domenico & Lenza, Michele, 2015. "Conditional forecasts and scenario analysis with vector autoregressions for large cross-sections," International Journal of Forecasting, Elsevier, vol. 31(3), pages 739-756.
    20. Giannone, Domenico & Reichlin, Lucrezia & Small, David, 2008. "Nowcasting: The real-time informational content of macroeconomic data," Journal of Monetary Economics, Elsevier, vol. 55(4), pages 665-676, May.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:stmapp:v:27:y:2018:i:1:d:10.1007_s10260-017-0394-y. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.