IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2005.11233.html
   My bibliography  Save this paper

Scanner data in inflation measurement: from raw data to price indices

Author

Listed:
  • Jacek Bia{l}ek
  • Maciej Berk{e}sewicz

Abstract

Scanner data offer new opportunities for CPI or HICP calculation. They can be obtained from a~wide variety of~retailers (supermarkets, home electronics, Internet shops, etc.) and provide information at the level of~the barcode. One of~advantages of~using scanner data is the fact that they contain complete transaction information, i.e. prices and quantities for every sold item. To use scanner data, it must be carefully processed. After clearing data and unifying product names, products should be carefully classified (e.g. into COICOP 5 or below), matched, filtered and aggregated. These procedures often require creating new IT or writing custom scripts (R, Python, Mathematica, SAS, others). One of~new challenges connected with scanner data is the appropriate choice of~the index formula. In this article we present a~proposal for the implementation of~individual stages of~handling scanner data. We also point out potential problems during scanner data processing and their solutions. Finally, we compare a~large number of~price index methods based on real scanner datasets and we verify their sensitivity on adopted data filtering and aggregating methods.

Suggested Citation

  • Jacek Bia{l}ek & Maciej Berk{e}sewicz, 2020. "Scanner data in inflation measurement: from raw data to price indices," Papers 2005.11233, arXiv.org.
  • Handle: RePEc:arx:papers:2005.11233
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2005.11233
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Zou, Hui, 2006. "The Adaptive Lasso and Its Oracle Properties," Journal of the American Statistical Association, American Statistical Association, vol. 101, pages 1418-1429, December.
    2. Ivancic, Lorraine & Erwin Diewert, W. & Fox, Kevin J., 2011. "Scanner data, time aggregation and the construction of price indexes," Journal of Econometrics, Elsevier, vol. 161(1), pages 24-35, March.
    3. Caves, Douglas W & Christensen, Laurits R & Diewert, W Erwin, 1982. "Multilateral Comparisons of Output, Input, and Productivity Using Superlative Index Numbers," Economic Journal, Royal Economic Society, vol. 92(365), pages 73-86, March.
    4. W. Erwin Diewert, 1999. "Axiomatic and Economic Approaches to International Comparisons," NBER Chapters, in: International and Interarea Comparisons of Income, Output, and Prices, pages 13-107, National Bureau of Economic Research, Inc.
    5. Diewert, W. Erwin & Fox, Kevin J., 2017. "Substitution Bias in Multilateral Methods for CPI Construction using Scanner Data," Microeconomics.ca working papers erwin_diewert-2017-3, Vancouver School of Economics, revised 23 Mar 2017.
    6. Friedman, Jerome H. & Hastie, Trevor & Tibshirani, Rob, 2010. "Regularization Paths for Generalized Linear Models via Coordinate Descent," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 33(i01).
    7. Diewert, W. E., 1976. "Exact and superlative index numbers," Journal of Econometrics, Elsevier, vol. 4(2), pages 115-145, May.
    8. Feenstra, Robert C. & Ma, Hong & Rao, D. S. Prasada, 2009. "Consistent Comparisons Of Real Incomes Across Time And Space," Macroeconomic Dynamics, Cambridge University Press, vol. 13(S2), pages 169-193, September.
    9. Simon, Noah & Friedman, Jerome H. & Hastie, Trevor & Tibshirani, Rob, 2011. "Regularization Paths for Cox's Proportional Hazards Model via Coordinate Descent," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 39(i05).
    10. Jan de Haan & Frances Krsinich, 2018. "Time Dummy Hedonic and Quality‐Adjusted Unit Value Indexes: Do They Really Differ?," Review of Income and Wealth, International Association for Research in Income and Wealth, vol. 64(4), pages 757-776, December.
    11. Inklaar, Robert & Diewert, W. Erwin, 2016. "Measuring industry productivity and cross-country convergence," Journal of Econometrics, Elsevier, vol. 191(2), pages 426-433.
    12. de Haan, Jan & van der Grient, Heymerik A., 2011. "Eliminating chain drift in price indexes based on scanner data," Journal of Econometrics, Elsevier, vol. 161(1), pages 36-46, March.
    13. Maddison, A. & Prasada Rao, D.S., 1996. "A generalized approach to international comparison of agricultural output and productivity," GGDC Research Memorandum 199627, Groningen Growth and Development Centre, University of Groningen.
    14. Abe, Naohito & Rao, D.S. Prasada, 2019. "Multilateral Sato–Vartia index for international comparisons of prices and real expenditures," Economics Letters, Elsevier, vol. 183(C), pages 1-1.
    15. Peter Levell, 2015. "Is the Carli index flawed?: assessing the case for the new retail price index RPIJ," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 178(2), pages 303-336, February.
    16. Sato, Kazuo, 1976. "The Ideal Log-Change Index Number," The Review of Economics and Statistics, MIT Press, vol. 58(2), pages 223-228, May.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Diewert, W. Erwin & Fox, Kevin J., 2017. "Substitution Bias in Multilateral Methods for CPI Construction using Scanner Data," Microeconomics.ca working papers erwin_diewert-2017-3, Vancouver School of Economics, revised 23 Mar 2017.
    2. Jacek Białek, 2023. "Improving quality of the scanner CPI: proposition of new multilateral methods," Quality & Quantity: International Journal of Methodology, Springer, vol. 57(3), pages 2893-2921, June.
    3. Kevin J. Fox & Peter Levell & Martin O'Connell, 2023. "Inflation measurement with high frequency data," IFS Working Papers W23/29, Institute for Fiscal Studies.
    4. W. Erwin Diewert, 2022. "Scanner Data, Elementary Price Indexes and the Chain Drift Problem," Springer Books, in: Duangkamon Chotikapanich & Alicia N. Rambaldi & Nicholas Rohde (ed.), Advances in Economic Measurement, chapter 0, pages 445-606, Springer.
    5. Diewert, Erwin, 2019. "Quality Adjustment and Hedonics: A Unified Approach," Microeconomics.ca working papers erwin_diewert-2019-2, Vancouver School of Economics, revised 14 Mar 2019.
    6. Diewert, Erwin & Marandola, Tina, 2018. "Scanner Data, Elementary Price Indexes and the Chain Drift Problem," Microeconomics.ca working papers tina_marandola-2018-9, Vancouver School of Economics, revised 10 Oct 2018.
    7. Diewert, W. Erwin & Fox, Kevin J., 2016. "Kevin J. Fox Interview of W. Erwin Diewert," Microeconomics.ca working papers erwin_diewert-2016-6, Vancouver School of Economics, revised 02 Jun 2016.
    8. Daniel Melser & Michael Webster, 2021. "Multilateral Methods, Substitution Bias, and Chain Drift: Some Empirical Comparisons," Review of Income and Wealth, International Association for Research in Income and Wealth, vol. 67(3), pages 759-785, September.
    9. Barnett, William A. & Erwin Diewert, W. & Zellner, Arnold, 2011. "Introduction to measurement with theory," Journal of Econometrics, Elsevier, vol. 161(1), pages 1-5, March.
    10. W. Erwin Diewert & Robert C. Feenstra, 2021. "Estimating the Benefits of New Products," NBER Chapters, in: Big Data for Twenty-First-Century Economic Statistics, pages 437-473, National Bureau of Economic Research, Inc.
    11. Ivancic, Lorraine & Erwin Diewert, W. & Fox, Kevin J., 2011. "Scanner data, time aggregation and the construction of price indexes," Journal of Econometrics, Elsevier, vol. 161(1), pages 24-35, March.
    12. de Haan, Jan & van der Grient, Heymerik A., 2011. "Eliminating chain drift in price indexes based on scanner data," Journal of Econometrics, Elsevier, vol. 161(1), pages 36-46, March.
    13. Adam Gorajek, 2018. "Econometric Perspectives on Economic Measurement," RBA Research Discussion Papers rdp2018-08, Reserve Bank of Australia.
    14. Kevin J, Fox. & Iqbal A. Syed, 2016. "Price Discounts and the Measurement of Inflation: Further Results," Discussion Papers 2016-05, School of Economics, The University of New South Wales.
    15. Ludwig Auer, 2012. "Räumliche Preisvergleiche: Aggregationskonzepte und Forschungsperspektiven," AStA Wirtschafts- und Sozialstatistisches Archiv, Springer;Deutsche Statistische Gesellschaft - German Statistical Society, vol. 6(1), pages 27-56, December.
    16. Daniel Melser, 2019. "Valuing the quantity and quality of product variety to consumers," Empirical Economics, Springer, vol. 57(6), pages 2107-2128, December.
    17. Fox, Kevin J. & Syed, Iqbal A., 2016. "Price discounts and the measurement of inflation," Journal of Econometrics, Elsevier, vol. 191(2), pages 398-406.
    18. Zhenkun Zhou & Zikun Song & Tao Ren, 2022. "Predicting China's CPI by Scanner Big Data," Papers 2211.16641, arXiv.org, revised Oct 2023.
    19. Sweitzer, Megan & Byrne, Anne T. & Page, Elina T. & Carlson, Andrea & Kantor, Linda & Muth, Mary K. & Karns, Shawn & Zhen, Chen, 2024. "Development of the Food-at-Home Monthly Area Prices Data," Technical Bulletins 342467, United States Department of Agriculture, Economic Research Service.
    20. Li, Qingxiao & Cakir, Metin, 2020. "Thrifty Food Plan Panel Price Index and the Real Value of SNAP Benefits," 2020 Annual Meeting, July 26-28, Kansas City, Missouri 304201, Agricultural and Applied Economics Association.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2005.11233. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.