IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v336y2023ics0306261923002155.html
   My bibliography  Save this article

A divide-and-conquer method for compression and reconstruction of smart meter data

Author

Listed:
  • Liu, Bo
  • Hou, Yufan
  • Luan, Wenpeng
  • Liu, Zishuai
  • Chen, Sheng
  • Yu, Yixin

Abstract

As smart grid sensors, smart meters generate abundant valuable data, laying the foundation for data-driven applications. However, the data collection brings huge communication pressure to electric utilities. In this context, considering that different types of devices have different power consumption patterns, and different types of data compression methods have their own applicable scenarios, we propose a divide-and-conquer method for compression and reconstruction of smart meter data. First, based on algorithm of voice activity detection (VAD), a load power fluctuation segment location method is proposed, which is combined with load event detection method to divide the load data into the event segments, fluctuation segments, and steady-state segments. Then, for the fluctuation segments, a cloud-device collaboration adaptive strategy based on the compressive sensing (CS) theory is designed, in which the sparse basis and measurement matrix are updated accordingly to ensure the high reconstruction accuracy in different scenarios. For the steady-state segments, a data compression method based on the improved symbolic aggregation approximation (SAX) is established, in which the dividing rectangle (DIRECT) algorithm and the irregular time partitioning method are combined to reduce the data volume for transmission without losing important information. For the event segments, the original data values are retained since the event power curves are relatively more complex and short duration. Finally, the received compressed data are reconstructed into the original power time series data in the master station on cloud to support advanced data analytics. Comparative experiments are conducted on the private and public datasets of 12 households in North America and China. The results show that our method has higher data reconstruction accuracy and compression efficiency compared to the existing methods.

Suggested Citation

  • Liu, Bo & Hou, Yufan & Luan, Wenpeng & Liu, Zishuai & Chen, Sheng & Yu, Yixin, 2023. "A divide-and-conquer method for compression and reconstruction of smart meter data," Applied Energy, Elsevier, vol. 336(C).
    • Handle: RePEc:eee:appene:v:336:y:2023:i:c:s0306261923002155
    DOI: 10.1016/j.apenergy.2023.120851
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261923002155
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2023.120851?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Luo, Zhe & Hong, SeungHo & Ding, YueMin, 2019. "A data mining-driven incentive-based demand response scheme for a virtual power plant," Applied Energy, Elsevier, vol. 239(C), pages 549-559.
    2. Jia, Kunqi & Guo, Ge & Xiao, Jucheng & Zhou, Huan & Wang, Zhihua & He, Guangyu, 2019. "Data compression approach for the home energy management system," Applied Energy, Elsevier, vol. 247(C), pages 643-656.
    3. Chen, Xiao & Zanocco, Chad & Flora, June & Rajagopal, Ram, 2022. "Constructing dynamic residential energy lifestyles using Latent Dirichlet Allocation," Applied Energy, Elsevier, vol. 318(C).
    4. McLoughlin, Fintan & Duffy, Aidan & Conlon, Michael, 2015. "A clustering approach to domestic electricity load profile characterisation using smart metering data," Applied Energy, Elsevier, vol. 141(C), pages 190-199.
    5. Liu, Bo & Luan, Wenpeng & Yu, Yixin, 2017. "Dynamic time warping based non-intrusive load transient identification," Applied Energy, Elsevier, vol. 195(C), pages 634-645.
    6. Zhou, Kaile & Yang, Shanlin & Shao, Zhen, 2016. "Energy Internet: The business perspective," Applied Energy, Elsevier, vol. 178(C), pages 212-222.
    7. Zhou, Kaile & Fu, Chao & Yang, Shanlin, 2016. "Big data driven smart energy management: From big data to big insights," Renewable and Sustainable Energy Reviews, Elsevier, vol. 56(C), pages 215-225.
    8. Yan, Lei & Tian, Wei & Wang, Hong & Hao, Xing & Li, Zuyi, 2023. "Robust event detection for residential load disaggregation," Applied Energy, Elsevier, vol. 331(C).
    9. Wen, Lulu & Zhou, Kaile & Yang, Shanlin & Li, Lanlan, 2018. "Compression of smart meter big data: A survey," Renewable and Sustainable Energy Reviews, Elsevier, vol. 91(C), pages 59-69.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jia, Kunqi & Guo, Ge & Xiao, Jucheng & Zhou, Huan & Wang, Zhihua & He, Guangyu, 2019. "Data compression approach for the home energy management system," Applied Energy, Elsevier, vol. 247(C), pages 643-656.
    2. Zhou, Kaile & Yang, Changhui & Shen, Jianxin, 2017. "Discovering residential electricity consumption patterns through smart-meter data mining: A case study from China," Utilities Policy, Elsevier, vol. 44(C), pages 73-84.
    3. Wen, Lulu & Zhou, Kaile & Yang, Shanlin & Li, Lanlan, 2018. "Compression of smart meter big data: A survey," Renewable and Sustainable Energy Reviews, Elsevier, vol. 91(C), pages 59-69.
    4. Das, Laya & Garg, Dinesh & Srinivasan, Babji, 2020. "NeuralCompression: A machine learning approach to compress high frequency measurements in smart grid," Applied Energy, Elsevier, vol. 257(C).
    5. Zou, Juan & Yang, Xu & Liu, Zhongbing & Liu, Jiangyang & Zhang, Ling & Zheng, Jinhua, 2021. "Multiobjective bilevel optimization algorithm based on preference selection to solve energy hub system planning problems," Energy, Elsevier, vol. 232(C).
    6. Al-Wakeel, Ali & Wu, Jianzhong & Jenkins, Nick, 2017. "k-means based load estimation of domestic smart meter measurements," Applied Energy, Elsevier, vol. 194(C), pages 333-342.
    7. Al-Wakeel, Ali & Wu, Jianzhong & Jenkins, Nick, 2016. "State estimation of medium voltage distribution networks using smart meter measurements," Applied Energy, Elsevier, vol. 184(C), pages 207-218.
    8. Dong, Lianxin & Fan, Shuai & Wang, Zhihua & Xiao, Jucheng & Zhou, Huan & Li, Zuyi & He, Guangyu, 2021. "An adaptive decentralized economic dispatch method for virtual power plant," Applied Energy, Elsevier, vol. 300(C).
    9. Vangelis Marinakis, 2020. "Big Data for Energy Management and Energy-Efficient Buildings," Energies, MDPI, vol. 13(7), pages 1-18, March.
    10. A-Ru-Han Bao & Yao Liu & Jun Dong & Zheng-Peng Chen & Zhen-Jie Chen & Chen Wu, 2022. "Evolutionary Game Analysis of Co-Opetition Strategy in Energy Big Data Ecosystem under Government Intervention," Energies, MDPI, vol. 15(6), pages 1-24, March.
    11. Chen, Wen & Zhou, Kaile & Yang, Shanlin & Wu, Cheng, 2017. "Data quality of electricity consumption data in a smart grid environment," Renewable and Sustainable Energy Reviews, Elsevier, vol. 75(C), pages 98-105.
    12. Wu, Ying & Wu, Yanpeng & Guerrero, Josep M. & Vasquez, Juan C., 2021. "A comprehensive overview of framework for developing sustainable energy internet: From things-based energy network to services-based management system," Renewable and Sustainable Energy Reviews, Elsevier, vol. 150(C).
    13. Lu, Xinhui & Liu, Zhaoxi & Ma, Li & Wang, Lingfeng & Zhou, Kaile & Feng, Nanping, 2020. "A robust optimization approach for optimal load dispatch of community energy hub," Applied Energy, Elsevier, vol. 259(C).
    14. Motlagh, Omid & Berry, Adam & O'Neil, Lachlan, 2019. "Clustering of residential electricity customers using load time series," Applied Energy, Elsevier, vol. 237(C), pages 11-24.
    15. Bossink, Bart A.G., 2017. "Demonstrating sustainable energy: A review based model of sustainable energy demonstration projects," Renewable and Sustainable Energy Reviews, Elsevier, vol. 77(C), pages 1349-1362.
    16. Wen, Hanguan & Liu, Xiufeng & Yang, Ming & Lei, Bo & Xu, Cheng & Chen, Zhe, 2024. "A novel approach for identifying customer groups for personalized demand-side management services using household socio-demographic data," Energy, Elsevier, vol. 286(C).
    17. Rongheng Lin & Budan Wu & Yun Su, 2018. "An Adaptive Weighted Pearson Similarity Measurement Method for Load Curve Clustering," Energies, MDPI, vol. 11(9), pages 1-17, September.
    18. Gianluca Trotta & Kirsten Gram-Hanssen & Pernille Lykke Jørgensen, 2020. "Heterogeneity of Electricity Consumption Patterns in Vulnerable Households," Energies, MDPI, vol. 13(18), pages 1-17, September.
    19. Cen, Xiao & Chen, Zengliang & Chen, Haifeng & Ding, Chen & Ding, Bo & Li, Fei & Lou, Fangwei & Zhu, Zhenyu & Zhang, Hongyu & Hong, Bingyuan, 2024. "User repurchase behavior prediction for integrated energy supply stations based on the user profiling method," Energy, Elsevier, vol. 286(C).
    20. Amasyali, Kadir & El-Gohary, Nora M., 2018. "A review of data-driven building energy consumption prediction studies," Renewable and Sustainable Energy Reviews, Elsevier, vol. 81(P1), pages 1192-1205.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:336:y:2023:i:c:s0306261923002155. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.