IDEAS home Printed from https://ideas.repec.org/a/gam/jeners/v16y2023i15p5689-d1205941.html
   My bibliography  Save this article

Proximal Policy Optimization for Energy Management of Electric Vehicles and PV Storage Units

Author

Listed:
  • Monica Alonso

    (Department of Electrical Engineering, Universidad Carlos III de Madrid, 28911 Leganes, Spain)

  • Hortensia Amaris

    (Department of Electrical Engineering, Universidad Carlos III de Madrid, 28911 Leganes, Spain)

  • David Martin

    (Department of Electrical Engineering, Universidad Carlos III de Madrid, 28911 Leganes, Spain)

  • Arturo de la Escalera

    (Department of Electrical Engineering, Universidad Carlos III de Madrid, 28911 Leganes, Spain)

Abstract

Connected autonomous electric vehicles (CAEVs) are essential actors in the decarbonization process of the transport sector and a key aspect of home energy management systems (HEMSs) along with PV units, CAEVs and battery energy storage systems. However, there are associated uncertainties which present new challenges to HEMSs, such as aleatory EV arrival and departure times, unknown EV battery states of charge at the connection time, and stochastic PV production due to weather and passing cloud conditions. The proposed HEMS is based on proximal policy optimization (PPO), which is a deep reinforcement learning algorithm suitable for continuous complex environments. The optimal solution for HEMS is a tradeoff between CAEV driver’s range anxiety, batteries degradation, and energy consumption, which is solved by means of incentives/penalties in the reinforcement learning formulation. The proposed PPO algorithm was compared to conventional methods such as business-as-usual (BAU) and value iteration (VI) solutions based on dynamic programming. Simulation results indicate that the proposed PPO’s performance showed a daily energy cost reduction of 54% and 27% compared to BAU and VI, respectively. Finally, the developed PPO algorithm is suitable for real-time operations due to its fast execution and good convergence to the optimal solution.

Suggested Citation

  • Monica Alonso & Hortensia Amaris & David Martin & Arturo de la Escalera, 2023. "Proximal Policy Optimization for Energy Management of Electric Vehicles and PV Storage Units," Energies, MDPI, vol. 16(15), pages 1-20, July.
    • Handle: RePEc:gam:jeners:v:16:y:2023:i:15:p:5689-:d:1205941
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1996-1073/16/15/5689/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1996-1073/16/15/5689/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Timo Kern & Patrick Dossow & Serafin von Roon, 2020. "Integrating Bidirectionally Chargeable Electric Vehicles into the Electricity Markets," Energies, MDPI, vol. 13(21), pages 1-30, November.
    2. Lund, Henrik & Kempton, Willett, 2008. "Integration of renewable energy into the transport and electricity sectors through V2G," Energy Policy, Elsevier, vol. 36(9), pages 3578-3587, September.
    3. Jian, Linni & Zheng, Yanchong & Xiao, Xinping & Chan, C.C., 2015. "Optimal scheduling for vehicle-to-grid operation with stochastic connection of plug-in electric vehicles to smart grid," Applied Energy, Elsevier, vol. 146(C), pages 150-161.
    4. Adil Amin & Wajahat Ullah Khan Tareen & Muhammad Usman & Haider Ali & Inam Bari & Ben Horan & Saad Mekhilef & Muhammad Asif & Saeed Ahmed & Anzar Mahmood, 2020. "A Review of Optimal Charging Strategy for Electric Vehicles under Dynamic Pricing Schemes in the Distribution Charging Network," Sustainability, MDPI, vol. 12(23), pages 1-28, December.
    5. Connor Scott & Mominul Ahsan & Alhussein Albarbar, 2021. "Machine Learning Based Vehicle to Grid Strategy for Improving the Energy Performance of Public Buildings," Sustainability, MDPI, vol. 13(7), pages 1-22, April.
    6. Antimo Barbato & Antonio Capone, 2014. "Optimization Models and Methods for Demand-Side Management of Residential Users: A Survey," Energies, MDPI, vol. 7(9), pages 1-38, September.
    7. Cedillo, Mónica Hernández & Sun, Hongjian & Jiang, Jing & Cao, Yue, 2022. "Dynamic pricing and control for EV charging stations with solar generation," Applied Energy, Elsevier, vol. 326(C).
    8. Lee, Sangyoon & Choi, Dae-Hyun, 2021. "Dynamic pricing and energy management for profit maximization in multiple smart electric vehicle charging stations: A privacy-preserving deep reinforcement learning approach," Applied Energy, Elsevier, vol. 304(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Qin Chen & Komla Agbenyo Folly, 2022. "Application of Artificial Intelligence for EV Charging and Discharging Scheduling and Dynamic Pricing: A Review," Energies, MDPI, vol. 16(1), pages 1-26, December.
    2. Mehrdad Tarafdar-Hagh & Kamran Taghizad-Tavana & Mohsen Ghanbari-Ghalehjoughi & Sayyad Nojavan & Parisa Jafari & Amin Mohammadpour Shotorbani, 2023. "Optimizing Electric Vehicle Operations for a Smart Environment: A Comprehensive Review," Energies, MDPI, vol. 16(11), pages 1-21, May.
    3. Kang Miao Tan & Vigna K. Ramachandaramurthy & Jia Ying Yong & Sanjeevikumar Padmanaban & Lucian Mihet-Popa & Frede Blaabjerg, 2017. "Minimization of Load Variance in Power Grids—Investigation on Optimal Vehicle-to-Grid Scheduling," Energies, MDPI, vol. 10(11), pages 1-21, November.
    4. Signer, Tim & Baumgartner, Nora & Ruppert, Manuel & Sandmeier, Thorben & Fichtner, Wolf, 2024. "Modeling V2G spot market trading: The impact of charging tariffs on economic viability," Energy Policy, Elsevier, vol. 189(C).
    5. Nnaemeka Vincent Emodi & Scott Dwyer & Kriti Nagrath & John Alabi, 2022. "Electromobility in Australia: Tariff Design Structure and Consumer Preferences for Mobile Distributed Energy Storage," Sustainability, MDPI, vol. 14(11), pages 1-18, May.
    6. Md. Rayid Hasan Mojumder & Fahmida Ahmed Antara & Md. Hasanuzzaman & Basem Alamri & Mohammad Alsharef, 2022. "Electric Vehicle-to-Grid (V2G) Technologies: Impact on the Power Grid and Battery," Sustainability, MDPI, vol. 14(21), pages 1-53, October.
    7. Zhao, Yang & Noori, Mehdi & Tatari, Omer, 2016. "Vehicle to Grid regulation services of electric delivery trucks: Economic and environmental benefit analysis," Applied Energy, Elsevier, vol. 170(C), pages 161-175.
    8. Mubbashir Ali & Jussi Ekström & Matti Lehtonen, 2018. "Sizing Hydrogen Energy Storage in Consideration of Demand Response in Highly Renewable Generation Power Systems," Energies, MDPI, vol. 11(5), pages 1-11, May.
    9. Kriegler, Elmar, 2011. "Comment," Energy Economics, Elsevier, vol. 33(4), pages 594-596, July.
    10. Bogdanov, Dmitrii & Breyer, Christian, 2024. "Role of smart charging of electric vehicles and vehicle-to-grid in integrated renewables-based energy systems on country level," Energy, Elsevier, vol. 301(C).
    11. Østergaard, P.A. & Lund, H. & Thellufsen, J.Z. & Sorknæs, P. & Mathiesen, B.V., 2022. "Review and validation of EnergyPLAN," Renewable and Sustainable Energy Reviews, Elsevier, vol. 168(C).
    12. Maria Taljegard & Lisa Göransson & Mikael Odenberger & Filip Johnsson, 2021. "To Represent Electric Vehicles in Electricity Systems Modelling—Aggregated Vehicle Representation vs. Individual Driving Profiles," Energies, MDPI, vol. 14(3), pages 1-25, January.
    13. Zhang, Qi & Mclellan, Benjamin C. & Tezuka, Tetsuo & Ishihara, Keiichi N., 2013. "A methodology for economic and environmental analysis of electric vehicles with different operational conditions," Energy, Elsevier, vol. 61(C), pages 118-127.
    14. Alqahtani, Mohammed & Hu, Mengqi, 2022. "Dynamic energy scheduling and routing of multiple electric vehicles using deep reinforcement learning," Energy, Elsevier, vol. 244(PA).
    15. Abolhosseini, Shahrouz & Heshmati, Almas & Altmann, Jörn, 2014. "A Review of Renewable Energy Supply and Energy Efficiency Technologies," IZA Discussion Papers 8145, Institute of Labor Economics (IZA).
    16. Tan, Kang Miao & Padmanaban, Sanjeevikumar & Yong, Jia Ying & Ramachandaramurthy, Vigna K., 2019. "A multi-control vehicle-to-grid charger with bi-directional active and reactive power capabilities for power grid support," Energy, Elsevier, vol. 171(C), pages 1150-1163.
    17. Thijs Klauw & Marco E. T. Gerards & Johann L. Hurink, 2017. "Resource allocation problems in decentralized energy management," OR Spectrum: Quantitative Approaches in Management, Springer;Gesellschaft für Operations Research e.V., vol. 39(3), pages 749-773, July.
    18. Simona-Vasilica Oprea & Adela Bâra & Adriana Reveiu, 2018. "Informatics Solution for Energy Efficiency Improvement and Consumption Management of Householders," Energies, MDPI, vol. 11(1), pages 1-31, January.
    19. Lucio Ciabattoni & Stefano Cardarelli & Marialaura Di Somma & Giorgio Graditi & Gabriele Comodi, 2021. "A Novel Open-Source Simulator Of Electric Vehicles in a Demand-Side Management Scenario," Energies, MDPI, vol. 14(6), pages 1-16, March.
    20. Liu, Wen & Hu, Weihao & Lund, Henrik & Chen, Zhe, 2013. "Electric vehicles and large-scale integration of wind power – The case of Inner Mongolia in China," Applied Energy, Elsevier, vol. 104(C), pages 445-456.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jeners:v:16:y:2023:i:15:p:5689-:d:1205941. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.