IDEAS home Printed from https://ideas.repec.org/a/ovi/oviste/vxxiiy2022i2p146-152.html
   My bibliography  Save this article

Why Is More Efficient to Combine BeautifulSoup and Selenium in Scraping For Data Under Energy Crisis

Author

Listed:
  • Simona Vasilica Oprea

    (The Bucharest University of Economic Studies, Romania)

  • Adela Bâra

    (The Bucharest University of Economic Studies, Romania)

Abstract

The electricity prices are often sensitive data that pose challenges in terms of collection. The volatility and soaring prices make the day-ahead electricity markets appealing for scientists. Several events took place from 2020, such as COVID-19 and military conflict in Ukraine leading to higher inflation, interest rates and energy crisis. Since 2020, the electricity prices on day-ahead market (DAM) increased even up to ten times. To perform the electricity price prediction, extensive feature engineering and historical data are required. Data sources from Romania are assessed to analyze the opportunity to extract relevant data for the electricity price and traded quantities. Thus, in this, paper, we investigate opportunity to extract data from web pages. We will suggest solutions to extract historical data from web sites that do not provide APIs or csv files. Several Python libraries such as BeautifulSoup and Selenium will be showcased, and approaches will be compared.

Suggested Citation

  • Simona Vasilica Oprea & Adela Bâra, 2022. "Why Is More Efficient to Combine BeautifulSoup and Selenium in Scraping For Data Under Energy Crisis," Ovidius University Annals, Economic Sciences Series, Ovidius University of Constantza, Faculty of Economic Sciences, vol. 0(2), pages 146-152, Decembrie.
    • Handle: RePEc:ovi:oviste:v:xxii:y:2022:i:2:p:146-152
    as

    Download full text from publisher

    File URL: https://stec.univ-ovidius.ro/html/anale/RO/2022-issue2/Section%201%20and%202/19.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Bevaola Kusumasari & Nias Phydra Aji Prabowo, 2020. "Scraping social media data for disaster communication: how the pattern of Twitter users affects disasters in Asia and the Pacific," Natural Hazards: Journal of the International Society for the Prevention and Mitigation of Natural Hazards, Springer;International Society for the Prevention and Mitigation of Natural Hazards, vol. 103(3), pages 3415-3435, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Seungil Yum, 2023. "Analyses of human responses to Winter storm Kai using the GWR model," Natural Hazards: Journal of the International Society for the Prevention and Mitigation of Natural Hazards, Springer;International Society for the Prevention and Mitigation of Natural Hazards, vol. 116(2), pages 1805-1821, March.

    More about this item

    Keywords

    electricity price; data scrapin; BeautifulSoup; Selenium; prediction;
    All these keywords.

    JEL classification:

    • Q47 - Agricultural and Natural Resource Economics; Environmental and Ecological Economics - - Energy - - - Energy Forecasting
    • C82 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Macroeconomic Data; Data Access
    • Q41 - Agricultural and Natural Resource Economics; Environmental and Ecological Economics - - Energy - - - Demand and Supply; Prices

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ovi:oviste:v:xxii:y:2022:i:2:p:146-152. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Gheorghiu Gabriela (email available below). General contact details of provider: https://edirc.repec.org/data/feoviro.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.