IDEAS home Printed from https://ideas.repec.org/a/spr/infosf/v23y2021i5d10.1007_s10796-021-10107-x.html
   My bibliography  Save this article

A Hybrid Approach of Machine Learning and Lexicons to Sentiment Analysis: Enhanced Insights from Twitter Data of Natural Disasters

Author

Listed:
  • Shalak Mendon

    (Electronic City
    Indian Institute of Technology Bombay)

  • Pankaj Dutta

    (Indian Institute of Technology Bombay)

  • Abhishek Behl

    (Indian Institute of Technology Bombay)

  • Stefan Lessmann

    (Humboldt-Universität zu Berlin)

Abstract

The success factor of sentimental analysis lies in identifying the most occurring and relevant opinions among users relating to the particular topic. In this paper, we develop a framework to analyze users’ sentiments on Twitter on natural disasters using the data pre-processing techniques and a hybrid of machine learning, statistical modeling, and lexicon-based approach. We choose TF-IDF and K-means for sentiment classification among affinitive and hierarchical clustering. Latent Dirichlet Allocation, a pipeline of Doc2Vec and K-means used to capture themes, then perform multi-level polarity indices classification and its time series analysis. In our study, we draw insights from 243,746 tweets for Kerala’s 2018 natural disasters in India. The key findings of the study are the classification of sentiments based on similarity and polarity indices and identifying themes among the topics discussed on Twitter. We observe different sets of emotions and influencers, among others. Through this case example of Kerala floods, it shows how the government and other organizations could track the positive/negative sentiments concerning time and location; gain a better understanding of the topic of discussion trending among the public, and collaborate with crucial Twitter users/influencers to spread and figure out the gaps in the implementation of schemes in terms of design and execution. This research’s uniqueness is the streamlined and efficient combination of algorithms and techniques embedded in the framework used in achieving the above output, which can be integrated into a platform with GUI for further automation.

Suggested Citation

  • Shalak Mendon & Pankaj Dutta & Abhishek Behl & Stefan Lessmann, 2021. "A Hybrid Approach of Machine Learning and Lexicons to Sentiment Analysis: Enhanced Insights from Twitter Data of Natural Disasters," Information Systems Frontiers, Springer, vol. 23(5), pages 1145-1168, September.
    • Handle: RePEc:spr:infosf:v:23:y:2021:i:5:d:10.1007_s10796-021-10107-x
    DOI: 10.1007/s10796-021-10107-x
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s10796-021-10107-x
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s10796-021-10107-x?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Vomfell, Lara & Härdle, Wolfgang Karl & Lessmann, Stefan, 2018. "Improving Crime Count Forecasts Using Twitter and Taxi Data," IRTG 1792 Discussion Papers 2018-013, Humboldt University of Berlin, International Research Training Group 1792 "High Dimensional Nonstationary Time Series".
    2. Tamal Mondal & Prithviraj Pramanik & Indrajit Bhattacharya & Naiwrita Boral & Saptarshi Ghosh, 2018. "Analysis and Early Detection of Rumors in a Post Disaster Scenario," Information Systems Frontiers, Springer, vol. 20(5), pages 961-979, October.
    3. Kawaljeet Kaur Kapoor & Kuttimani Tamilmani & Nripendra P. Rana & Pushp Patil & Yogesh K. Dwivedi & Sridhar Nerur, 2018. "Advances in Social Media Research: Past, Present and Future," Information Systems Frontiers, Springer, vol. 20(3), pages 531-558, June.
    4. Lingzi Hong & Cheng Fu & Jiahui Wu & Vanessa Frias-Martinez, 2018. "Information Needs and Communication Gaps between Citizens and Local Governments Online during Natural Disasters," Information Systems Frontiers, Springer, vol. 20(5), pages 1027-1039, October.
    5. Ayan Bandyopadhyay & Debasis Ganguly & Mandar Mitra & Sanjoy Kumar Saha & Gareth J.F. Jones, 2018. "An Embedding Based IR Model for Disaster Situations," Information Systems Frontiers, Springer, vol. 20(5), pages 925-932, October.
    6. Juheng Zhang & Selwyn Piramuthu, 2018. "Product recommendation with latent review topics," Information Systems Frontiers, Springer, vol. 20(3), pages 617-625, June.
    7. Xiaomo Liu & G. Alan Wang & Aditya Johri & Mi Zhou & Weiguo Fan, 2014. "Harnessing global expertise: A comparative study of expertise profiling methods for online communities," Information Systems Frontiers, Springer, vol. 16(4), pages 715-727, September.
    8. Babak Abedin & Abdul Babar, 2018. "Institutional vs. Non-institutional use of Social Media during Emergency Response: A Case of Twitter in 2014 Australian Bush Fire," Information Systems Frontiers, Springer, vol. 20(4), pages 729-740, August.
    9. Fang Liu & Dongming Xu, 2018. "Social Roles and Consequences in Using Social Media in Disasters: a Structurational Perspective," Information Systems Frontiers, Springer, vol. 20(4), pages 693-711, August.
    10. Koustav Rudra & Ashish Sharma & Niloy Ganguly & Muhammad Imran, 2018. "Classifying and Summarizing Information from Microblogs During Epidemics," Information Systems Frontiers, Springer, vol. 20(5), pages 933-948, October.
    11. Juheng Zhang & Selwyn Piramuthu, 0. "Product recommendation with latent review topics," Information Systems Frontiers, Springer, vol. 0, pages 1-9.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Wajdi Aljedaani & Eysha Saad & Furqan Rustam & Isabel de la Torre Díez & Imran Ashraf, 2022. "Role of Artificial Intelligence for Analysis of COVID-19 Vaccination-Related Tweets: Opportunities, Challenges, and Future Trends," Mathematics, MDPI, vol. 10(17), pages 1-33, September.
    2. Paras Bhatt & Naga Vemprala & Rohit Valecha & Govind Hariharan & H. Raghav Rao, 2023. "User Privacy, Surveillance and Public Health during COVID-19 – An Examination of Twitterverse," Information Systems Frontiers, Springer, vol. 25(5), pages 1667-1682, October.
    3. Peng Xie, 2022. "The Interplay Between Investor Activity on Virtual Investment Community and the Trading Dynamics: Evidence From the Bitcoin Market," Information Systems Frontiers, Springer, vol. 24(4), pages 1287-1303, August.
    4. Abhinav Kumar & Jyoti Prakash Singh & Nripendra P. Rana & Yogesh K. Dwivedi, 2023. "Multi-Channel Convolutional Neural Network for the Identification of Eyewitness Tweets of Disaster," Information Systems Frontiers, Springer, vol. 25(4), pages 1589-1604, August.
    5. Yuko Murayama & Hans Jochen Scholl & Dimiter Velev, 2021. "Information Technology in Disaster Risk Reduction," Information Systems Frontiers, Springer, vol. 23(5), pages 1077-1081, September.
    6. Jyoti Choudrie & Shruti Patil & Ketan Kotecha & Nikhil Matta & Ilias Pappas, 2021. "Applying and Understanding an Advanced, Novel Deep Learning Approach: A Covid 19, Text Based, Emotions Analysis Study," Information Systems Frontiers, Springer, vol. 23(6), pages 1431-1465, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Saptarshi Ghosh & Kripabandhu Ghosh & Debasis Ganguly & Tanmoy Chakraborty & Gareth J. F. Jones & Marie-Francine Moens & Muhammad Imran, 2018. "Exploitation of Social Media for Emergency Relief and Preparedness: Recent Research and Trends," Information Systems Frontiers, Springer, vol. 20(5), pages 901-907, October.
    2. Yanxin Wang & Jian Li & Xi Zhao & Gengzhong Feng & Xin (Robert) Luo, 2020. "Using Mobile Phone Data for Emergency Management: a Systematic Literature Review," Information Systems Frontiers, Springer, vol. 22(6), pages 1539-1559, December.
    3. Fatuma Namisango & Kyeong Kang & Ghassan Beydoun, 2022. "How the Structures Provided by Social Media Enable Collaborative Outcomes: A Study of Service Co-creation in Nonprofits," Information Systems Frontiers, Springer, vol. 24(2), pages 517-535, April.
    4. Milad Mirbabaie & Deborah Bunker & Stefan Stieglitz & Annika Deubel, 0. "Who Sets the Tone? Determining the Impact of Convergence Behaviour Archetypes in Social Media Crisis Communication," Information Systems Frontiers, Springer, vol. 0, pages 1-13.
    5. Jyoti Prakash Singh & Abhinav Kumar & Nripendra P. Rana & Yogesh K. Dwivedi, 2022. "Attention-Based LSTM Network for Rumor Veracity Estimation of Tweets," Information Systems Frontiers, Springer, vol. 24(2), pages 459-474, April.
    6. Qingqi Long & Ke Song, 2022. "Operational Performance Evaluation of E-government Microblogs Under Emergencies Based on a DEA Method," Information Systems Frontiers, Springer, vol. 24(5), pages 1-18, October.
    7. Milad Mirbabaie & Deborah Bunker & Stefan Stieglitz & Annika Deubel, 2020. "Who Sets the Tone? Determining the Impact of Convergence Behaviour Archetypes in Social Media Crisis Communication," Information Systems Frontiers, Springer, vol. 22(2), pages 339-351, April.
    8. Hung-Pin Shih & Pei-Chen Sung, 2021. "Addressing the Review-Based Learning and Private Information Approaches to Foster Platform Continuance," Information Systems Frontiers, Springer, vol. 23(3), pages 649-661, June.
    9. Ghassan Beydoun & Sergiu Dascalu & Dale Dominey-Howes & Andrew Sheehan, 2018. "Disaster Management and Information Systems: Insights to Emerging Challenges," Information Systems Frontiers, Springer, vol. 20(4), pages 649-652, August.
    10. Abhinav Kumar & Jyoti Prakash Singh & Nripendra P. Rana & Yogesh K. Dwivedi, 2023. "Multi-Channel Convolutional Neural Network for the Identification of Eyewitness Tweets of Disaster," Information Systems Frontiers, Springer, vol. 25(4), pages 1589-1604, August.
    11. Luvai Motiwalla & Amit V. Deokar & Surendra Sarnikar & Angelika Dimoka, 2019. "Leveraging Data Analytics for Behavioral Research," Information Systems Frontiers, Springer, vol. 21(4), pages 735-742, August.
    12. Paras Bhatt & Naga Vemprala & Rohit Valecha & Govind Hariharan & H. Raghav Rao, 2023. "User Privacy, Surveillance and Public Health during COVID-19 – An Examination of Twitterverse," Information Systems Frontiers, Springer, vol. 25(5), pages 1667-1682, October.
    13. Peng Xie, 2022. "The Interplay Between Investor Activity on Virtual Investment Community and the Trading Dynamics: Evidence From the Bitcoin Market," Information Systems Frontiers, Springer, vol. 24(4), pages 1287-1303, August.
    14. Carine Dominguez-Péry & Rana Tassabehji & Lakshmi Narasimha Raju Vuddaraju & Vikhram Kofi Duffour, 2021. "Improving emergency response operations in maritime accidents using social media with big data analytics: a case study of the MV Wakashio disaster," Post-Print hal-04021179, HAL.
    15. Mohammadreza Mousavizadeh & Mehrdad Koohikamali & Mohammad Salehan & Dam J. Kim, 2022. "An Investigation of Peripheral and Central Cues of Online Customer Review Voting and Helpfulness through the Lens of Elaboration Likelihood Model," Information Systems Frontiers, Springer, vol. 24(1), pages 211-231, February.
    16. Irina Wedel & Michael Palk & Stefan Voß, 2022. "A Bilingual Comparison of Sentiment and Topics for a Product Event on Twitter," Information Systems Frontiers, Springer, vol. 24(5), pages 1635-1646, October.
    17. Naveena Prakasam & Louisa Huxtable-Thomas, 2021. "Reddit: Affordances as an Enabler for Shifting Loyalties," Information Systems Frontiers, Springer, vol. 23(3), pages 723-751, June.
    18. Bo Yang & Chao Liu & Xusen Cheng & Xi Ma, 2022. "Understanding Users' Group Behavioral Decisions About Sharing Articles in Social Media: An Elaboration Likelihood Model Perspective," Group Decision and Negotiation, Springer, vol. 31(4), pages 819-842, August.
    19. Stefano Di Lauro & Aizhan Tursunbayeva & Gilda Antonelli & Marcello Martinez, 2021. "Organizational and Corporate Identity on Social Media: A Literature Review," International Journal of Business and Management, Canadian Center of Science and Education, vol. 15(4), pages 1-53, July.
    20. Aladwani, Adel M. & Dwivedi, Yogesh K., 2018. "Towards a theory of SocioCitizenry: Quality anticipation, trust configuration, and approved adaptation of governmental social media," International Journal of Information Management, Elsevier, vol. 43(C), pages 261-272.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:infosf:v:23:y:2021:i:5:d:10.1007_s10796-021-10107-x. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.