IDEAS home Printed from https://ideas.repec.org/a/eee/phsmap/v597y2022ics0378437122002291.html
   My bibliography  Save this article

On the application of deep learning and multifractal techniques to classify emotions and instruments using Indian Classical Music

Author

Listed:
  • Nag, Sayan
  • Basu, Medha
  • Sanyal, Shankha
  • Banerjee, Archi
  • Ghosh, Dipak

Abstract

Music is often considered as the language of emotions. The way it stimulates the emotional appraisal across people from different communities, culture and demographics has long been known and hence categorizing on the basis of emotions is indeed an intriguing basic research area. Indian Classical Music (ICM) is famous for its ambiguous nature, i.e. its ability to evoke a number of mixed emotions through only a single musical narration, and hence classifying evoked emotions from ICM becomes a more challenging task. With the rapid advancements in the field of Deep Learning, this Music Emotion Recognition (MER) task is becoming more and more relevant and robust, hence can be applied to one of the most challenging test case i.e. classifying emotions elicited from ICM. In this paper we present a new dataset called JUMusEmoDB which presently has 1600 audio clips (approximately 30 s each) where 400 clips each correspond to happy, sad, calm and anxiety emotional scales. The initial annotations and emotional classification of the database was done based on an emotional rating test (5-point Likert scale) performed by 100 participants. The clips have been taken from different conventional ‘raga’ renditions played in two Indian stringed instruments – sitar and sarod by eminent maestros of ICM and digitized in 44.1 kHz frequency. The ragas, which are unique to ICM, are described as musical structures capable of inducing different moods or emotions. For supervised classification purposes, we have used Convolutional Neural Network (CNN) based architectures (resnet50, mobilenet v2.0, squeezenet v1.0 and a proposed ODE-Net) on corresponding music spectrograms of the 6400 sub-clips (where every clip was segmented into 4 sub-clips) which contain both time as well as frequency domain information. Along with emotion classification, instrument classification based response was also attempted on the same dataset using the CNN based architectures. In this context, a nonlinear technique, Multifractal Detrended Fluctuation Analysis (MFDFA) was also applied on the musical clips to classify them on the basis of complexity values extracted from the method. The initial classification accuracy obtained from the applied methods are quite inspiring and have been corroborated with ANOVA results to determine the statistical significance. This type of CNN based classification algorithm using a rich corpus of Indian Classical Music is unique even in the global perspective and can be replicated in other modalities of music also. The link to this newly developed dataset has been provided in the dataset description section of the paper. This dataset is still under development and we plan to include more data containing other emotional as well as instrumental entities into consideration.

Suggested Citation

  • Nag, Sayan & Basu, Medha & Sanyal, Shankha & Banerjee, Archi & Ghosh, Dipak, 2022. "On the application of deep learning and multifractal techniques to classify emotions and instruments using Indian Classical Music," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 597(C).
  • Handle: RePEc:eee:phsmap:v:597:y:2022:i:c:s0378437122002291
    DOI: 10.1016/j.physa.2022.127261
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0378437122002291
    Download Restriction: Full text for ScienceDirect subscribers only. Journal offers the option of making the article available online on Science direct for a fee of $3,000

    File URL: https://libkey.io/10.1016/j.physa.2022.127261?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Banerjee, Archi & Sanyal, Shankha & Roy, Souparno & Nag, Sayan & Sengupta, Ranjan & Ghosh, Dipak, 2021. "A novel study on perception–cognition scenario in music using deterministic and non-deterministic approach," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 567(C).
    2. Sanyal, Shankha & Banerjee, Archi & Patranabis, Anirban & Banerjee, Kaushik & Sengupta, Ranjan & Ghosh, Dipak, 2016. "A study on Improvisation in a Musical performance using Multifractal Detrended Cross Correlation Analysis," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 462(C), pages 67-83.
    3. Kantelhardt, Jan W. & Zschiegner, Stephan A. & Koscielny-Bunde, Eva & Havlin, Shlomo & Bunde, Armin & Stanley, H.Eugene, 2002. "Multifractal detrended fluctuation analysis of nonstationary time series," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 316(1), pages 87-114.
    4. Stanislaw Drozdz & Jaroslaw Kwapien & Pawel Oswiecimka & Rafal Rak, 2009. "Quantitative features of multifractal subtleties in time series," Papers 0907.2866, arXiv.org, revised Feb 2010.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Bowen Zhang & Jinping Lin & Man Luo & Changxian Zeng & Jiajia Feng & Meiqi Zhou & Fuying Deng, 2022. "Changes in Public Sentiment under the Background of Major Emergencies—Taking the Shanghai Epidemic as an Example," IJERPH, MDPI, vol. 19(19), pages 1-20, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kristjanpoller, Werner & Nekhili, Ramzi & Bouri, Elie, 2024. "Blockchain ETFs and the cryptocurrency and Nasdaq markets: Multifractal and asymmetric cross-correlations," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 637(C).
    2. Meo, Marcos M. & Iaconis, Francisco R. & Del Punta, Jessica A. & Delrieux, Claudio A. & Gasaneo, Gustavo, 2024. "Multifractal information on reading eye tracking data," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 638(C).
    3. Marcin Wk{a}torek & Stanis{l}aw Dro.zd.z & Jaros{l}aw Kwapie'n & Ludovico Minati & Pawe{l} O'swik{e}cimka & Marek Stanuszek, 2020. "Multiscale characteristics of the emerging global cryptocurrency market," Papers 2010.15403, arXiv.org, revised Mar 2021.
    4. Schadner, Wolfgang, 2022. "U.S. Politics from a multifractal perspective," Chaos, Solitons & Fractals, Elsevier, vol. 155(C).
    5. Zhan, Cun & Liang, Chuan & Zhao, Lu & Jiang, Shouzheng & Niu, Kaijie & Zhang, Yaling, 2023. "Multifractal characteristics of multiscale drought in the Yellow River Basin, China," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 609(C).
    6. Wang, Jian & Jiang, Wenjing & Wu, Xinpei & Yang, Mengdie & Shao, Wei, 2023. "Role of vaccine in fighting the variants of COVID-19," Chaos, Solitons & Fractals, Elsevier, vol. 168(C).
    7. Banerjee, Archi & Sanyal, Shankha & Roy, Souparno & Nag, Sayan & Sengupta, Ranjan & Ghosh, Dipak, 2021. "A novel study on perception–cognition scenario in music using deterministic and non-deterministic approach," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 567(C).
    8. Kristjanpoller, Werner & Minutolo, Marcel C., 2021. "Asymmetric multi-fractal cross-correlations of the price of electricity in the US with crude oil and the natural gas," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 572(C).
    9. Liu, Chenggong & Shang, Pengjian & Feng, Guochen, 2017. "The high order dispersion analysis based on first-passage-time probability in financial markets," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 471(C), pages 1-9.
    10. Choi, Sun-Yong, 2021. "Analysis of stock market efficiency during crisis periods in the US stock market: Differences between the global financial crisis and COVID-19 pandemic," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 574(C).
    11. Olivares, Felipe & Zanin, Massimiliano, 2022. "Corrupted bifractal features in finite uncorrelated power-law distributed data," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 603(C).
    12. Gündüz, Güngör, 2023. "Entropy, energy, and instability in music," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 609(C).
    13. Lee, Min-Jae & Choi, Sun-Yong, 2024. "Insights into the dynamics of market efficiency spillover of financial assets in different equity markets," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 641(C).
    14. López, J.L. & Veleva, L., 2022. "2D-DFA as a tool for non-destructive characterisation of copper surface exposed to substitute ocean water," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 586(C).
    15. Kelty-Stephen, Damian G. & Mangalam, Madhur, 2024. "Additivity suppresses multifractal nonlinearity due to multiplicative cascade dynamics," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 637(C).
    16. İşcanoğlu-Çekiç, Ayşegül & Gülteki̇n, Havva, 2019. "Are cross-correlations between Turkish Stock Exchange and three major country indices multifractal or monofractal?," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 525(C), pages 978-990.
    17. Lavička, Hynek & Kracík, Jiří, 2020. "Fluctuation analysis of electric power loads in Europe: Correlation multifractality vs. Distribution function multifractality," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 545(C).
    18. L. Telesca & T. Matcharashvili & T. Chelidze & N. Zhukova & Z. Javakhishvili, 2015. "Investigating the dynamical features of the time distribution of the reservoir-induced seismicity in Enguri area (Georgia)," Natural Hazards: Journal of the International Society for the Prevention and Mitigation of Natural Hazards, Springer;International Society for the Prevention and Mitigation of Natural Hazards, vol. 77(1), pages 117-125, May.
    19. Kakinaka, Shinji & Umeno, Ken, 2021. "Exploring asymmetric multifractal cross-correlations of price–volatility and asymmetric volatility dynamics in cryptocurrency markets," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 581(C).
    20. Longfeng Zhao & Wei Li & Andrea Fenu & Boris Podobnik & Yougui Wang & H. Eugene Stanley, 2017. "The q-dependent detrended cross-correlation analysis of stock market," Papers 1705.01406, arXiv.org, revised Jun 2017.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:phsmap:v:597:y:2022:i:c:s0378437122002291. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/physica-a-statistical-mechpplications/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.