IDEAS home Printed from https://ideas.repec.org/a/gam/jftint/v12y2020i8p133-d396680.html
   My bibliography  Save this article

Multimodal Deep Learning for Group Activity Recognition in Smart Office Environments

Author

Listed:
  • George Albert Florea

    (Department of Computer Science, Malmö University, 20506 Malmö, Sweden)

  • Radu-Casian Mihailescu

    (Department of Computer Science, Malmö University, 20506 Malmö, Sweden
    Internet of Things and People Research Center, Malmö University, 20506 Malmö, Sweden)

Abstract

Deep learning (DL) models have emerged in recent years as the state-of-the-art technique across numerous machine learning application domains. In particular, image processing-related tasks have seen a significant improvement in terms of performance due to increased availability of large datasets and extensive growth of computing power. In this paper we investigate the problem of group activity recognition in office environments using a multimodal deep learning approach, by fusing audio and visual data from video. Group activity recognition is a complex classification task, given that it extends beyond identifying the activities of individuals, by focusing on the combinations of activities and the interactions between them. The proposed fusion network was trained based on the audio–visual stream from the AMI Corpus dataset. The procedure consists of two steps. First, we extract a joint audio–visual feature representation for activity recognition, and second, we account for the temporal dependencies in the video in order to complete the classification task. We provide a comprehensive set of experimental results showing that our proposed multimodal deep network architecture outperforms previous approaches, which have been designed for unimodal analysis, on the aforementioned AMI dataset.

Suggested Citation

  • George Albert Florea & Radu-Casian Mihailescu, 2020. "Multimodal Deep Learning for Group Activity Recognition in Smart Office Environments," Future Internet, MDPI, vol. 12(8), pages 1-13, August.
  • Handle: RePEc:gam:jftint:v:12:y:2020:i:8:p:133-:d:396680
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1999-5903/12/8/133/pdf
    Download Restriction: no

    File URL: https://www.mdpi.com/1999-5903/12/8/133/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Ahmadi-Karvigh, Simin & Ghahramani, Ali & Becerik-Gerber, Burcin & Soibelman, Lucio, 2018. "Real-time activity recognition for energy efficiency in buildings," Applied Energy, Elsevier, vol. 211(C), pages 146-160.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Chuanyan Hao & Anqi Zheng & Yuqi Wang & Bo Jiang, 2021. "Experiment Information System Based on an Online Virtual Laboratory," Future Internet, MDPI, vol. 13(2), pages 1-19, January.
    2. Kerang Cao & Jingyu Gao & Kwang-nam Choi & Lini Duan, 2020. "Learning a Hierarchical Global Attention for Image Classification," Future Internet, MDPI, vol. 12(11), pages 1-11, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Moe Soheilian & Géza Fischl & Myriam Aries, 2021. "Smart Lighting Application for Energy Saving and User Well-Being in the Residential Environment," Sustainability, MDPI, vol. 13(11), pages 1-17, May.
    2. Jia, Kunqi & Guo, Ge & Xiao, Jucheng & Zhou, Huan & Wang, Zhihua & He, Guangyu, 2019. "Data compression approach for the home energy management system," Applied Energy, Elsevier, vol. 247(C), pages 643-656.
    3. Ghahramani, Ali & Pantelic, Jovan & Lindberg, Casey & Mehl, Matthias & Srinivasan, Karthik & Gilligan, Brian & Arens, Edward, 2018. "Learning occupants’ workplace interactions from wearable and stationary ambient sensing systems," Applied Energy, Elsevier, vol. 230(C), pages 42-51.
    4. Yu, Xinran & Ergan, Semiha & Dedemen, Gokmen, 2019. "A data-driven approach to extract operational signatures of HVAC systems and analyze impact on electricity consumption," Applied Energy, Elsevier, vol. 253(C), pages 1-1.
    5. Fathia Chekired & Oussama Taabli & Zakaria Mehdi Khellili & Amar Tilmatine & Aníbal T. de Almeida & Laurent Canale, 2022. "Near-Zero-Energy Building Management Based on Arduino Microcontroller—On-Site Lighting Management Application," Energies, MDPI, vol. 15(23), pages 1-20, November.
    6. Baldi, Simone & Korkas, Christos D. & Lv, Maolong & Kosmatopoulos, Elias B., 2018. "Automating occupant-building interaction via smart zoning of thermostatic loads: A switched self-tuning approach," Applied Energy, Elsevier, vol. 231(C), pages 1246-1258.
    7. Bui, Dac-Khuong & Nguyen, Tuan Ngoc & Ngo, Tuan Duc & Nguyen-Xuan, H., 2020. "An artificial neural network (ANN) expert system enhanced with the electromagnetism-based firefly algorithm (EFA) for predicting the energy consumption in buildings," Energy, Elsevier, vol. 190(C).
    8. Ru-Guan Wang & Wen-Jen Ho & Kuei-Chun Chiang & Yung-Chieh Hung & Jen-Kuo Tai & Jia-Cheng Tan & Mei-Ling Chuang & Chi-Yun Ke & Yi-Fan Chien & An-Ping Jeng & Chien-Cheng Chou, 2023. "Analyzing Long-Term and High Instantaneous Power Consumption of Buildings from Smart Meter Big Data with Deep Learning and Knowledge Graph Techniques," Energies, MDPI, vol. 16(19), pages 1-24, September.
    9. J. R. S. Iruela & L. G. B. Ruiz & M. I. Capel & M. C. Pegalajar, 2021. "A TensorFlow Approach to Data Analysis for Time Series Forecasting in the Energy-Efficiency Realm," Energies, MDPI, vol. 14(13), pages 1-22, July.
    10. Yan Ding & Xiao Pan & Wanyue Chen & Zhe Tian & Zhiyao Wang & Qing He, 2022. "Prediction Method for Office Building Energy Consumption Based on an Agent-Based Model Considering Occupant–Equipment Interaction Behavior," Energies, MDPI, vol. 15(22), pages 1-31, November.
    11. Qicong Cai & Baizhan Li & Wenbo He & Miao Guo, 2024. "Energy Consumption Calculation of Civil Buildings in Regional Integrated Energy Systems: A Review of Characteristics, Methods and Application Prospects," Sustainability, MDPI, vol. 16(13), pages 1-25, July.
    12. Eduardo Viciana & Alfredo Alcayde & Francisco G. Montoya & Raul Baños & Francisco M. Arrabal-Campos & Antonio Zapata-Sierra & Francisco Manzano-Agugliaro, 2018. "OpenZmeter: An Efficient Low-Cost Energy Smart Meter and Power Quality Analyzer," Sustainability, MDPI, vol. 10(11), pages 1-13, November.
    13. Rashid, Haroon & Singh, Pushpendra & Stankovic, Vladimir & Stankovic, Lina, 2019. "Can non-intrusive load monitoring be used for identifying an appliance’s anomalous behaviour?," Applied Energy, Elsevier, vol. 238(C), pages 796-805.
    14. Hossein Moayedi & Amir Mosavi, 2021. "Suggesting a Stochastic Fractal Search Paradigm in Combination with Artificial Neural Network for Early Prediction of Cooling Load in Residential Buildings," Energies, MDPI, vol. 14(6), pages 1-19, March.
    15. Yu, Xinran & Ergan, Semiha, 2022. "Estimating power demand shaving capacity of buildings on an urban scale using extracted demand response profiles through machine learning models," Applied Energy, Elsevier, vol. 310(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jftint:v:12:y:2020:i:8:p:133-:d:396680. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.