IDEAS home Printed from https://ideas.repec.org/a/gam/jmathe/v12y2024i11p1624-d1399516.html
   My bibliography  Save this article

LAMBERT: Leveraging Attention Mechanisms to Improve the BERT Fine-Tuning Model for Encrypted Traffic Classification

Author

Listed:
  • Tao Liu

    (Institute of Cyberspace Security, Guangzhou University, Guangzhou 510006, China)

  • Xiting Ma

    (Institute of Cyberspace Security, Guangzhou University, Guangzhou 510006, China)

  • Ling Liu

    (Institute of Cyberspace Security, Guangzhou University, Guangzhou 510006, China)

  • Xin Liu

    (College of Computer Engineering and Applied Math, Changsha University, Changsha 410022, China)

  • Yue Zhao

    (Science and Technology on Communication Security Laboratory, Chengdu 610041, China)

  • Ning Hu

    (Institute of Cyberspace Security, Guangzhou University, Guangzhou 510006, China)

  • Kayhan Zrar Ghafoor

    (Department of Computer Science, Knowledge University, Erbil 44001, Iraq)

Abstract

Encrypted traffic classification is a crucial part of privacy-preserving research. With the great success of artificial intelligence technology in fields such as image recognition and natural language processing, how to classify encrypted traffic based on AI technology has become an attractive topic in information security. With good generalization ability and high training accuracy, pre-training-based encrypted traffic classification methods have become the first option. The accuracy of this type of method depends highly on the fine-tuning model. However, it is a challenge for existing fine-tuned models to effectively integrate the representation of packet and byte features extracted via pre-training. A novel fine-tuning model, LAMBERT, is proposed in this article. By introducing an attention mechanism to capture the relationship between BiGRU and byte sequences, LAMBERT not only effectively improves the sequence loss phenomenon of BiGRU but also improves the processing performance of encrypted stream classification. LAMBERT can quickly and accurately classify multiple types of encrypted traffic. The experimental results show that our model performs well on datasets with uneven sample distribution, no pre-training, and large sample classification. LAMBERT was tested on four datasets, namely, ISCX-VPN-Service, ISCX-VPN-APP, USTC-TFC and CSTNET-TLS 1.3, and the F1 scores reached 99.15%, 99.52%, 99.30%, and 97.41%, respectively.

Suggested Citation

  • Tao Liu & Xiting Ma & Ling Liu & Xin Liu & Yue Zhao & Ning Hu & Kayhan Zrar Ghafoor, 2024. "LAMBERT: Leveraging Attention Mechanisms to Improve the BERT Fine-Tuning Model for Encrypted Traffic Classification," Mathematics, MDPI, vol. 12(11), pages 1-22, May.
    • Handle: RePEc:gam:jmathe:v:12:y:2024:i:11:p:1624-:d:1399516
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2227-7390/12/11/1624/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2227-7390/12/11/1624/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Rahman, Aowabin & Srikumar, Vivek & Smith, Amanda D., 2018. "Predicting electricity consumption for commercial and residential buildings using deep recurrent neural networks," Applied Energy, Elsevier, vol. 212(C), pages 372-385.
    2. Niu, Dongxiao & Yu, Min & Sun, Lijie & Gao, Tian & Wang, Keke, 2022. "Short-term multi-energy load forecasting for integrated energy systems based on CNN-BiGRU optimized by attention mechanism," Applied Energy, Elsevier, vol. 313(C).
    3. Petr Velan & Milan Čermák & Pavel Čeleda & Martin Drašar, 2015. "A survey of methods for encrypted traffic classification and analysis," International Journal of Network Management, John Wiley & Sons, vol. 25(5), pages 355-374, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zhou, Guangzhao & Guo, Zanquan & Sun, Simin & Jin, Qingsheng, 2023. "A CNN-BiGRU-AM neural network for AI applications in shale oil production prediction," Applied Energy, Elsevier, vol. 344(C).
    2. Guanqun Wang & Haibo Teng & Lei Qiao & Hongtao Yu & You Cui & Kun Xiao, 2024. "Well Logging Reconstruction Based on a Temporal Convolutional Network and Bidirectional Gated Recurrent Unit Network with Attention Mechanism Optimized by Improved Sand Cat Swarm Optimization," Energies, MDPI, vol. 17(11), pages 1-15, June.
    3. Jiang, Ben & Li, Yu & Rezgui, Yacine & Zhang, Chengyu & Wang, Peng & Zhao, Tianyi, 2024. "Multi-source domain generalization deep neural network model for predicting energy consumption in multiple office buildings," Energy, Elsevier, vol. 299(C).
    4. Qingbo Hua & Zengliang Fan & Wei Mu & Jiqiang Cui & Rongxin Xing & Huabo Liu & Junwei Gao, 2024. "A Short-Term Power Load Forecasting Method Using CNN-GRU with an Attention Mechanism," Energies, MDPI, vol. 18(1), pages 1-17, December.
    5. Liu, Liqi & Liu, Yanli, 2022. "Load image inpainting: An improved U-Net based load missing data recovery method," Applied Energy, Elsevier, vol. 327(C).
    6. Chen, Zhelun & O’Neill, Zheng & Wen, Jin & Pradhan, Ojas & Yang, Tao & Lu, Xing & Lin, Guanjing & Miyata, Shohei & Lee, Seungjae & Shen, Chou & Chiosa, Roberto & Piscitelli, Marco Savino & Capozzoli, , 2023. "A review of data-driven fault detection and diagnostics for building HVAC systems," Applied Energy, Elsevier, vol. 339(C).
    7. Lu, Yakai & Tian, Zhe & Zhou, Ruoyu & Liu, Wenjing, 2021. "A general transfer learning-based framework for thermal load prediction in regional energy system," Energy, Elsevier, vol. 217(C).
    8. Ibrahim, Muhammad Sohail & Dong, Wei & Yang, Qiang, 2020. "Machine learning driven smart electric power systems: Current trends and new perspectives," Applied Energy, Elsevier, vol. 272(C).
    9. Pesantez, Jorge E. & Li, Binbin & Lee, Christopher & Zhao, Zhizhen & Butala, Mark & Stillwell, Ashlynn S., 2023. "A Comparison Study of Predictive Models for Electricity Demand in a Diverse Urban Environment," Energy, Elsevier, vol. 283(C).
    10. Hyunsoo Kim & Jiseok Jeong & Changwan Kim, 2022. "Daily Peak-Electricity-Demand Forecasting Based on Residual Long Short-Term Network," Mathematics, MDPI, vol. 10(23), pages 1-17, November.
    11. Ajith, Meenu & Martínez-Ramón, Manel, 2023. "Deep learning algorithms for very short term solar irradiance forecasting: A survey," Renewable and Sustainable Energy Reviews, Elsevier, vol. 182(C).
    12. Qing Yin & Chunmiao Han & Ailin Li & Xiao Liu & Ying Liu, 2024. "A Review of Research on Building Energy Consumption Prediction Models Based on Artificial Neural Networks," Sustainability, MDPI, vol. 16(17), pages 1-30, September.
    13. Zheng Wan & Hui Li, 2023. "Short-Term Power Load Forecasting Based on Feature Filtering and Error Compensation under Imbalanced Samples," Energies, MDPI, vol. 16(10), pages 1-22, May.
    14. Khan, Zulfiqar Ahmad & Khan, Shabbir Ahmad & Hussain, Tanveer & Baik, Sung Wook, 2024. "DSPM: Dual sequence prediction model for efficient energy management in micro-grid," Applied Energy, Elsevier, vol. 356(C).
    15. Akhter, Muhammad Naveed & Mekhilef, Saad & Mokhlis, Hazlie & Ali, Raza & Usama, Muhammad & Muhammad, Munir Azam & Khairuddin, Anis Salwa Mohd, 2022. "A hybrid deep learning method for an hour ahead power output forecasting of three different photovoltaic systems," Applied Energy, Elsevier, vol. 307(C).
    16. Wenzhi Cao & Houdun Liu & Xiangzhi Zhang & Yangyan Zeng & Xiao Ling, 2025. "Short-Term Residential Load Forecasting Based on the Fusion of Customer Load Uncertainty Feature Extraction and Meteorological Factors," Sustainability, MDPI, vol. 17(3), pages 1-21, January.
    17. Li, Ao & Xiao, Fu & Zhang, Chong & Fan, Cheng, 2021. "Attention-based interpretable neural network for building cooling load prediction," Applied Energy, Elsevier, vol. 299(C).
    18. Hu, Yue & Liu, Hanjing & Wu, Senzhen & Zhao, Yuan & Wang, Zhijin & Liu, Xiufeng, 2024. "Temporal collaborative attention for wind power forecasting," Applied Energy, Elsevier, vol. 357(C).
    19. Zulfiqar, M. & Kamran, M. & Rasheed, M.B. & Alquthami, T. & Milyani, A.H., 2023. "A hybrid framework for short term load forecasting with a navel feature engineering and adaptive grasshopper optimization in smart grid," Applied Energy, Elsevier, vol. 338(C).
    20. Chitalia, Gopal & Pipattanasomporn, Manisa & Garg, Vishal & Rahman, Saifur, 2020. "Robust short-term electrical load forecasting framework for commercial buildings using deep recurrent neural networks," Applied Energy, Elsevier, vol. 278(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jmathe:v:12:y:2024:i:11:p:1624-:d:1399516. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.