IDEAS home Printed from https://ideas.repec.org/a/gam/jmathe/v11y2023i9p2129-d1137920.html
   My bibliography  Save this article

Modelling Sign Language with Encoder-Only Transformers and Human Pose Estimation Keypoint Data

Author

Listed:
  • Luke T. Woods

    (Digital Aviation Research and Technology Centre (DARTeC), Cranfield University, Cranfield, Bedfordshire MK43 0AL, UK
    Leidos Industrial Engineers Limited, Unit 3, Bedford Link Logistics Park, Bell Farm Way, Kempston, Bedfordshire MK43 9SS, UK)

  • Zeeshan A. Rana

    (Centre for Aeronautics, School of Aerospace, Transport and Manufacturing (SATM), Cranfield University, Cranfield, Bedfordshire MK43 0AL, UK)

Abstract

We present a study on modelling American Sign Language (ASL) with encoder-only transformers and human pose estimation keypoint data. Using an enhanced version of the publicly available Word-level ASL (WLASL) dataset, and a novel normalisation technique based on signer body size, we show the impact model architecture has on accurately classifying sets of 10, 50, 100, and 300 isolated, dynamic signs using two-dimensional keypoint coordinates only. We demonstrate the importance of running and reporting results from repeated experiments to describe and evaluate model performance. We include descriptions of the algorithms used to normalise the data and generate the train, validation, and test data splits. We report top-1, top-5, and top-10 accuracy results, evaluated with two separate model checkpoint metrics based on validation accuracy and loss. We find models with fewer than 100k learnable parameters can achieve high accuracy on reduced vocabulary datasets, paving the way for lightweight consumer hardware to perform tasks that are traditionally resource-intensive, requiring expensive, high-end equipment. We achieve top-1, top-5, and top-10 accuracies of 97 % , 100 % , and 100 % , respectively, on a vocabulary size of 10 signs; 87 % , 97 % , and 98 % on 50 signs; 83 % , 96 % , and 97 % on 100 signs; and 71 % , 90 % , and 94 % on 300 signs, thereby setting a new benchmark for this task.

Suggested Citation

  • Luke T. Woods & Zeeshan A. Rana, 2023. "Modelling Sign Language with Encoder-Only Transformers and Human Pose Estimation Keypoint Data," Mathematics, MDPI, vol. 11(9), pages 1-28, May.
    • Handle: RePEc:gam:jmathe:v:11:y:2023:i:9:p:2129-:d:1137920
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2227-7390/11/9/2129/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2227-7390/11/9/2129/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Konstantinos Poulinakis & Dimitris Drikakis & Ioannis W. Kokkinakis & Stephen Michael Spottswood, 2023. "Machine-Learning Methods on Noisy and Sparse Data," Mathematics, MDPI, vol. 11(1), pages 1-19, January.
    2. Homero V. Rios-Figueroa & Angel J. Sánchez-García & Candy Obdulia Sosa-Jiménez & Ana Luisa Solís-González-Cosío, 2022. "Use of Spherical and Cartesian Features for Learning and Recognition of the Static Mexican Sign Language Alphabet," Mathematics, MDPI, vol. 10(16), pages 1-25, August.
    3. Shanker, M. & Hu, M. Y. & Hung, M. S., 1996. "Effect of data standardization on neural network training," Omega, Elsevier, vol. 24(4), pages 385-397, August.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Matteo Picozzi & Antonio Giovanni Iaccarino, 2021. "Forecasting the Preparatory Phase of Induced Earthquakes by Recurrent Neural Network," Forecasting, MDPI, vol. 3(1), pages 1-20, January.
    2. Luis Alberto Geraldo-Campos & Juan J. Soria & Tamara Pando-Ezcurra, 2022. "Machine Learning for Credit Risk in the Reactive Peru Program: A Comparison of the Lasso and Ridge Regression Models," Economies, MDPI, vol. 10(8), pages 1-21, July.
    3. Samuka Mohanty & Rajashree Dash, 2023. "A New Dual Normalization for Enhancing the Bitcoin Pricing Capability of an Optimized Low Complexity Neural Net with TOPSIS Evaluation," Mathematics, MDPI, vol. 11(5), pages 1-28, February.
    4. Joana Dias & Humberto Rocha & Brígida Ferreira & Maria Lopes, 2014. "A genetic algorithm with neural network fitness function evaluation for IMRT beam angle optimization," Central European Journal of Operations Research, Springer;Slovak Society for Operations Research;Hungarian Operational Research Society;Czech Society for Operations Research;Österr. Gesellschaft für Operations Research (ÖGOR);Slovenian Society Informatika - Section for Operational Research;Croatian Operational Research Society, vol. 22(3), pages 431-455, September.
    5. Zhang, Guoqiang & Eddy Patuwo, B. & Y. Hu, Michael, 1998. "Forecasting with artificial neural networks:: The state of the art," International Journal of Forecasting, Elsevier, vol. 14(1), pages 35-62, March.
    6. Miguel Núñez-Peiró & Anna Mavrogianni & Phil Symonds & Carmen Sánchez-Guevara Sánchez & F. Javier Neila González, 2021. "Modelling Long-Term Urban Temperatures with Less Training Data: A Comparative Study Using Neural Networks in the City of Madrid," Sustainability, MDPI, vol. 13(15), pages 1-23, July.
    7. Semenoglou, Artemios-Anargyros & Spiliotis, Evangelos & Makridakis, Spyros & Assimakopoulos, Vassilios, 2021. "Investigating the accuracy of cross-learning time series forecasting methods," International Journal of Forecasting, Elsevier, vol. 37(3), pages 1072-1084.
    8. Shigeyuki Hamori & Takahiro Kume, 2018. "Artificial Intelligence And Economic Growth," Advances in Decision Sciences, Asia University, Taiwan, vol. 22(1), pages 256-278, December.
    9. James R. Coakley & Carol E. Brown, 2000. "Artificial neural networks in accounting and finance: modeling issues," Intelligent Systems in Accounting, Finance and Management, John Wiley & Sons, Ltd., vol. 9(2), pages 119-144, June.
    10. Zhang, Guoqiang & Y. Hu, Michael & Eddy Patuwo, B. & C. Indro, Daniel, 1999. "Artificial neural networks in bankruptcy prediction: General framework and cross-validation analysis," European Journal of Operational Research, Elsevier, vol. 116(1), pages 16-32, July.
    11. Zhang, Gioqinang & Hu, Michael Y., 1998. "Neural network forecasting of the British Pound/US Dollar exchange rate," Omega, Elsevier, vol. 26(4), pages 495-506, August.
    12. Wu, Menglong & Xiong, Jiajie & Li, Ruoyu & Dong, Aihong & Lv, Chang & Sun, Dan & Abdelghany, Ahmed Elsayed & Zhang, Qian & Wang, Yaqiong & Siddique, Kadambot H.M. & Niu, Wenquan, 2024. "Precision forecasting of fertilizer components’ concentrations in mixed variable-rate fertigation through machine learning," Agricultural Water Management, Elsevier, vol. 298(C).
    13. Qian, Cheng & Xu, Binghui & Chang, Liang & Sun, Bo & Feng, Qiang & Yang, Dezhen & Ren, Yi & Wang, Zili, 2021. "Convolutional neural network based capacity estimation using random segments of the charging curves for lithium-ion batteries," Energy, Elsevier, vol. 227(C).
    14. Apostolos Ampountolas, 2023. "Comparative Analysis of Machine Learning, Hybrid, and Deep Learning Forecasting Models: Evidence from European Financial Markets and Bitcoins," Forecasting, MDPI, vol. 5(2), pages 1-15, June.
    15. Apostolos Ampountolas, 2023. "Comparative Analysis of Machine Learning, Hybrid, and Deep Learning Forecasting Models Evidence from European Financial Markets and Bitcoins," Papers 2307.08853, arXiv.org.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jmathe:v:11:y:2023:i:9:p:2129-:d:1137920. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.