End-to-end system for translating bahasa isyarat Indonesia sign language gestures into Indonesian text

Abstract

This study addresses critical challenges in developing an end-to-end bahasa isyarat Indonesia (BISINDO) SLT by integrating advanced deep learning techniques to overcome complex background interference, transitional gesture recognition, and limitations in dataset availability. While existing SLT systems struggle with isolated word recognition and manual preprocessing, our work introduces three key innovations: (1) implementation of YOLOv8 for optimized object detection, achieving 88% mAP and reducing WER to 11.40%, outperforming YOLOv5/v7 in handling complex backgrounds; (2) automated removal of transitional gestures using Threshold conditional random fields (TCRF), which attained 95.68% accuracy, significantly improving upon MobileNetV2’s performance (WER: 6.89% vs. 93.53%); and (3) end-to-end BISINDO SLT by expansion of the BISINDO dataset to 435 word labels, enabling comprehensive sentencelevel translation. Experimental results demonstrate the system’s robustness, with 8.31% of WER, 84.13% of SAcc, and 87.08% of SacreBLEU after dataset expansion and redundancy elimination through grouping methods. The proposed framework operates without manual intervention, marking a substantial advancement toward real-world applicability.