Real-time recognition of Indonesian sign language SIBI using CNN-SVM model combination

Abstract

Real-time Sistem Isyarat Bahasa Indonesia (SIBI) sign language recognition plays a crucial role in improving accessibility for individuals with hearing and speech impairments. Despite advancements in SIBI recognition research, challenges remain in ensuring model stability and accuracy in realtime settings, particularly in handling gesture variations and classification inconsistencies. This study addresses these challenges by developing a convolutional neural network-support vector machine (CNN-SVM) combination model, integrating MediaPipe for hand coordinate extraction, CNN for feature extraction, and SVM for classification. To improve generalization and prevent overfitting, data augmentation is applied to expand the dataset. The model's performance is further enhanced through hyperparameter optimization (HPO) and post-processing techniques such as multi-window majority voting (MWMV) and SymSpell. Experimental results show that the CNN-SVM model trained on augmented data with HPO achieves 91% testing accuracy, outperforming both standalone CNN and SVM models. Furthermore, MWMV improves recognition stability, while SymSpell enhances spelling errors, ensuring more meaningful outputs. The system is integrated with OpenCV for real-time recognition, but current deployment remains limited to local execution. Future work will focus on developing lightweight models for web-based and mobile applications, making the system more accessible and scalable.