Real-Time BISINDO Alphabet Recognition via Faster R-CNN Incorporating Skin Tone Diversity as a Classification Feature
DOI:
https://doi.org/10.12928/biste.v8i3.15587Keywords:
BISINDO, Faster R-CNN, Skin Color Features, Hand Gesture Recognition, Assistive TechnologyAbstract
Indonesian Sign Language (Bahasa Isyarat Indonesia/BISINDO) enables communication for deaf individuals through hand gestures, yet limited public awareness creates significant barriers between deaf and hearing communities. Existing recognition systems often fail to generalize across diverse skin tones, reducing their effectiveness in inclusive real-world deployment. The contribution of this research is a BISINDO alphabet recognition system that integrates skin color features - extracted via HSV-based skin segmentation - as an additional preprocessing layer within the Faster R-CNN framework, explicitly improving detection robustness across varied skin tones. The dataset consists of 8,000 images from ten adult actors representing light, medium-brown, and dark skin tones, augmented through flipping and brightness variation, with a 90:10 training-to-testing ratio. The model was trained over 15,000 steps with a batch size of 24, selected through empirical validation to balance convergence stability and dataset size. Experimental results show that indoor conditions outperform outdoor settings due to controlled lighting. Light-skinned and dark-skinned participants achieved the highest accuracy of 87.5% and F1-score of 85.71%, while medium-brown-skinned participants showed slightly lower performance, likely attributed to greater variability in reflectance under mixed lighting. The system achieves 24 frames per second, demonstrating potential for real-time communication support. These findings confirm that Faster R-CNN with skin color feature integration is effective for BISINDO alphabet recognition, with skin tone diversity being a critical performance factor. Future work will explore larger participant pools and dynamic gesture recognition under varied real-world lighting scenarios.
References
F. I. Pranto et al., “A comprehensive image dataset of American Sign Language hand gestures,” Data Br., vol. 65, p. 112492, 2026, https://doi.org/10.1016/j.dib.2026.112492.
I. M. M. Violet and R. L. Sri, “A comprehensive survey on recent advances and challenges in sign language recognition systems,” Discov. Artif. Intell., vol. 7, 2025, https://doi.org/10.1007/s44163-025-00629-7.
L. N. Hayati, A. N. Handayani, W. Sakti, G. Irianto, and R. A. Asmara, “Improving Indonesian Sign Alphabet Recognition for Assistive Learning Robots Using Gamma-Corrected MobileNetV2,” Bul. Ilm. Sarj. Tek. Elektro, vol. 7, no. 3, pp. 350–361, 2025, https://doi.org/10.12928/biste.v7i3.13300.
S. Daniels, N. Suciati, and C. Fathichah, “Indonesian sign language recognition using yolo method,” In IOP Conference Series: Materials Science and Engineering, vol. 1077, no. 1, p. 012029, 2021, https://doi.org/10.1088/1757-899X/1077/1/012029.
L. N. Hayati, A. N. Handayani, W. S. G. Irianto, R. A. Asmara, D. Indra, and M. Fahmi, M. , “Classifying BISINDO Alphabet using Tensorflow Object Detection API,” Ilk. J. Ilm., vol. 15, no. 2, pp. 358–364, 2023, https://doi.org/10.33096/ilkom.v15i2.1692.358-364.
N. H. Amir, C. Kusuma, and A. Luthfi, “Refining the Performance of Neural Networks with Simple Architectures for Indonesian Sign Language System ( SIBI ) Letter Recognition Using Keypoint Detection,” Ilk. J. Ilm., vol. 17, no. 1, pp. 64–73, 2025, https://doi.org/10.33096/ilkom.v17i1.2522.64-73.
A. Umar, W. Qingshang, M. Badreddine, and Z. Jiangtao, “SignViT: An enhanced vision transformer framework for Attention-Based sign language hand gesture recognition,” Elsevier Biomed. Signal Process. Control, vol. 112, no. 108602, 2026, https://doi.org/10.1016/j.bspc.2025.108602.
Q. Zhu, J. Li, F. Yuan, J. Fan, and Q. Gan, “A Chinese Continuous Sign Language Dataset Based on Complex Environments,” arXiv preprint arXiv:2409.11960, 2024, https://doi.org/10.48550/arXiv.2409.11960.
S. T. Abd Al-Latief, S. Yussof, A. Ahmad, S. M. Khadim, and R. A. Abdulhasan, “Instant Sign Language Recognition by WAR Strategy Algorithm Based Tuned Machine Learning,” Int. J. Networked Distrib. Comput., vol. 12, no. 2, pp. 344–361, 2024, https://doi.org/10.1007/s44227-024-00039-8.
N. C. Camgoz, O. Koller, S. Hadfield, and R. Bowden, “Sign language transformers: Joint end-to-end sign language recognition and translation,” In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10023-10033, 2020, https://doi.org/10.1109/CVPR42600.2020.01004.
Y. Yusmanto, H. Ar, and A. Prasetya, “Design and development of face recognition-based security system using expression game as liveness detection,” Bull. Soc. Informatics Theory Appl., vol. 8, no. 2, pp. 280–294, 2024, https://doi.org/10.31763/businta.v8i2.756.
Y. Yang, T. Wan, M. Zhang, and L. Li, “High-precision YOLOv5 object detection technology based on multi-module collaborative optimization,” Multimed. Tools Appl., vol. 85, no. 2, p. 69, 2026, https://doi.org/10.1007/s11042-026-21227-4.
J. Moni, R. R. Varghese, A. Binoy, B. S. Benny, L. Rajan and B. Benni, "Sign Language Translation Assistant using Machine Learning," 2022 International Conference on Decision Aid Sciences and Applications (DASA), pp. 1062-1066, 2022, https://doi.org/10.1109/DASA54658.2022.9765097.
N. Harlinda; Rendi, Ahmad; Azis, Huzain; Indra, Dolly; Hayati, Lilis Nur; Kurniati, “Classification of Cia-Cia Letters Using MobileNetV2 and CNN Methods,” 2025 19th Int. Conf. Ubiquitous Inf. Manag. Commun., 2025, https://doi.org/10.1109/IMCOM64595.2025.10857478.
J. Ahmed and B. Shawon, “A comparative analysis of video vision transformers on word-level sign language datasets,” PLoS One, vol. 2, no. 21, pp. 1–22, 2026, https://doi.org/10.1371/journal.pone.0341909.
M. Martawidjaja, R. D. Tandiono and L. Ayu Wulandhari, "Artificial Intelligence-Based System for Word Recognition and Extraction in Indonesian Sign Language Videos," 2025 8th International Seminar on Research of Information Technology and Intelligent Systems (ISRITI), pp. 802-808, 2025, https://doi.org/10.1109/ISRITI68345.2025.11393297.
Y. V. Via, W. S. Saputra, M. I. Fachrurrozi, E. Y. Puspaningrum, F. T. Anggraeny, and S. R. Nudin, S. R. (2023)., “Object Localization and Detecting Alphabet in Sign Language BISINDO Using Convolution Neural Network,” Technium, vol. 16, no. 143, 2023, https://doi.org/10.47577/technium.v16i.9973.
F. M. Najib, “Sign language interpretation using machine learning and artificial intelligence,” Neural Comput. Appl., vol. 9, 2024, https://doi.org/10.1007/s00521-024-10395-9.
A. M. J. AL Moustafa et al., “Arabic Sign Language Recognition Systems: a Systematic Review,” Indian J. Comput. Sci. Eng., vol. 15, no. 1, pp. 1–18, 2024, https://doi.org/10.21817/indjcse/2023/v15i1/241501008.
S. Al Ahmadi, F. Mohammad, and H. Al Dawsari, “Efficient YOLO-Based Deep Learning Model for Arabic Sign Language Recognition,” J. Disabil. Res., vol. 3, no. 4, 2024, https://doi.org/10.57197/jdr-2024-0051.
H. Yang et al., “Systemic representational biases in anatomical education: A multi-modal content analysis of racial, sex, and skin tone diversity across literature, textbooks, and digital platforms,” Ann. Anat. - Anat. Anzeiger, vol. 265, 2026, https://doi.org/10.1016/j.aanat.2026.152801.
S. M. K. Raja’a M. Mohammed, “Iraqi Sign Language Translator system using Deep Learning,” Al-Salam J. Eng. Technol., vol. 1, pp. 109–116, 2023, https://doi.org/10.55145/ajest.2023.01.01.0013.
S. Aiouez, A. Hamitouche, M. Belmadoui, K. Belattar, and F. Souami, “Real-time Arabic Sign Language Recognition based on YOLOv5,” SCITEPRESS, vol. 17–25, pp. 17–25, 2022, https://doi.org/10.5220/0010979300003209.
W.-B. Ma, Y. Yang, and W.-C. Fang, “An Effective Tuberculosis Detection System Based on Improved Faster R-CNN with RoI Align Method,” 2023 IEEE Biomed. Circuits Syst. Conf., 2023, https://doi.org/10.1109/BioCAS58349.2023.10388704.
M. R. Ningsih et al., “Sign Language Detection System Using YOLOv5 Algorithm to Promote Communication Equality People with Disabilities,” Sci. J. Informatics, vol. 11, no. 2, pp. 549–558, 2024, https://doi.org/10.15294/sji.v11i2.6007.
G. Priyadharshini and D. R. Judie Dolly, "Comparative Investigations on Tomato Leaf Disease Detection and Classification Using CNN, R-CNN, Fast R-CNN and Faster R-CNN," 2023 9th International Conference on Advanced Computing and Communication Systems (ICACCS), pp. 1540-1545, 2023, https://doi.org/10.1109/ICACCS57279.2023.10112860.
I. Panopoulos, E. Topalis, and N. Petrellis, “Greek Sign Language Detection with Artificial Intelligence,” Electronics, vol. 14, no. 16, p. 3241, 2025, https://doi.org/10.3390/electronics14163241.
Y. Farhan, Z. Haimer, and A. A. Madi, “Real-time interpretation of American Sign Language using SSD-MobileNet,” International Journal of Computational Vision and Robotics, vol. 16, no. 2, pp. 222-245. 2026, https://doi.org/10.1504/IJCVR.2026.151535.
B. Tej, F. Nasri, and A. Mtibaa, “Detection of Pepper and Tomato leaf diseases using deep learning techniques,” Proc. 2022 5th Int. Conf. Adv. Syst. Emergent Technol. IC_ASET 2022, no. January 2024, pp. 149–154, 2022, https://doi.org/10.1109/IC_ASET53395.2022.9765923.
A. S. G. Raharjo and E. Sugiharti, “Alphabet Classification of Sign System Using Convolutional Neural Network with Contrast Limited Adaptive Histogram Equalization and Canny Edge Detection,” Sci. J. Informatics, vol. 10, no. 3, pp. 239–250, 2023, https://doi.org/10.15294/sji.v10i3.44137.
S. Feng, L. Zhao, H. Shi, M. Wang, S. Shen, and W. Wang, “One-dimensional VGGNet for high-dimensional data,” Appl. Soft Comput., vol. 135, p. 110035, 2023, https://doi.org/10.1016/j.asoc.2023.110035.
S. Wang, “Domain-adaptive faster R-CNN for non-PPE identification on construction sites from body-worn and general images,” Sci. Rep., pp. 1–22, 2026, https://doi.org/10.1038/s41598-026-35148-7.
I. Lamaakal, C. Yahyati, Y. Maleh, and K. El Makkaoui, “An explainable hybrid CNN – transformer model for sign language recognition on edge devices using adaptive fusion and knowledge distillation,” Sci. Rep., vol. 16, no. 7143, pp. 1–23, 2026, https://doi.org/10.1038/s41598-026-38478-8.
V. E. Nurfita Sari, A. Arifin and F. Arrofiqi, "Design of Two-Way Indonesian Sign Language System based on Smart-Glove with Artificial Neural Network Classification Method," 2024 International Conference on Computer Engineering, Network, and Intelligent Multimedia (CENIM), pp. 1-6, 2024, https://doi.org/10.1109/CENIM64038.2024.10882818.
P. Hu, J. Lu, Y. Cui, B. Hu, and F. Liang, “Insulator Defect Detection based on Faster R-CNN and YOLOv3 Algorithm,” 2023 IEEE 4th China Int. Youth Conf. Electr. Eng., 2023, https://doi.org/10.1109/CIYCEE59789.2023.10401677.
N. Azizah, “Improving Smear-Negative Tuberculosis Detection Using Data Augmentation and Faster R-CNN,” Int. J. Cyber IT Serv. Manag., vol. 6, no. 1, pp. 65–77, 2026, https://doi.org/10.34306/ijcitsm.v6i1.233.
H. M. Hamza and A. Wali, “Pakistan Sign Language Recognition: From Videos to Images,” Signal, Image Video Process., vol. 19, no. 8, p. 682, 2025, https://doi.org/10.1007/s11760-025-04230-4.
D. A. Rachmawati, C. Supriyanto, M. A. Soeleman and N. Hendriyanto, "Performance Improvement of the Faster R-CNN Model Using ResNet Architecture and Data Augmentation for Indonesian Sign Language (BISINDO) Detection," 2025 International Seminar on Application for Technology of Information and Communication (iSemantic), pp. 298-304, 2025, https://doi.org/10.1109/ISemantic67418.2025.11291950.
Y. Tian, Y. Dong, M. Ahmed, S. O. Shah, and E. Alabdulkreem, “Real-Time Chinese Sign Language Recognition Based on Convolutional Neural Network,” Int. J. Humanoid Robot., p. 2540022, 2026, https://doi.org/10.1142/S0219843625400225.
W. Luo, "Research on gesture recognition based on YOLOv5," 2022 3rd International Conference on Big Data, Artificial Intelligence and Internet of Things Engineering (ICBAIE), pp. 447-450, 2022, https://doi.org/10.1109/ICBAIE56435.2022.9985931.
D. A. Rachmawati, C. Supriyanto, M. A. Soeleman and N. Hendriyanto, "Performance Improvement of the Faster R-CNN Model Using ResNet Architecture and Data Augmentation for Indonesian Sign Language (BISINDO) Detection," 2025 International Seminar on Application for Technology of Information and Communication (iSemantic), pp. 298-304, 2025, https://doi.org/10.1109/ISemantic67418.2025.11291950.
E. L. R. Ewe, C. P. Lee, L. C. Kwek, and K. M. Lim, “Hand Gesture Recognition via Lightweight VGG16 and Ensemble Classifier,” Appl. Sci., vol. 12, no. 15, 2022, https://doi.org/10.3390/app12157643.
M. G. F. Odounfa, C. D. S. J. Gbemavo, S. P. G. Tahi, and R. L. Glèlè Kakaï, “Deep learning methods for enhanced stress and pest management in market garden crops: A comprehensive analysis,” Smart Agric. Technol., vol. 9, p. 100521, 2024, https://doi.org/10.1016/j.atech.2024.100521.
A. Alayed, “Machine Learning and Deep Learning Approaches for Arabic Sign Language Recognition: A Decade Systematic Literature Review,” Sensors, vol. 24, no. 23, 2024, https://doi.org/10.3390/s24237798.
Z. Haimer, K. Mateur, Y. Farhan and A. A. Madi, "Road Marking Detection and Instance Segmentation Using YOLOv8 Models," 2024 10th International Conference on Optimization and Applications (ICOA), pp. 1-8, 2024, https://doi.org/10.1109/ICOA62581.2024.10754303.
D. O. Pratama, U. N. Malang, A. N. Handayani, and U. N. Malang, “Development of Embedded System Learning Module Using Project-based Learning Method for Industrial Electronics Department,” Lect. J. Pendidik., vol. 16, pp. 225–238, 2025, https://doi.org/10.31849/lectura.v16i1.25415.
N. Ahmad, E. S. Wijaya, C. Tjoaquinn, H. Lucky, and I. A. Iswanto, “Transforming Sign Language using CNN Approach based on BISINDO Dataset,” 2023 Int. Conf. Informatics, Multimedia, Cyber Informations Syst., 2023, https://doi.org/10.1109/ICIMCIS60089.2023.10349011.
D. Joan, V. Vincent, K. J. Daniel, S. Achmad and R. Sutoyo, "BISINDO Hand-Sign Detection Using Transfer Learning," 2023 IEEE 8th International Conference on Recent Advances and Innovations in Engineering (ICRAIE), pp. 1-7, 2023, https://doi.org/10.1109/ICRAIE59459.2023.10468194.
M. Z. U. Rahman et al., "A Real-Time System for Classification of Pakistan Sign Language using Machine Learning," 2023 17th International Conference on Open Source Systems and Technologies (ICOSST), pp. 1-5, 2023, https://doi.org/10.1109/ICOSST60641.2023.10414200.
A. Youssef, A. Gaber, and S. M. EL-Metwally, “A Dual-Architecture Deep Learning Pipeline for Real- Time High-Accuracy Arabic Sign Language Recognition,” Discov. Artif. Intell., 2026, https://doi.org/10.21203/rs.3.rs-8605046/v1.
R. Dababo and M. -H. Wang, "Speech2Sign with ASL-Transformers and SL-GAN," 2024 International Conference on Image Processing, Computer Vision and Machine Learning (ICICML), pp. 965-971, 2024, https://doi.org/10.1109/ICICML63543.2024.10958134.
K. Malik, C. Robertson, S. A. Roberts, T. K. Remmel, and J. A. Long, “Computer vision models for comparing spatial patterns: understanding spatial scale,” Int. J. Geogr. Inf. Sci., vol. 37, no. 1, pp. 1–35, 2023, https://doi.org/10.1080/13658816.2022.2103562.
R. A. Asmara et al., “YOLO-based object detection performance evaluation for automatic target aimbot in first-person shooter games,” Bull. Electr. Eng. Informatics, vol. 13, no. 4, pp. 2456–2470, 2024, https://doi.org/10.11591/eei.v13i4.6895.
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Lilis Nur Hayati, Anik Nur Handayani, Wahyu Sakti Gunawan Irianto, Rosa Andrie Asmara, Dolly Indra, Nor Salwa Damanhuri
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
This journal is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
