Improving Indonesian Sign Alphabet Recognition for Assistive Learning Robots Using Gamma-Corrected MobileNetV2

Lilis Nur Hayati; Anik Nur Handayani; Wahyu Sakti Gunawan Irianto; Rosa Andrie Asmara; Dolly Indra; Nor Salwa Damanhuri

doi:10.12928/biste.v7i3.13300

Authors

Lilis Nur Hayati Universitas Muslim Indonesia
Anik Nur Handayani Universitas Negeri Malang
Wahyu Sakti Gunawan Irianto Universitas Negeri Malang
Rosa Andrie Asmara Politeknik Negeri Malang
Dolly Indra Universitas Muslim Indonesia
Nor Salwa Damanhuri Universiti Teknologi MARA (UiTM)

DOI:

https://doi.org/10.12928/biste.v7i3.13300

Keywords:

BISINDO, MobileNetV2, Gamma Correction, SSD, Assistive Robots

Abstract

Sign language recognition plays a critical role in promoting inclusive education, particularly for deaf children in Indonesia. However, many existing systems struggle with real-time performance and sensitivity to lighting variations, limiting their applicability in real-world settings. This study addresses these issues by optimizing a BISINDO (Bahasa Isyarat Indonesia) alphabet recognition system using the SSD MobileNetV2 architecture, enhanced with gamma correction as a luminance normalization technique. The research contribution is the integration of gamma correction preprocessing with SSD MobileNetV2, tailored for BISINDO and implemented on a low-cost assistive robot platform. This approach aims to improve robustness under diverse lighting conditions while maintaining real-time capability without the use of specialized sensors or wearables. The proposed method involves data collection, image augmentation, gamma correction (γ = 1.2, 1.5, and 2.0), and training using the SSD MobileNetV2 FPNLite 320x320 model. The dataset consists of 1,820 original images expanded to 5,096 via augmentation, with 26 BISINDO alphabet classes. The system was evaluated under indoor and outdoor conditions. Experimental results showed significant improvements with gamma correction. Indoor accuracy increased from 94.47% to 97.33%, precision from 91.30% to 95.23%, and recall from 97.87% to 99.57%. Outdoor accuracy improved from 93.80% to 97.30%, with precision rising from 90.33% to 94.73%, and recall reaching 100%. In conclusion, the proposed system offers a reliable, real-time solution for BISINDO recognition in low-resource educational environments. Future work includes the recognition of two-handed gestures and integration with natural language processing for enhanced contextual understanding.

References

N. Thiracitta and H. Gunawan, “SIBI sign language recognition using convolutional neural network combined with transfer learning and non-trainable parameters,” Procedia Computer Science, vol. 179, pp. 72-80, 2021, https://doi.org/10.1016/j.procs.2020.12.011.

R. Sutjiadi, S. Sendari, H. W. Herwanto, and Y. Kristian, “Generating High-quality Synthetic Mammogram Images Using Denoising Diffusion Probabilistic Models: A Novel Approach for Augmenting Deep Learning Datasets,” 2024 Int. Conf. Inf. Technol. Syst. Innov. ICITSI 2024 - Proc., pp. 386–392, 2024, https://doi.org/10.1109/ICITSI65188.2024.10929446.

S. Dwijayanti, Hermawati, S. I. Taqiyyah, H. Hikmarika, and B. Y. Suprapto, “Indonesia Sign Language Recognition using Convolutional Neural Network,” Int. J. Adv. Comput. Sci. Appl., vol. 12, no. 10, pp. 415–422, 2021, https://doi.org/10.14569/IJACSA.2021.0121046.

S. Subburaj and S. Murugavalli, “Survey on sign language recognition in context of vision-based and deep learning,” Measurement: Sensors, vol. 23, p. 100385, 2022, https://doi.org/10.1016/j.measen.2022.100385.

K. G. Nalbant and Ş. Uyanik, “Computer Vision in the Metaverse,” J. Metaverse, vol. 1, no. 1, pp. 9–12, 2021, https://dergipark.org.tr/en/pub/jmv/issue/67581/1051377.

Y. Yusmanto, H. Ar, and A. Prasetya, “Design and development of face recognition-based security system using expression game as liveness detection,” Bull. Soc. Informatics Theory Appl., vol. 8, no. 2, pp. 280–294, 2024, https://doi.org/10.31763/businta.v8i2.756.

A. Elhagry and R. G. Elrayes, “Egyptian Sign Language Recognition Using CNN and LSTM,” arXiv preprint arXiv:2107.13647, 2021, https://doi.org/10.48550/arXiv.2107.13647.

B. T. Abeje, A. O. Salau, A. D. Mengistu, and N. K. Tamiru, “Ethiopian sign language recognition using deep convolutional neural network,” Multimed. Tools Appl., vol. 81, no. 20, pp. 29027–29043, 2022, https://doi.org/10.1007/s11042-022-12768-5.

F. Shah, M. S. Shah, W. Akram, A. Manzoor, R. O. Mahmoud, and D. S. Abdelminaam, “Sign Language Recognition Using Multiple Kernel Learning: A Case Study of Pakistan Sign Language,” IEEE Access, vol. 9, pp. 67548–67558, 2021, https://doi.org/10.1109/ACCESS.2021.3077386.

E. Aldhahri et al., “Arabic Sign Language Recognition Using Convolutional Neural Network and MobileNet,” Arab. J. Sci. Eng., vol. 48, no. 2, pp. 2147–2154, 2023, https://doi.org/10.1007/s13369-022-07144-2.

N. C. Basjaruddin, E. Rakhman, Y. Sudarsa, M. B. Z. Asyikin, and S. Permana, “Attendance system with face recognition, body temperature, and use of mask using multi-task cascaded convolutional neural network (MTCNN) Method,” Green Intelligent Systems and Applications, vol. 2, no. 2, pp. 71-83, 2024, https://doi.org/10.53623/gisa.v2i2.109.

R. A. Asmara, U. D. Rosiani, M. Mentari, A. R. Syulistyo, M. N. Shoumi, and M. Astiningrum, “An Experimental Study on Deep Learning Technique Implemented on Low Specification OpenMV Cam H7 Device,” Int. J. Informatics Vis., vol. 8, no. 2, pp. 1017–1029, 2024, https://doi.org/10.62527/joiv.8.2.2299.

M. G. F. Odounfa, C. D. S. J. Gbemavo, S. P. G. Tahi, and R. L. Glèlè Kakaï, “Deep learning methods for enhanced stress and pest management in market garden crops: A comprehensive analysis,” Smart Agric. Technol., vol. 9, p. 100521, 2024, https://doi.org/10.1016/j.atech.2024.100521.

M. Chitra, H. Balasubramanian, V. L. Srinivaas and K. Jayanth, "An Adaptation and Evaluation for Object Detection of Small Objects Using YOLO8," 2024 International Conference on Intelligent Computing and Emerging Communication Technologies (ICEC), pp. 1-9, 2024, https://doi.org/10.1109/ICEC59683.2024.10837540.

B. Alsharif, E. Alalwany, and M. Ilyas, “Transfer learning with YOLOV8 for real-time recognition system of American Sign Language Alphabet,” Franklin Open, vol. 8, p. 100165, 2024, https://doi.org/10.1016/j.fraope.2024.100165.

K. Wangchuk, P. Riyamongkol, and R. Waranusast, “Bhutanese Sign Language Alphabets Recognition Using Convolutional Neural Network,” InCIT 2020 - 5th Int. Conf. Inf. Technol., pp. 44–49, 2020, https://doi.org/10.1109/InCIT50588.2020.9310955.

S. Aiouez, A. Hamitouche, M. Belmadoui, K. Belattar, and F. Souami, “Real-time Arabic Sign Language Recognition based on YOLOv5,” SCITEPRESS, vol. 17–25, pp. 17–25, 2022, https://doi.org/10.5220/0010979300003209.

R. A. Alawwad, O. Bchir, and M. M. Ben Ismail, “Arabic Sign Language Recognition using Faster R-CNN,” Int. J. Adv. Comput. Sci. Appl., vol. 12, no. 3, pp. 692–700, 2021, https://doi.org/10.14569/IJACSA.2021.0120380.

S. T. Abd Al-Latief, S. Yussof, A. Ahmad, S. M. Khadim, and R. A. Abdulhasan, “Instant Sign Language Recognition by WAR Strategy Algorithm Based Tuned Machine Learning,” Int. J. Networked Distrib. Comput., vol. 12, no. 2, pp. 344–361, 2024, https://doi.org/10.1007/s44227-024-00039-8.

Y. Saleh and G. F. Issa, “Arabic sign language recognition through deep neural networks fine-tuning,” Int. J. online Biomed. Eng., vol. 16, no. 5, pp. 71–83, 2020, https://doi.org/10.3991/ijoe.v16i05.13087.

Y. C. Chiu, C. Y. Tsai, M. Da Ruan, G. Y. Shen, and T. T. Lee, “Mobilenet-SSDv2: An Improved Object Detection Model for Embedded Systems,” 2020 Int. Conf. Syst. Sci. Eng. ICSSE 2020, pp. 0–4, 2020, https://doi.org/10.1109/ICSSE50014.2020.9219319.

M. Kumar and R. Mann, “Masked Face Recognition using Deep Learning Model,” Proc. - 2021 3rd Int. Conf. Adv. Comput. Commun. Control Networking, ICAC3N 2021, pp. 428–432, 2021, https://doi.org/10.1109/ICAC3N53548.2021.9725368.

S. D. K, S. K, D. T, S. R, S. S. R. S and T. V. Dixit, "Efficient Object Detection on Low-Resource Devices Using Lightweight MobileNet-SSD," 2025 International Conference on Intelligent Systems and Computational Networks (ICISCN), pp. 1-6, 2025, https://doi.org/10.1109/ICISCN64258.2025.10934442.

N. H. Amir, C. Kusuma, and A. Luthfi, “Refining the Performance of Neural Networks with Simple Architectures for Indonesian Sign Language System ( SIBI ) Letter Recognition Using Keypoint Detection,” Ilk. J. Ilm., vol. 17, no. 1, pp. 64–73, 2025, https://doi.org/10.33096/ilkom.v17i1.2522.64-73.

S. Sharma and S. Singh, “Recognition of Indian Sign Language (ISL) Using Deep Learning Model,” Wirel. Pers. Commun., vol. 123, no. 1, pp. 671–692, 2022, https://doi.org/10.1007/s11277-021-09152-1.

E. Martinez-Martin and F. Morillas-Espejo, “Deep Learning Techniques for Spanish Sign Language Interpretation,” Comput. Intell. Neurosci., vol. 2021, no. i, 2021, https://doi.org/10.1155/2021/5532580.

A. Kannoth, C. Yang and M. A. Guanipa Larice, "Hand Gesture Recognition Using CNN & Publication of World's Largest ASL Database," 2021 IEEE Symposium on Computers and Communications (ISCC), pp. 1-6, 2021, https://doi.org/10.1109/ISCC53001.2021.9631255.

D. Liu, S. Gao, W. Chi, and D. Fan, “Pedestrian detection algorithm based on improved SSD,” Int. J. Comput. Appl. Technol., vol. 65, no. 1, pp. 25–35, 2021, https://doi.org/10.1504/IJCAT.2021.113643.

H. Wang and X. Wang, “Chinese Sign Language Alphabet Recognition Based on Random Forest Algorithm,” pp. 340–344, 2020, https://doi.org/10.1109/MetroInd4.0IoT48571.2020.9138285.

D. Indra, S. Madenda, and E. P. Wibowo, “Feature Extraction of Bisindo Alphabets Using Chain Code Contour,” Int. J. Eng. Technol., vol. 9, no. 4, pp. 3415–3419, 2017, https://doi.org/10.21817/ijet/2017/v9i4/170904142.

K. Tong, Y. Wu, and F. Zhou, “Recent advances in small object detection based on deep learning : A review,” Image Vis. Comput., vol. 97, p. 103910, 2020, https://doi.org/10.1016/j.imavis.2020.103910.

H. W. Herwanto, A. N. Handayani, K. L. Chandrika, and A. P. Wibawa, “Zoning Feature Extraction for Handwritten Javanese Character Recognition,” ICEEIE 2019 - Int. Conf. Electr. Electron. Inf. Eng. Emerg. Innov. Technol. Sustain. Futur., pp. 264–268, 2019, https://doi.org/10.1109/ICEEIE47180.2019.8981462.

A. Mujahid et al., “Real-Time Hand Gesture Recognition Based on Deep Learning YOLOv3 Model,” Appl. Sci., vol. 11, no. 9, 2021, https://doi.org/10.3390/app11094164.

S. Gharge, A. Patil, S. Patel, V. Shetty and N. Mundhada, "Real-time Object Detection using Haar Cascade Classifier for Robot Cars," 2023 4th International Conference on Electronics and Sustainable Communication Systems (ICESC), Coimbatore, India, 2023, pp. 64-70, 2023, https://doi.org/10.1109/ICESC57686.2023.10193401.

A. Kloska et al., “Influence of augmentation on the performance of the double ResNet-based model for chest X-ray classification,” Polish J. Radiol., vol. 88, pp. e244–e250, 2023, https://doi.org/10.5114/pjr.2023.126717.

B. Li, Y. Hou, and W. Che, “Data augmentation approaches in natural language processing: A survey,” AI Open, vol. 3, pp. 71–90, 2022, https://doi.org/10.1016/j.aiopen.2022.03.001.

H. Zhao, H. Zhang, and Y. Zhao, “YOLOv7-sea: Object Detection of Maritime UAV Images based on Improved YOLOv7,” Proc. - 2023 IEEE/CVF Winter Conf. Appl. Comput. Vis. Work. WACVW 2023, pp. 233–238, 2023, https://doi.org/10.1109/WACVW58289.2023.00029.

A. Buslaev, V. I. Iglovikov, E. Khvedchenya, A. Parinov, M. Druzhinin, and A. A. Kalinin, “Albumentations: Fast and flexible image augmentations,” Inf., vol. 11, no. 2, pp. 1–20, 2020, https://doi.org/10.3390/info11020125.

R. A. Asmara et al., “Chicken meat freshness identification using colors and textures feature,” 2018 Jt. 7th Int. Conf. Informatics, Electron. Vis. 2nd Int. Conf. Imaging, Vis. Pattern Recognition, ICIEV-IVPR 2018, pp. 93–98, 2018, https://doi.org/10.1109/ICIEV.2018.8640992.

A. Juneja, S. Juneja, A. Soneja, and S. Jain, “Real time object detection using CNN based single shot detector model,” J. Inf. Technol. Manag., vol. 13, no. 1, pp. 62–80, 2021, https://doi.org/10.22059/jitm.2021.80025.

M. Zaki, “Hand Keypoint-Based CNN for SIBI Sign Language Recognition,” Int. J. Robot. Control Syst., vol. 5, no. 2, pp. 813–829, 2025, https://doi.org/10.31763/ijrcs.v5i2.1745.

A. Asokan, D. E. Popescu, J. Anitha, and D. J. Hemanth, “Bat algorithm based non-linear contrast stretching for satellite image enhancement,” Geosci., vol. 10, no. 2, pp. 1–12, 2020, https://doi.org/10.3390/geosciences10020078.

L. H. Zhenghao Shi, Yaning Feng, Minghua Zhao, Erhu Zhang, “Normalised gamma transformation‐based contrast‐limited adaptive histogram equalisation.pdf.” p. 747, 2019, https://doi.org/10.1049/iet-ipr.2019.0992.

A. Kumar, R. K. Jha, and N. K. Nishchal, “An improved Gamma correction model for image dehazing in a multi-exposure fusion framework,” J. Vis. Commun. Image Represent., vol. 78, p. 103122, 2021, https://doi.org/10.1016/j.jvcir.2021.103122.

M. Al-Faris, J. Chiverton, D. Ndzi, and A. I. Ahmed, “A review on computer vision-based methods for human action recognition,” J. Imaging, vol. 6, no. 6, 2020, https://doi.org/10.3390/jimaging6060046.

A. M. Sarosa, A. Badriyah, R. A. Asmara, M. K. Wardani, D. F. Al Riza, and Y. Mulyani, “Performance Analysis of MobileNET on Grape Leaf Disease Detection,” 2024 Int. Conf. Adv. Inf. Sci. Dev. (ICAISD). IEEE, vol. 64–68, 2024, https://doi.org/10.1109/ICAISD63055.2024.10894920.

G R. Andrie Asmara, M. Ridwan and G. Budiprasetyo, "Haar Cascade and Convolutional Neural Network Face Detection in Client-Side for Cloud Computing Face Recognition," 2021 International Conference on Electrical and Information Technology (IEIT), pp. 1-5, 2021, https://doi.org/10.1109/IEIT53149.2021.9587388.

I. P. Sari, “Closer Look at Image Classification for Indonesian Sign Language with Few-Shot Learning Using Matching Network Approach,” Int. J. Informatics Vis., vol. 7, no. 3, pp. 638–643, 2023, https://doi.org/10.30630/joiv.7.3.1320.

R. A. Asmara et al., “YOLO-based object detection performance evaluation for automatic target aimbot in first-person shooter games,” Bull. Electr. Eng. Informatics, vol. 13, no. 4, pp. 2456–2470, 2024, https://doi.org/10.11591/eei.v13i4.6895.

R. A. Asmara, B. Syahputro, D. Supriyanto, and A. N. Handayani, “Prediction of Traffic Density using YOLO Object Detection and Implemented in Raspberry Pi 3b + and Intel NCS 2,” 4th Int. Conf. Vocat. Educ. Training, ICOVET 2020, pp. 391–395, 2020, https://doi.org/10.1109/ICOVET50258.2020.9230145.