Image Classification of Wayang Using Transfer Learning and Fine-Tuning of CNN Models

Muhammad Banjaransari; Adhi Prahara

doi:10.12928/biste.v5i4.9977

Authors

Muhammad Banjaransari Universitas Ahmad Dahlan
Adhi Prahara Universitas Ahmad Dahlan

DOI:

https://doi.org/10.12928/biste.v5i4.9977

Keywords:

Wayang, Transfer Learning, CNN, Classification, Computer Vision

Abstract

Wayang (shadow puppetry) is a traditional puppetry used in a performance to tell a story about the heroism of its main characters. Wayang has gained recognition as a cultural masterpiece by UNESCO. However, this cultural heritage now declining and not many people know about wayang. One of the solutions is using computer vision technology to classify wayang images. In this research, a transfer learning approach using Convolutional Neural Network (CNN) models namely MobileNetV2 and VGG16 followed by fine-tuning was proposed to classify wayang. The dataset consists of 3,000 images divided into 30 classes. This data is split into training and test data that are utilized for training and evaluating the model. Based on the evaluation, the MobileNetV2 model achieved precision, recall, F1-score, and accuracy of 95%, 94%, 94%, and 94.17%, respectively. Meanwhile, the VGG-16 model obtained 93% for all metrics. It can be concluded that transfer learning and fine-tuning using the MobileNetV2 model produces the best result in classifying wayang images compared to the VGG16 model. With good performance, the proposed method can be implemented on mobile applications to provide information about wayang from the captured images, thus indirectly supporting the preservation of cultural heritage in Indonesia.

References

B. H. Prilosadoso, B. Pujiono, S. Supeni, and B. W. Setyawan, “Wayang beber animation media as an effort for preserving wayang tradition based on information and technology,” In Journal of Physics: Conference Series, vol. 1339, no. 1, p. 012109, 2019, https://doi.org/10.1088/1742-6596/1339/1/012109.

J. C. Oman. The Stories of the Ramayana and the Mahabharata. Routledge. 2019. https://books.google.co.id/books?hl=id&lr=&id=0I2lDwAAQBAJ.

R. C. I. Prahmana and A. Istiandaru, “Learning sets theory using shadow puppet: A study of Javanese ethnomathematics,” Mathematics, vol. 9, no. 22, p. 2938. 2021, https://doi.org/10.3390/math9222938.

D. P. Prabowo, M. K. A. Nugraha, D. I. I. Ulumuddin, R. A. Pramunendar and S. Santosa, "Indonesian Traditional Shadow Puppet Classification using Convolutional Neural Network," 2021 International Seminar on Application for Technology of Information and Communication (iSemantic), pp. 1-5, 2021, https://doi.org/10.1109/iSemantic52711.2021.9573218.

M. R. A. Yudianto and H. Al Fatta, “The effect of Gaussian filter and data preprocessing on the classification of Punakawan puppet images with the convolutional neural network algorithm,” International Journal of Electrical & Computer Engineering (2088-8708), vol. 12, no. 4, 2022, https://doi.org/10.11591/ijece.v12i4.pp3752-3761.

W. Supriyanti and D. A. Anggoro, “Classification of Pandavas Figure in Shadow Puppet Images using Convolutional Neural Networks,” Khazanah Informatika: Jurnal Ilmu Komputer dan Informatika, vol. 7, no. 1, 2021, https://doi.org/10.23917/khif.v7i1.12484.

A. Mustafid, M. M. Pamuji, and S. Helmiyah, “A Comparative Study of Transfer Learning and Fine-Tuning Method on Deep Learning Models for Wayang Dataset Classification,” IJID (International J. Informatics Dev., vol. 9, no. 2, pp. 100–110, 2020, https://doi.org/10.14421/ijid.2020.09207.

A. P. Wibawa, W. A. Y. Pratama, A. N. Handayani, and A. Ghosh, “Convolutional Neural Network (CNN) to determine the character of wayang kulit,” Int. J. Vis. Perform. Arts, vol. 3, no. 1, pp. 1–8, 2021, https://doi.org/10.31763/viperarts.v3i1.373.

I. B. K. Sudiatmika, M. Artana, N. W. Utami, M. A. P. Putra, and E. G. A. Dewi, “Mask R-CNN for indonesian shadow puppet recognition and classification,” in Journal of Physics: Conference Series, 2021, vol. 1783, no. 1, p. 12032, https://doi.org/10.1088/1742-6596/1783/1/012032.

Muhathir and Al-Khowarizmi, "Measuring the Accuracy of SVM with Varying Kernel Function for Classification of Indonesian Wayang on Images," 2020 International Conference on Decision Aid Sciences and Application (DASA), pp. 1190-1196, 2020, https://doi.org/10.1109/DASA51403.2020.9317197.

M. R. Arif Yudianto, R. A. Hasani, and P. Sukmasetya, “Study comparison of color channel to median filter in wayang image using convolutional neural network algorithm,” in AIP Conference Proceedings, vol. 2706, no. 1, 2023, https://doi.org/10.1063/5.0120384.

N. Khairina, R. K. I. Barus, M. Ula, I. Sahputra, and others, “Preserving Cultural Heritage Through AI: Developing LeNet Architecture for Wayang Image Classification,” Int. J. Adv. Comput. Sci. Appl., vol. 14, no. 9, 2023, https://doi.org/10.14569/IJACSA.2023.0140919.

Lis, M. (2019). The Art of Wayang. Art of the Orient, 8, 137-150, 2019, https://doi.org/10.15804/aoto201907.

K. Ghosh, C. Bellinger, R. Corizzo, P. Branco, B. Krawczyk, and N. Japkowicz, “The class imbalance problem in deep learning,” Mach. Learn., pp. 1–57, 2022, https://doi.org/10.1007/s10994-022-06268-8.

J. Folmsbee, S. Johnson, X. Liu, M. Brandwein-Weber, and S. Doyle, “Fragile neural networks: the importance of image standardization for deep learning in digital pathology,” in Medical Imaging 2019: Digital Pathology, vol. 10956, pp. 222–228, 2019, https://doi.org/10.1117/12.2512992.

J. D. Kelleher. Deep learning. MIT press. 2019. https://books.google.co.id/books?hl=id&lr=&id=b06qDwAAQBAJ.

P. N. Srinivasu, J. G. SivaSai, M. F. Ijaz, A. K. Bhoi, W. Kim, and J. J. Kang, “Classification of skin disease using deep learning neural networks with MobileNet V2 and LSTM,” Sensors, vol. 21, no. 8, p. 2852, 2021, https://doi.org/10.3390/s21082852.

S. R. Joshi and M. El-Sharkawy, "HBONext: HBONet with Flipped Inverted Residual," 2021 IEEE International Conference on Design & Test of Integrated Micro & Nano-Systems (DTS), pp. 1-5, 2021, https://doi.org/10.1109/DTS52014.2021.9498121.

Z. Zhu, et al., “Webface260m: A benchmark unveiling the power of million-scale deep face recognition,” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10492-10502, 2021, https://doi.org/10.1109/CVPR46437.2021.01035.

N. Kumar, M. Rathee, N. Chandran, D. Gupta, A. Rastogi and R. Sharma, "CrypTFlow: Secure TensorFlow Inference," 2020 IEEE Symposium on Security and Privacy (SP), pp. 336-353, 2020, https://doi.org/10.1109/SP40000.2020.00092.

C. Tan, F. Sun, T. Kong, W. Zhang, C. Yang, and C. Liu, “A survey on deep transfer learning,” in Artificial Neural Networks and Machine Learning--ICANN 2018: 27th International Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, Proceedings, Part III 27, pp. 270–279, 2018, https://doi.org/10.1007/978-3-030-01424-7_27.

G. Vrbančič and V. Podgorelec, "Transfer Learning With Adaptive Fine-Tuning," in IEEE Access, vol. 8, pp. 196197-196211, 2020, https://doi.org/10.1109/ACCESS.2020.3034343.

M. Grandini, E. Bagli, and G. Visani, “Metrics for multi-class classification: an overview,” arXiv Prepr. arXiv2008.05756, 2020, https://doi.org/10.48550/arXiv.2008.05756.