Review on User Interaction for Robotic Arm in Digital Twin
DOI:
https://doi.org/10.12928/biste.v8i2.15851Keywords:
Virtual Robotic Arm Control, Human–Computer Interaction, Motion Controllers, Hand Gesture Recognition, Usability Evaluation, Digital Twin, Virtual Reality, Haptic FeedbackAbstract
The integration of human interaction techniques in digital twin (DT) systems has become increasingly important in manufacturing, industrial automation, and remote operations, particularly for robotic arm control. However, existing approaches joystick control, gesture-based input, and virtual reality (VR) are often disconnected across modalities, limiting effectiveness in real-time environments. The research contribution is a systematic literature review (SLR) that critically analyzes and synthesizes interaction techniques to identify performance trends, evaluation gaps, and design challenges in virtual robotic arm control within digital twin frameworks. The review covers studies published between 2020 and 2025, selected to reflect the rapid emergence of immersive technologies in real-time digital twin systems. Following PRISMA 2020 guidelines, 180 records were identified from IEEE Xplore, Scopus, and Web of Science, from which 77 peer-reviewed studies were selected. Interaction techniques were evaluated using task completion time, positional accuracy, NASA Task Load Index (NASA-TLX), and System Usability Scale (SUS). The findings reveal that VR-based techniques dominate due to their intuitiveness and immersive experience in human-in-the-loop control. However, evaluation remains inconsistent across studies, with significant variation in metrics and experimental setups. Latency and synchronization were identified as critical challenges in real-time control, where delays degrade precision and responsiveness. Traditional methods such as joysticks offer stability but lack the natural interaction of immersive techniques. These findings underscore the need for standardized evaluation frameworks and improved synchronization strategies, offering practical guidance for designing robust, human-centered digital twin interaction systems for robotic arms.
References
D. Jones, C. Snider, A. Nassehi, J. Yon, and B. Hicks, “Characterising the digital twin: A systematic literature review,” CIRP Journal of Manufacturing Science and Technology, vol. 29, pp. 36–52, 2020, https://doi.org/10.1016/j.cirpj.2020.02.002.
Y. Lu, C. Liu, I. K. Wang, H. Huang, and X. Xu, “Digital twin-driven smart manufacturing: Connotation, reference model, applications and research issues,” Robotics and Computer-Integrated Manufacturing, vol. 61, p. 101837, 2020, https://doi.org/10.1016/j.rcim.2019.101837.
A. Mazumder, M. Sahed, Z. Tasneem, M. S. Kaiser, and M. Mahmud, “Towards next generation digital twin in robotics: Trends, scopes, challenges and future,” Heliyon, vol. 9, no. 6, p. e16889, 2023, https://doi.org/10.1016/j.heliyon.2023.e16889.
A. P. Burghardt, J. Szybicki, P. Gierlak, K. Musial, P. Pietruś, R. Cygan, “Programming of Industrial Robots Using Virtual Reality and Digital Twins,” Applied Sciences, vol. 10, no. 2, p. 486, 2020, https://doi.org/10.3390/app10020486.
W. Wu, J. Liu, H. Zhang, and Y. Li, “Research on guidance methods of digital twin robotic arms,” Sensors, vol. 23, no. 7, p. 3521, 2023, https://doi.org/10.3390/s23073521.
D. Mourtzis, J. Angelopoulos, N. Panopoulos, “Smart Manufacturing and Tactile Internet Based on 5G in Industry 4.0: Challenges, Applications and New Trends,” Electronics, vol. 10, no. 24, p. 3175, 2021, https://doi.org/10.3390/electronics10243175.
J. J. Lopez-Huanca, C. Flores-Urizar, A. Ramos-Cabezas, “Augmented Reality for Human–Robot Interaction in Collaborative Industrial Environments,” Sensors, vol. 22, no. 3, P. 808, 2022, https://doi.org/10.3390/s22030808.
V. Villani, F. Pini, F. Leali, and C. Secchi, “Survey on human–robot collaboration in industrial settings: Safety, intuitive interfaces and applications,” Mechatronics, vol. 55, pp. 248–266, 2020, https://doi.org/10.1016/j.mechatronics.2018.02.009.
K. Wan, Y. Wang, J. Zhang, and X. Li, “A virtual reality-based immersive teleoperation system for robotic manipulation,” Journal of Manufacturing Systems, vol. 72, pp. 230–243, 2024, https://doi.org/10.1016/j.jmsy.2023.12.004.
A. Vysocký, M. Novák, and P. Bartoš, “Hand gesture interface for robot path definition in collaborative workspaces,” Sensors, vol. 23, no. 5, p. 2417, 2023, https://doi.org/10.3390/s23052417.
C. Savur. A Physiological Computing System to Improve Human-Robot Collaboration by Using Human Comfort Index. Rochester Institute of Technology, 2022, https://doi.org/10.3390/machines11050536.
S. Hopko, J. Wang, and R. Mehta, “Human factors considerations and metrics in shared space human-robot collaboration: A systematic review,” Frontiers in Robotics and AI, vol. 9, p. 799522, 2022, https://doi.org/10.3389/frobt.2022.799522.
E. Artetxe, O. Barambones, I. Calvo, P. Fernández-Bustamante, I. Martin, and J. Uralde, “Wireless technologies for industry 4.0 applications,” Energies, vol. 16, no. 3, p. 1349, 2023, https://doi.org/10.3390/en16031349.
U. Asad, M. Khan, A. Khalid, and W. A. Lughmani, “Human-centric digital twins in industry: A comprehensive review of enabling technologies and implementation strategies,” Sensors, vol. 23, no. 8, p. 3938, 2023, https://doi.org/10.3390/s23083938.
S. Mancin, M. Sguanci, D. Andreoli, F. Soekeland, G. Anastasi, M. Piredda, and M. G. De Marinis, “Systematic review of clinical practice guidelines and systematic reviews: a method for conducting comprehensive analysis,” MethodsX, vol. 12, p. 102532, 2024, https://doi.org/10.1016/j.mex.2023.102532.
B. Kitchenham, D. Budgen, and O. P. Brereton, “Evidence-based software engineering and systematic reviews,” IEEE Software, vol. 28, no. 1, pp. 28–32, 2011, https://doi.org/10.1109/MS.2010.122.
T. Arksey and L. O’Malley, “Scoping studies: Towards a methodological framework,” International Journal of Social Research Methodology, vol. 8, no. 1, pp. 19–32, 2005, https://doi.org/10.1080/1364557032000119616.
P. Runeson and M. Höst, “Guidelines for conducting and reporting case study research in software engineering,” IEEE Transactions on Software Engineering, vol. 35, no. 2, pp. 131–164, 2009, https://doi.org/10.1109/TSE.2009.6.
D. Page et al., “The PRISMA 2020 statement: An updated guideline for reporting systematic reviews,” BMJ, vol. 372, p. n71, 2021, https://doi.org/10.1136/bmj.n71.
M. L. Rethlefsen et al., “PRISMA-S: An extension to the PRISMA statement for reporting literature searches in systematic reviews,” Journal of the Medical Library Association, vol. 109, no. 2, pp. 174–200, 2021, https://doi.org/10.5195/jmla.2021.962.
M. Gusenbauer and N. R. Haddaway, “Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources,” PLOS ONE, vol. 15, no. 7, p. e0238237, 2020, https://doi.org/10.1371/journal.pone.0238237.
H. Snyder, “Literature review as a research methodology: An overview and guidelines,” Journal of Business Research, vol. 104, pp. 333–339, 2019, https://doi.org/10.1016/j.jbusres.2019.07.039.
A. Carrera-Rivera, W. Ochoa, F. Larrinaga, and G. Lasa, “How-to conduct a systematic literature review: A quick guide for computer science research,” MethodsX, vol. 9, p. 101895, 2022, https://doi.org/10.1016/j.mex.2022.101895.
J. Wohlin et al., “Guidelines for snowballing in systematic literature studies and a replication in software engineering,” Information and Software Technology, vol. 64, pp. 1–12, 2019, https://doi.org/10.1016/j.infsof.2015.03.006.
D. Moher, A. Liberati, J. Tetzlaff, and D. G. Altman, “Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement,” PLoS Medicine, vol. 6, no. 7, p. e1000097, 2009, https://doi.org/10.1371/journal.pmed.1000097.
S. Castillo and P. Grbovic, "The APISSER Methodology for Systematic Literature Reviews in Engineering," in IEEE Access, vol. 10, pp. 23700-23707, 2022, https://doi.org/10.1109/ACCESS.2022.3148206.
B. Kitchenham, D. Budgen, and O. P. Brereton, “Using mapping studies as the basis for further research – A participant-observer case study,” Information and Software Technology, vol. 53, no. 6, pp. 638–651, 2011, https://doi.org/10.1016/j.infsof.2010.12.011.
K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson, “Systematic mapping studies in software engineering,” Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering, pp. 68–77, 2008, https://doi.org/10.14236/ewic/EASE2008.8.
S. A. Perrig, L. F. Aeschbach, N. Scharowski, N. von Felten, K. Opwis, and F. Brühlmann, “Measurement practices in user experience (UX) research: a systematic quantitative literature review,” Frontiers in Computer Science, vol. 6, p. 1368860, 2024, https://doi.org/10.3389/fcomp.2024.1368860.
E. Negri, L. Fumagalli, and M. Macchi, “A review of the roles of digital twin in CPS-based production systems,” Procedia Manufacturing, vol. 51, pp. 939–948, 2020, https://doi.org/10.1016/j.promfg.2020.10.131.
F. Tao and Q. Qi, “Make more digital twins,” Nature, vol. 573, pp. 490–491, 2019, https://doi.org/10.1038/d41586-019-02849-1.
Q. Qi, F. Tao, T. Hu, N. Anwer, A. Liu, Y. Wei, L. Wang, A. Y. C. Nee, “Enabling technologies and tools for digital twin,” Journal of Manufacturing Systems, vol. 58, pp. 3–21, 2021, https://doi.org/10.1016/j.jmsy.2019.10.001.
J. Liu, H. Zhang, and Y. Li, “Digital twin-driven robotic system for intelligent manufacturing,” IEEE Access, vol. 9, pp. 135482–135493, 2021, https://doi.org/10.1109/ACCESS.2021.3116114.
C. Li, P. Zheng, S. Li, Y. Pang, and C. K. Lee, “AR-assisted digital twin-enabled robot collaborative manufacturing system with human-in-the-loop,” Robotics and Computer-Integrated Manufacturing, 76, 102321, 2022, https://doi.org/10.1016/j.rcim.2022.102321.
S. Boschert and R. Rosen, “Digital twin—The simulation aspect,” in Mechatronic Futures, pp. 59–74, 2021, https://doi.org/10.1007/978-3-030-65726-0_5.
R. Fuller, L. Fan, C. Day, C. Barlow, “Digital Twin: Enabling Technologies, Challenges and Open Research,” IEEE Access, vol. 8, pp. 108952–108971, 2020, https://doi.org/10.1109/ACCESS.2020.2998358.
A. Rasheed, O. San, T. Kvamsdal, “Digital Twin: Values, Challenges and Enablers From a Modeling Perspective,” IEEE Access, vol. 8, pp. 21980–22012, 2020, https://doi.org/10.1109/ACCESS.2020.2970143.
T. Klauber, S. Zander, J. Straub, “Virtual Reality as an Interface for Human–Robot Collaboration in Precision Manufacturing,” Robotics, vol. 11, no. 5, P. 94, 2022, https://doi.org/10.3390/robotics11050094.
S. Kim, J. Lee, and K. Kim, “Immersive virtual reality-based teleoperation of a robotic manipulator,” IEEE Access, vol. 8, pp. 174019–174030, 2020, https://doi.org/10.1109/ACCESS.2020.3025778.
B. R. Galarza, P. Ayala, S. Manzano, and M. V. Garcia, “Virtual reality teleoperation system for mobile robot manipulation,” Robotics, vol. 12, no. 6, p. 163, 2023, https://doi.org/10.3390/robotics12060163.
P. Suárez-Fernández, J. Albo-Canals, C. Martell, “Interaction Design Principles for Teleoperated Robotic Systems in Industrial Applications,” Applied Sciences, vol. 12, no. 7, p. 3502, 2022, https://doi.org/10.3390/app12073502.
A. Ajoudani et al., “Progress and prospects of the human–robot collaboration,” Autonomous Robots, vol. 44, pp. 957–975, 2020, https://doi.org/10.1007/s10514-019-09914-1.
J. Rosen, M. Brand, M. B. Fuchs, and M. Arcan, “A myosignal-based powered exoskeleton system,” IEEE Transactions on Systems, Man, and Cybernetics, vol. 31, no. 3, pp. 210–222, 2020, https://doi.org/10.1109/3468.925661.
H. Li et al., "Towards the Design and Optimization of a Local Teleoperation Cockpit for Customs Remote Inspection," 2025 IEEE International Conference on Robotics and Biomimetics (ROBIO), pp. 2105-2110, 2025, https://doi.org/10.1109/ROBIO66223.2025.11377237.
J. L. Raheja, K. Das, A. Chaudhary, “Gesture Control of Industrial Robots Using Convolutional Neural Networks,” Computers and Electrical Engineering, vol. 91, P. 107043, 2021, https://doi.org/10.1016/j.compeleceng.2021.107043.
A. Fang, H. Zhong, P. Zhao, “A Real-Time Hand Gesture Recognition and Human–Robot Interaction System,” Sensors, vol. 21, no. 22, p. 7556, 2021, https://doi.org/10.3390/s21227556.
V. F. de Oliveira, G. Matiolli, C. J. B. Júnior, R. Gaspar and R. G. Lins, "Digital Twin and Cyber-Physical System Integration in Commercial Vehicles: Latest Concepts, Challenges and Opportunities," in IEEE Transactions on Intelligent Vehicles, vol. 9, no. 4, pp. 4804-4819, 2024, https://doi.org/10.1109/TIV.2024.3378579.
M. Peral, A. Sanfeliu and A. Garrell, "Efficient Hand Gesture Recognition for Human-Robot Interaction," in IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 10272-10279, 2022, https://doi.org/10.1109/LRA.2022.3193251.
H. Lee, S. D. Kim, and M. A. U. Al Amin, “Control framework for collaborative robot using imitation learning-based teleoperation from human digital twin to robot digital twin,” Mechatronics, vol. 85, p. 102833, 2022, https://doi.org/10.1016/j.mechatronics.2022.102833.
D. -N. Song, W. -C. Tang, Y. -N. Zhao, Y. -G. Zhong and J. -W. Ma, "Convolution-Based Velocity-Smoothing Principle and Its Application to Real-Time Parametric Curve Interpolation," in IEEE Transactions on Automation Science and Engineering, vol. 22, pp. 23443-23454, 2025, https://doi.org/10.1109/TASE.2025.3625244.
K. Foteinos, M. Linardakis, P. Radoglou-Grammatikis, V. Argyriou, P. Sarigiannidis, I. Varlamis, and G. T. Papadopoulos, “Visual hand gesture recognition with deep learning: A comprehensive review of methods, datasets, challenges and future research directions,” arXiv preprint arXiv:2507.04465, 2025, https://doi.org/10.48550/arXiv.2507.04465.
A. Khalil and J. Kwon, “Nonlinear Performance Degradation of Vision-Based Teleoperation under Network Latency,” arXiv preprint arXiv:2603.06850, 2026, https://doi.org/10.48550/arXiv.2603.06850.
M. Zhu, S. He, T. Chen, and C. Lee, “Wearable Intelligent Human–Machine Interfaces Ready for Sustainable Edge Computing Systems,” AI Sensors, vol. 1, no. 2, p. 9, 2025, https://doi.org/10.3390/aisens1020009.
R. E. Kondo et al., “An industrial edge computing architecture for Local Digital Twin,” Computers & Industrial Engineering, vol. 193, p. 110257, 2024, https://doi.org/10.1016/j.cie.2024.110257.
H. Culbertson, S. B. Schorr, A. M. Okamura, “Haptics: The Present and Future of Artificial Touch Sensation,” Annual Review of Control, Robotics, and Autonomous Systems, vol. 1, pp. 385–409, 2018, https://doi.org/10.1146/annurev-control-060117-105043.
M. S. Nawaz, S. U. R. Khan, S. Hussain, and J. Iqbal, “A study on application programming interface recommendation: state-of-the-art techniques, challenges and future directions,” Library Hi Tech, vol. 41, no. 2, pp. 355-385, 2023, https://doi.org/10.1108/LHT-02-2022-0103.
A. Baratta, A. Cimino, F. Longo, and L. Nicoletti, “Digital twin for human-robot collaboration enhancement in manufacturing systems: Literature review and direction for future developments,” Computers & Industrial Engineering, vol. 187, p. 109764, 2024, https://doi.org/10.1016/j.cie.2023.109764.
N. Mizuno, Y. Tazaki, T. Hashimoto, and Y. Yokokohji, “A comparative study of manipulator teleoperation methods for debris retrieval phase in nuclear power plant decommissioning,” Advanced Robotics, vol. 37, no. 9, pp. 541-559, 2023, https://doi.org/10.1080/01691864.2023.2169588.
M. T. B. Touhid, E. Zhu, M. V. Ehteshamfara, and S. Yang, “Evaluation of digital twin synchronization in robotic assembly using YOLOv8,” The International Journal of Advanced Manufacturing Technology, vol. 134, no. 1, pp. 871-885, 2024, https://doi.org/10.1007/s00170-024-14182-7.
S. Hart, “NASA Task Load Index (NASA-TLX): 20 years later,” Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 64, no. 1, pp. 904–908, 2020, https://doi.org/10.1177/1071181320641236.
Y. Kim, H. Park, and J. Lee, “Usability evaluation of immersive teleoperation interfaces for robotic manipulation,” IEEE Access, vol. 9, pp. 147920–147931, 2021, https://doi.org/10.1109/ACCESS.2021.3123864.
S. Gholami, M. Lorenzini, E. De Momi and A. Ajoudani, "Quantitative Physical Ergonomics Assessment of Teleoperation Interfaces," in IEEE Transactions on Human-Machine Systems, vol. 52, no. 2, pp. 169-180, April 2022, https://doi.org/10.1109/THMS.2022.3149167.
W. Fan et al., "Digital Twin-Driven Mixed Reality Framework for Immersive Teleoperation With Haptic Rendering," in IEEE Robotics and Automation Letters, vol. 8, no. 12, pp. 8494-8501, 2023, https://doi.org/10.1109/LRA.2023.3325784.
F. Chinello, M. Malvezzi, C. Pacchierotti, D. Prattichizzo, “Design and Characterization of a Kinesthetic Wearable Haptic Interface for the Hand,” IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 3293–3300, 2020, https://doi.org/10.1109/LRA.2020.2975705.
R. V. Patel, S. F. Atashzar and M. Tavakoli, "Haptic Feedback and Force-Based Teleoperation in Surgical Robotics," in Proceedings of the IEEE, vol. 110, no. 7, pp. 1012-1027, 2022, https://doi.org/10.1109/JPROC.2022.3180052.
M. Sarac, M. Solazzi, A. Frisoli, “Design Requirements of Generic Hand Exoskeletons and Survey of Hand Exoskeletons for Rehabilitation, Assistive, or Haptic Use,” IEEE Transactions on Haptics, vol. 12, no. 4, pp. 400–413, 2019, https://doi.org/10.1109/TOH.2019.2924881.
M. Walker, T. Phung, T. Chakraborti, T. Williams, and D. Szafir, “Virtual, augmented, and mixed reality for human-robot interaction: A survey and virtual design element taxonomy,” ACM Transactions on Human-Robot Interaction, vol. 12, no. 4, pp. 1-39, 2023, https://doi.org/10.1145/3597623.
B. Liang, N. Li, G. Li, J. Huang, Z. Yu and X. Zhao, "Sensing Technologies for Hand Gesture Recognition in Human–Robot Interaction: A Review," in IEEE Sensors Journal, vol. 26, no. 2, pp. 1501-1519, 2026, https://doi.org/10.1109/JSEN.2025.3635622.
Q. Ouyang et al., "Bio-Inspired Haptic Feedback for Artificial Palpation in Robotic Surgery," in IEEE Transactions on Biomedical Engineering, vol. 68, no. 10, pp. 3184-3193, 2021, https://doi.org/10.1109/TBME.2021.3076094.
J. H. Bong, S. Choi, J. Hong, and S. Park, “Force feedback haptic interface for bilateral teleoperation of robot manipulation,” Microsystem Technologies, vol. 28, no. 10, pp. 2381-2392, 2022, https://doi.org/10.1007/s00542-022-05382-w.
J. Endsley, “Human factors in automation: A review of issues and challenges,” IEEE Transactions on Human-Machine Systems, vol. 50, no. 6, pp. 521–531, 2020, https://doi.org/10.1109/THMS.2020.3001651.
M. Charalambous, S. Fletcher, P. Webb, “The Development of a Scale to Evaluate Trust in Industrial Human–Robot Collaboration,” International Journal of Social Robotics, vol. 8, no. 2, pp. 193–209, 2016, https://doi.org/10.1007/s12369-015-0333-8.
A. Chaddad, J. Peng, J. Xu, and A. Bouridane, “Survey of explainable AI techniques in healthcare,” Sensors, vol. 23, no. 2, p. 634, 2023, https://doi.org/10.3390/s23020634.
V. Kumbhar, G. Sireesha, V. D. Jadhav, A. Barve, R. Sharma and M. E. M. Soudagar, "Integrating Explainable AI with Human-in-the-Loop Systems for Transparent Decision-Making in Autonomous Robots," 2025 International Conference on Intelligent Communication Networks and Computational Techniques (ICICNCT), pp. 1-6, 2025, https://doi.org/10.1109/ICICNCT66124.2025.11232580.
A. K. Ramasubramanian, R. Mathew, M. Kelly, V. Hargaden, and N. Papakostas, “Digital twin for human–robot collaboration in manufacturing: Review and outlook,” Applied Sciences, vol. 12, no. 10, p. 4811, 2022, https://doi.org/10.3390/app12104811.
M. Simonetto, A. Montanari, T. Rossi, M. Giordani, “5G Wireless Connectivity for Industrial Robots: Key Challenges and Open Issues,” IEEE Communications Magazine, vol. 59, no. 2, pp. 28–34, 2021, https://doi.org/10.1109/MCOM.001.2000598.
A. W. Yu and A. Nayak, "The Internet of Humanoids: A Survey of Technologies, Applications, and Challenges," in IEEE Internet of Things Journal, vol. 13, no. 6, pp. 10498-10521, 2026, https://doi.org/10.1109/JIOT.2026.3653698.
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Aiman Hakim Azahari, Mohd Khalid Mokhtar, Nazreen Abdullasim, Mohd Hafiz Bin Zakaria, Asniyani Nur Haidar Binti Abdullah, Shafina Binti Abd Karim Ishigaki, Ikmal Faiq Albakri Mustafa Albakri, Muhamad Najib Zamri
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
This journal is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
