Meta-Heuristic Hyperparameter Optimization for Q-Learning: A Snake Optimizer Approach to Computation Offloading in the Industrial Internet of Things
DOI:
https://doi.org/10.12928/biste.v8i4.16274Keywords:
Computation Offloading, Hyperparameter Tuning, Industrial Internet of Things, Meta-Heuristic Optimization, Reinforcement Learning, Snake Optimizer, Three-Tier ArchitectureAbstract
The proliferation of computationally intensive Industrial Internet of Things (IIoT) applications requires offloading policies that reduce latency and energy consumption under constrained device, edge, and cloud resources. Although reinforcement learning (RL) is suitable for sequential offloading decisions, its performance is highly sensitive to hyperparameters that are often fixed or manually tuned. The research contribution is an automated QLSO framework that uses the Snake Optimizer (SO) to tune the learning rate, discount factor, and exploration-decay rate of tabular Q-learning (QL) in a three-tier IIoT-edge-cloud architecture. Unlike generic external Hyperparameter Optimization (HPO) methods, QLSO embeds SO as a derivative-free, population-based tuning layer within the QL offloading workflow, where candidate hyperparameters are evaluated based on latency-energy-aware performance under three-tier IIoT resource constraints. The problem is formulated as a Markov decision process with discrete local, edge, and cloud execution actions, and candidate hyperparameters are evaluated through episodic training followed by greedy validation. Across five independent random seeds, QLSO converged about 1.5× faster in the complex scenario and achieved lower latency and energy than the QL and SARSA baselines, while remaining close to the strongest raw-metric baseline, DQN (0.0493 s and 1.72×10⁻² J for QLSO versus 0.0488 s and 1.65×10⁻² J for DQN). However, QLSO achieved the highest latency-optimal decision rate (49.0%, +3.3 percentage points over DQN), and the ablation study showed a 9.1% fitness improvement over fixed-parameter QL. The framework remains simulation-based and single-agent, requiring validation with partial offloading, dynamic network models, and realistic platforms such as NS-3, CloudSim, EdgeCloudSim, or physical IIoT testbeds.
References
T. Qiu, J. Chi, X. Zhou, Z. Ning, M. Atiquzzaman, and D. O. Wu, “Edge Computing in Industrial Internet of Things: Architecture, Advances and Challenges,” IEEE Commun. Surv. Tutorials, vol. 22, no. 4, pp. 2462–2488, 2020, https://doi.org/10.1109/COMST.2020.3009103.
X. Jiao et al., “Deep Reinforcement Learning for Time-Energy Tradeoff Online Offloading in MEC-Enabled Industrial Internet of Things,” IEEE Trans. Netw. Sci. Eng., vol. 10, no. 6, pp. 3465–3479, 2023, https://doi.org/10.1109/TNSE.2023.3263169.
B. Chen, J. Wan, Y. Lan, M. Imran, D. Li, and N. Guizani, “Improving cognitive ability of edge intelligent IIoT through machine learning,” IEEE Netw., vol. 33, no. 5, pp. 61–67, 2019, https://doi.org/10.1109/MNET.001.1800505.
L. Huang, S. Bi, and Y. J. A. Zhang, “Deep reinforcement learning for online computation offloading in wireless powered mobile-edge computing networks,” IEEE Trans. Mob. Comput., vol. 19, no. 11, pp. 2581–2593, 2020, https://doi.org/10.1109/TMC.2019.2928811.
H. Yang, Y. Liang, J. Yuan, Q. Yao, A. Yu, and J. Zhang, “Distributed Blockchain-Based Trusted Multidomain Collaboration for Mobile Edge Computing in 5G and beyond,” IEEE Trans. Ind. Informatics, vol. 16, no. 11, pp. 7094–7104, 2020, https://doi.org/10.1109/TII.2020.2964563.
B. Xu et al., “Mobility-Aware Task Offloading in Industrial Fog Networks: A Submodular-Based MARL Approach,” IEEE Internet Things J., vol. 12, no. 8, pp. 10795–10807, 2025, https://doi.org/10.1109/JIOT.2024.3514095.
X. H. Deng, J. Yin, P. Y. Guan, N. N. Xiong, L. Zhang, and S. Mumtaz, “Intelligent Delay-Aware Partial Computing Task Offloading for Multiuser Industrial Internet of Things Through Edge Computing,” IEEE INTERNET THINGS J., vol. 10, no. 4, pp. 2954–2966, 2023, https://doi.org/10.1109/JIOT.2021.3123406.
W. Qin, H. Chen, L. Wang, Y. Xia, A. Nascita, and A. Pescapè, “MCOTM: Mobility-aware computation offloading and task migration for edge computing in industrial IoT,” Futur. Gener. Comput. Syst., vol. 151, no. March 2023, pp. 232–241, 2024, https://doi.org/10.1016/j.future.2023.10.004.
J. Cai, H. T. Fu, and Y. Liu, “Multitask Multiobjective Deep Reinforcement Learning-Based Computation Offloading Method for Industrial Internet of Things,” IEEE INTERNET THINGS J., vol. 10, no. 2, pp. 1848–1859, 2023, https://doi.org/10.1109/JIOT.2022.3209987.
M. Wu, W. Chen, L. Qian, L. Guo, and I. Lee, “Joint Service Caching and Secure Computation Offloading for Reconfigurable-Intelligent-Surface-Assisted Edge Computing Networks,” IEEE Internet Things J., vol. 11, no. 19, pp. 30469–30482, 2024, https://doi.org/10.1109/JIOT.2024.3404972.
X. Li et al., “ABM-SpConv-SIMD: Accelerating Convolutional Neural Network Inference for Industrial IoT Applications on Edge Devices,” IEEE Trans. Netw. Sci. Eng., vol. 10, no. 5, pp. 3071–3085, 2023, https://doi.org/10.1109/TNSE.2022.3154412.
J. Chen, H. Xing, Z. Xiao, L. Xu, and T. Tao, “A DRL Agent for Jointly Optimizing Computation Offloading and Resource Allocation in MEC,” IEEE Internet Things J., vol. 8, no. 24, pp. 17508–17524, 2021, https://doi.org/10.1109/JIOT.2021.3081694.
F. Zhang, G. J. Han, L. Liu, M. Martínez-García, and Y. Peng, “Deep Reinforcement Learning Based Cooperative Partial Task Offloading and Resource Allocation for IIoT Applications,” IEEE Trans. Netw. Sci. Eng., vol. 10, no. 5, pp. 2991–3006, 2023, https://doi.org/10.1109/TNSE.2022.3167949.
M. S. Hossain, C. I. Nwakanma, J. M. Lee, and D. S. Kim, “Edge computational task offloading scheme using reinforcement learning for IIoT scenario,” ICT EXPRESS, vol. 6, no. 4, pp. 291–299, 2020, https://doi.org/10.1016/j.icte.2020.06.002.
Y. Zhao, Z. Chai, Y. Li, H. Huang, and H. Kang, “Multi-Objective Computation Offloading Based on Decentralized Deep Reinforcement Learning in Industrial Internet of Things,” IEEE Trans. Cogn. Commun. Netw., vol. 11, no. 3, pp. 1940–1953, 2025, https://doi.org/10.1109/TCCN.2024.3466889.
S. Koo and Y. Lim, “A Cluster-Based Optimal Computation Offloading Decision Mechanism Using RL in the IIoT Field,” Appl. Sci., vol. 12, no. 1, 2022, https://doi.org/10.3390/app12010384.
Z. Zabihi, A. M. Eftekhari Moghadam, and M. H. Rezvani, “Reinforcement Learning Methods for Computation Offloading: A Systematic Review,” ACM Comput. Surv., vol. 56, no. 1, 2023, https://doi.org/10.1145/3603703.
K. H. Liu and W. Liao, “Intelligent Offloading for Multi-Access Edge Computing: A New Actor-Critic Approach,” IEEE Int. Conf. Commun., vol. 2020-June, 2020, https://doi.org/10.1109/ICC40277.2020.9149387.
B. Zhang et al., “On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning,” Proc. Mach. Learn. Res., vol. 130, pp. 4015–4023, 2021, https://proceedings.mlr.press/v130/zhang21n.html.
F. A. Hashim and A. G. Hussien, “Snake Optimizer: A novel meta-heuristic optimization algorithm,” Knowledge-Based Syst., vol. 242, p. 108320, 2022, https://doi.org/10.1016/j.knosys.2022.108320.
J. Youn, “Intelligent Task Offloading Method using Deep Q-Network for Collaborative Edge Computing System,” 5th Int. Conf. Artif. Intell. Inf. Commun. ICAIIC 2023, pp. 872–875, 2023, https://doi.org/10.1109/ICAIIC57133.2023.10067111.
H. Zhai, X. Zhou, H. Zhang, and D. Yuan, “Delay Minimization in Hybrid Edge Computing Networks: A DDQN-Based Task Offloading Approach,” IEEE Trans. Veh. Technol., vol. PP, pp. 1–11, 2024, https://doi.org/10.1109/TVT.2024.3407483.
B. Dai, Y. Qiu, and W. Feng, “Scalable Computation Offloading for Industrial IoTs via Distributed Deep Reinforcement Learning,” Proc. 2024 27th Int. Conf. Comput. Support. Coop. Work Des. CSCWD 2024, pp. 1681–1686, 2024, https://doi.org/10.1109/CSCWD61410.2024.10580311.
G. Nieto, I. de la Iglesia, U. Lopez-Novoa, and C. Perfecto, “Deep Reinforcement Learning techniques for dynamic task offloading in the 5G edge-cloud continuum,” J. Cloud Comput., vol. 13, no. 1, 2024, https://doi.org/10.1186/s13677-024-00658-0.
Y. Chen, X. Li, J. Ye, X. Wang, and Q. Chen, “Multi-agent Reinforcement Learning Based Resource Allocation in End-Edge-Cloud Enabled Industrial Internet of Things,” Proc. - IEEE Congr. Cybermatics 2023 IEEE Int. Conf. Internet Things, iThings 2023, IEEE Green Comput. Commun. GreenCom 2023, IEEE Cyber, Phys. Soc. Comput. CPSCom 2023 IEEE Smart Data, SmartD, pp. 13–19, 2023, https://doi.org/10.1109/iThings-GreenCom-CPSCom-SmartData-Cybermatics60724.2023.00027.
S. Chouikhi, M. Esseghir, and L. Merghem-Boulahia, “Energy-Efficient Computation Offloading Based on Multiagent Deep Reinforcement Learning for Industrial Internet of Things Systems,” IEEE INTERNET THINGS J., vol. 11, no. 7, pp. 12228–12239, 2024, https://doi.org/10.1109/JIOT.2023.3333044.
L. Ai, B. Tan, J. Zhang, R. Wang, and J. Wu, “Dynamic Offloading Strategy for Delay-Sensitive Task in Mobile-Edge Computing Networks,” IEEE Internet Things J., vol. 10, no. 1, pp. 526–538, 2023, https://doi.org/10.1109/JIOT.2022.3202797.
R. Ma, X. Zhou, H. Zhang, and D. Yuan, “Joint Optimization of Energy Consumption and Latency Based on DRL: An Edge Server Activation and Task Scheduling Scheme in IIoT,” 2022 IEEE 14th Int. Conf. Wirel. Commun. Signal Process. WCSP 2022, pp. 203–208, 2022, https://doi.org/10.1109/WCSP55476.2022.10039283.
A. Xu et al., “QDRL: Queue-Aware Online DRL for Computation Offloading in Industrial Internet of Things,” IEEE Internet Things J., vol. 11, no. 5, pp. 7772–7786, 2024, https://doi.org/10.1109/JIOT.2023.3316139.
H. Mai Do, T. P. Tran, and M. Yoo, “Deep Reinforcement Learning-Based Task Offloading and Resource Allocation for Industrial IoT in MEC Federation System,” IEEE Access, vol. 11, no. July 2023, pp. 83150–83170, 2023, https://doi.org/10.1109/ACCESS.2023.3302518.
Y. Gong, H. Yao, J. Wang, M. Li, and S. Guo, “Edge Intelligence-Driven Joint Offloading and Resource Allocation for Future 6G Industrial Internet of Things,” IEEE Trans. Netw. Sci. Eng., vol. 11, no. 6, pp. 5644–5655, 2024, https://doi.org/10.1109/TNSE.2022.3141728.
Z. Chai, H. Hou, and Y. Li, “A dynamic queuing model based distributed task offloading algorithm using deep reinforcement learning in mobile edge computing,” Appl. Intell., vol. 53, no. 23, pp. 28832–28847, 2023, https://doi.org/10.1007/s10489-023-05065-7.
F. Zhang, G. Han, L. Liu, Y. Zhang, Y. Peng, and C. Li, “Cooperative Partial Task Offloading and Resource Allocation for IIoT Based on Decentralized Multiagent Deep Reinforcement Learning,” IEEE Internet Things J., vol. 11, no. 3, pp. 5526–5544, 2024, https://doi.org/10.1109/JIOT.2023.3306803.
W. J. Cheng, X. S. Liu, X. T. Wang, and G. F. Nie, “Task Offloading and Resource Allocation for Industrial Internet of Things: A Double-Dueling Deep Q-Network Approach,” IEEE ACCESS, vol. 10, pp. 103111–103120, 2022, https://doi.org/10.1109/ACCESS.2022.3210248.
S. Chen, J. Chen, Y. Miao, Q. Wang, and C. Zhao, “Deep Reinforcement Learning-Based Cloud-Edge Collaborative Mobile Computation Offloading in Industrial Networks,” IEEE Trans. Signal Inf. Process. over Networks, vol. 8, pp. 364–375, 2022, https://doi.org/10.1109/TSIPN.2022.3171336.
G. Wu, Z. Xu, H. Zhang, S. Shen, and S. Yu, “Multi-agent DRL for joint completion delay and energy consumption with queuing theory in MEC-based IIoT,” J. Parallel Distrib. Comput., vol. 176, pp. 80–94, 2023, https://doi.org/10.1016/j.jpdc.2023.02.008.
T. Eimer, M. Lindauer, and R. Raileanu, “Hyperparameters in Reinforcement Learning and How To Tune Them,” Proc. Mach. Learn. Res., vol. 202, pp. 9104–9149, 2023, https://proceedings.mlr.press/v202/eimer23a.
J. Parker-Holder et al., “Automated Reinforcement Learning (AutoRL): A Survey and Open Problems,” J. Artif. Intell. Res., vol. 74, pp. 517–568, 2022, https://doi.org/10.1613/jair.1.13596.
B. Bischl et al., “Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges,” Wiley Interdiscip. Rev. Data Min. Knowl. Discov., vol. 13, no. 2, pp. 1–43, 2023, https://doi.org/10.1002/widm.1484.
N. Awad, N. Mallik, and F. Hutter, “DEHB: Evolutionary Hyberband for Scalable, Robust and Efficient Hyperparameter Optimization,” IJCAI Int. Jt. Conf. Artif. Intell., no. 3, pp. 2147–2153, 2021, https://doi.org/10.24963/ijcai.2021/296.
J. Parker-Holder, V. Nguyen, and S. J. Roberts, “Provably efficient online hyperparameter optimization with population-based bandits,” Adv. Neural Inf. Process. Syst., vol. 33, pp. 17200-17211, 2020, https://proceedings.neurips.cc/paper/2020/hash/c7af0926b294e47e52e46cfebe173f20-Abstract.html.
T. Yu and H. Zhu, “Hyper-Parameter Optimization: A Review of Algorithms and Applications,” arXiv preprint arXiv:2003.05689, 2020, https://doi.org/10.48550/arXiv.2003.05689.
Y. Qin, J. Chen, L. Jin, R. Yao, and Z. Gong, “Task offloading optimization in mobile edge computing based on a deep reinforcement learning algorithm using density clustering and ensemble learning,” Sci. Rep., vol. 15, no. 1, pp. 1–25, 2025, https://doi.org/10.1038/s41598-024-84038-3.
H. Ke, J. Wang, L. Deng, Y. Ge, and H. Wang, “Deep Reinforcement Learning-Based Adaptive Computation Offloading for MEC in Heterogeneous Vehicular Networks,” IEEE Trans. Veh. Technol., vol. 69, no. 7, pp. 7916–7929, 2020, https://doi.org/10.1109/TVT.2020.2993849.
Y. He, J. Ren, G. Yu, and Y. Cai, “D2D communications meet mobile edge computing for enhanced computation capacity in cellular networks,” IEEE Trans. Wirel. Commun., vol. 18, no. 3, pp. 1750–1763, 2019, https://doi.org/10.1109/TWC.2019.2896999.
X. Chen, H. Zhang, C. Wu, S. Mao, Y. Ji, and M. Bennis, “Optimized computation offloading performance in virtual edge computing systems via deep reinforcement learning,” IEEE Internet Things J., vol. 6, no. 3, pp. 4005–4018, 2019, https://doi.org/10.1109/JIOT.2018.2876279.
A. Hazra and T. Amgoth, “CeCO: Cost-Efficient Computation Offloading of IoT Applications in Green Industrial Fog Networks,” IEEE Trans. Ind. INFORMATICS, vol. 18, no. 9, pp. 6255–6263, 2022, https://doi.org/10.1109/TII.2021.3130255.
J. Peng, H. Qiu, J. Cai, W. Xu, and J. Wang, “D2D-Assisted Multi-User Cooperative Partial Offloading, Transmission Scheduling and Computation Allocating for MEC,” IEEE Trans. Wirel. Commun., vol. 20, no. 8, pp. 4858–4873, 2021, https://doi.org/10.1109/TWC.2021.3062616.
X. Jiang, F. R. Yu, T. Song, and V. C. M. Leung, “A Survey on Multi-Access Edge Computing Applied to Video Streaming: Some Research Issues and Challenges,” IEEE Commun. Surv. Tutorials, vol. 23, no. 2, pp. 871–903, 2021, https://doi.org/10.1109/COMST.2021.3065237.
R. Chai, J. Lin, M. Chen, and Q. Chen, “Task execution cost minimization-based joint computation offloading and resource allocation for cellular D2D MEC systems,” IEEE Syst. J., vol. 13, no. 4, pp. 4110–4121, 2019, https://doi.org/10.1109/JSYST.2019.2921115.
M. E. Yüksel, “Power consumption analysis of a Wi-Fi-based IoT device,” Electrica, vol. 20, no. 1, pp. 62–70, 2020, https://doi.org/10.5152/ELECTRICA.2020.19081.
L. P. Qian, Y. Wu, F. L. Jiang, N. N. Yu, W. D. Lu, and B. Lin, “NOMA Assisted Multi-Task Multi-Access Mobile Edge Computing via Deep Reinforcement Learning for Industrial Internet of Things,” IEEE Trans. Ind. INFORMATICS, vol. 17, no. 8, pp. 5688–5698, 2021, https://doi.org/10.1109/TII.2020.3001355.
X. Jin, S. Zhang, Y. Ding, and Z. Wang, “Task offloading for multi-server edge computing in industrial Internet with joint load balance and fuzzy security,” Sci. Rep., vol. 14, no. 1, pp. 1–26, 2024, https://doi.org/10.1038/s41598-024-79464-2.
Z. Aghapour, S. Sharifian, and H. Taheri, “Task offloading and resource allocation algorithm based on deep reinforcement learning for distributed AI execution tasks in IoT edge computing environments,” Comput. Networks, vol. 223, no. January, p. 109577, 2023, https://doi.org/10.1016/j.comnet.2023.109577.
J. Chi, X. Zhou, F. Xiao, Y. Lim, and T. Qiu, “Task Offloading via Prioritized Experience-Based Double Dueling DQN in Edge-Assisted IIoT,” IEEE Trans. Mob. Comput., vol. 23, no. 12, pp. 14575–14591, 2024, https://doi.org/10.1109/TMC.2024.3452502.
S. M. Shithil and M. A. Adnan, “A Prediction Based Replica Selection Strategy for Reducing Tail Latency in Geo-Distributed Systems,” IEEE Trans. Cloud Comput., vol. 11, no. 3, pp. 2954–2965, 2023, https://doi.org/10.1109/TCC.2023.3244203.
Z. Zhou, L. Pan, and S. Liu, “Online Traffic Allocation for Video Service Providers in Cloud-Edge Cooperative Systems,” IEEE Trans. Serv. Comput., vol. 18, no. 3, pp. 1618–1626, 2025, https://doi.org/10.1109/TSC.2025.3565363.
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Hayder Salman Dawood, Ismail Fauzi Bin Isnin, Muhalim bin Mohamed Amin, Ahmad Najmi bin Amerhaider Nuar
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
This journal is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
