Reinforcement Learning for Portfolio Optimization: Evidence from the Indonesian Stock Market

Rachmawaty; Rahmawati; Hartini; Andi Aris Mattunruang

doi:10.12928/jreksa.v13i1.14579

Authors

Rachmawaty Universitas Patompo, Makassar, Indonesia
Rahmawati Universitas Patompo, Makassar, Indonesia
Hartini Universitas Patompo, Makassar, Indonesia
Andi Aris Mattunruang Universitas Patompo, Makassar, Indonesia

DOI:

https://doi.org/10.12928/jreksa.v13i1.14579

Keywords:

Reinforcement learning, portfolio optimization, Indonesia Stock Exchange (IDX), Deep Q-Network (DQN), investment strategy

Abstract

Stock portfolio management in emerging markets such as Indonesia remains challenging due to high volatility, market inefficiencies, and the strong presence of retail investors. In this setting, conventional approaches, including buy-and-hold strategies, the Markowitz framework, and the Capital Asset Pricing Model (CAPM), often struggle to perform consistently under rapidly changing market conditions. While reinforcement learning (RL) has gained increasing traction in global finance, its application in the Indonesian stock market remains limited. This study examines the effectiveness of an RL-based approach, specifically the Deep Q-Network (DQN) algorithm, in optimizing stock portfolios on the Indonesia Stock Exchange (IDX). Using a quantitative experimental design, the analysis is based on back-testing simulations of IDX30 stocks over the 2022–2024 period, with samples selected purposively based on liquidity and market capitalization. The findings show that the DQN-based strategy consistently outperforms conventional methods, delivering higher returns, improved risk–return efficiency, and better control of downside risk. These results suggest that RL models are better suited to adapt to dynamic market conditions. Theoretically, this study extends portfolio optimization literature by incorporating adaptive, learning-based models into emerging market contexts. Practically, it offers evidence for investors and practitioners to consider AI-driven strategies as a more responsive alternative to traditional approaches in a volatile market.

References

Abhirama, M. A. (2025). Optimasi portofolio saham berbasis deep reinforcement learning: Studi algoritma deep Q-network (Tesis Magister). Institut Teknologi Bandung.

Acero, F., Zehtabi, P., Marchesotti, N., Cashmore, M., Magazzeni, D., & Veloso, M. (2024). Deep reinforcement learning and mean-variance strategies for responsible portfolio optimization. arXiv. http://arxiv.org/abs/2403.16667

Baradja, A., & Tjendrowasono, T. I. (2024). Pengaplikasian deep reinforcement Q-learning untuk prediksi perdagangan valas otomatis. Jurnal Rekayasa Sistem Informasi Dan Teknologi, 1(3), 190–198. https://doi.org/10.59407/jrsit.v1i3.519

Dong, Z., Huang, S., Ma, S., & Qian, Y. (2021). Factor representation and decision making in stock markets using deep reinforcement learning. arXiv. http://arxiv.org/abs/2108.01758

Gao, Z., Gao, Y., Hu, Y., Jiang, Z., & Su, J. (2020). Application of Deep Q-Network in portfolio management. International Conference on Big Data Analytics, ICBDA 2020, 268–275. https://doi.org/10.1109/ICBDA49040.2020.9101333

Green, J., & Zhao, W. (2022). Forecasting earnings and returns: a review of recent advancements. Journal of Finance and Data Science, 8, 120–137. https://doi.org/10.1016/j.jfds.2022.04.004

Gutiérrez, Ó. (2020). On the definition of the investment-uncertainty relationship. Journal of Economics and Business, 112, 105934. https://doi.org/10.1016/j.jeconbus.2020.105934

Jang, J., & Seong, N. (2023). Deep reinforcement learning for stock portfolio optimization by connecting with modern portfolio theory. Expert System with Applications, 218, 119556. https://doi.org/10.1016/j.eswa.2023.119556

Jiang, Z., Xu, D., & Liang, J. (2017). A deep reinforcement learning framework for the financial portfolio management problem. 1–31. http://arxiv.org/abs/1706.10059

Ju, C., & Zhu, Y. (2024). Reinforcement learning-based model for enterprise financial asset risk assessment and intelligent decision-making. Applied and Computational Engineering, 97(1), 181–186. https://doi.org/10.54254/2755-2721/97/20241365

Kadir, R., Alisyahbana, A. N. Q., Amrullah, N., Syakur, R. M., & Amreani. (2024). How digital payment and online marketing strategies affect consumer experience in the culinary industry? Indonesian Journal of Business and Entrepreneurship Research, 2(2), 76–85. https://doi.org/10.62794/ijober.v2i2.2493

Lin, Y.-C., Chen, C.-T., Sang, C.-Y., & Huang, S.-H. (2022). Multiagent-based deep reinforcement learning for risk-shifting portfolio management. Applied Soft Computing, 123. https://doi.org/10.1016/j.asoc.2022.108894

Liu, W., Gu, Y., & Ge, Y. (2024). Multi-factor stock trading strategy based on DQN with multi-BiGRU and multi-head ProbSparse self-attention. Applied Intelligence, 54(7), 5417–5440. https://doi.org/10.1007/s10489-024-05463-5

Ngo, V. M., Nguyen, H. H., & Nguyen, P. Van. (2023). Does reinforcement learning outperform deep learning and traditional portfolio optimization models in frontier and developed financial markets? Research in International Business and Finance, 65. https://doi.org/10.1016/j.ribaf.2023.101936

Niu, H., Li, S., & Li, J. (2022). MetaTrader: A reinforcement learning approach integrating diverse policies for portfolio optimization. In International Conference on Information and Knowledge Management, Proceedings (Vol. 1, Issue 1). Association for Computing Machinery. https://doi.org/10.1145/3511808.3557363

Nurmawati, E., Abaysa, R., & Putra, R. A. (2025). Optimalisasi portofolio saham syariah berbasis prediksi menggunakan long short-term memory (LSTM). Jurnal Informatika: Jurnal Pengembangan IT, 10(2), 464–475. https://doi.org/10.30591/jpit.v9ix.xxx

Orăștean, R., & Mărginean, S. C. (2023). Renminbi internationalization process: A quantitative literature review. International Journal of Financial Studies, 11(1), 1–25. https://doi.org/10.3390/ijfs11010015

Purnomo, D. T. (2025). Optimisasi portofolio di Bursa Efek Indonesia: analisis VAR untuk investasi konvensional dan syariah. Jurnal Riset Ekonomi Dan Bisnis, 18(1), 63–76. https://doi.org/10.26623/jreb.v18i1.12114

Putri, M. Z., Wulandari, D., Aristawati, P. A., & Apridasari, E. (2025). Tantangan dan peluang pasar modal Indonesia dalam meningkatkan minat investasi di era digital. Jurnal Ekonomi Dan Manajemen, 2(2), 3546–3562. https://doi.org/10.57141/kompeten.v3i1.133

Putri, V. A., & Mandayanti, E. (2021). Perspektif perkembangan dan tantangan pasar modal di Indonesia. Jurnal Pendidikan Tambusai, 5(3), 10904–10908.

Setiawan, E. P., & Rosadi, D. (2019). Model pengoptimuman portofolio mean-variance dan perkembangan praktisnya. Jurnal Optimasi Sistem Industri, 18(1), 25–36. https://doi.org/10.25077/josi.v18.n1.p25-36.2019

Takara, L. de A., Santos, A. A. P., Mariani, V. C., & Coelho, L. dos S. (2024). Deep reinforcement learning applied to a sparse-reward trading environment with intraday data. Expert System with Applications.

Yang, H., Liu, X. Y., Zhong, S., & Walid, A. (2020). Deep reinforcement learning for automated stock trading: An ensemble strategy. ICAIF 2020 - 1st ACM International Conference on AI in Finance. https://doi.org/10.1145/3383455.3422540

Zhao, T., Ma, X., Li, X., & Zhang, C. (2023). Asset correlation based deep reinforcement learning for the portfolio selection. Expert System with Applications, 221.