A dai-liao hybrid hestenes-stiefel and fletcher-revees methods for unconstrained optimization
DOI:
https://doi.org/10.12928/ijio.v2i1.3054Keywords:
Unconstrained optimization, Descent property, Conjugacy condition, Global convergence, Conjugate gradient algorithmAbstract
Some problems have no analytical solution or too difficult to solve by scientists, engineers, and mathematicians, so the development of numerical methods to obtain approximate solutions became necessary. Gradient methods are more efficient when the function to be minimized continuously in its first derivative. Therefore, this article presents a new hybrid Conjugate Gradient (CG) method to solve unconstrained optimization problems. The method requires the first-order derivatives but overcomes the steepest descent method’s shortcoming of slow convergence and needs not to save or compute the second-order derivatives needed by the Newton method. The CG update parameter is suggested from the Dai-Liao conjugacy condition as a convex combination of Hestenes-Stiefel and Fletcher-Revees algorithms by employing an optimal modulating choice parameterto avoid matrix storage. Numerical computation adopts an inexact line search to obtain the step-size that generates a decent property, showing that the algorithm is robust and efficient. The scheme converges globally under Wolfe line search, and it’s like is suitable in compressive sensing problems and M-tensor systems.
References
Abubakar, A. B., Kumam, P., Mohammad, H., Awwal., A. M. & Sitthithakerngkiet, K.(2019). A modified fletcher–reeves conjugate gradient method for monotone nonlinear equations with some applications, MPDI, mathematics, 7, 745 doi:10.3390/math7080745.
Al-Baali, M. (1985). Descent property and global convergence of the Fletcher Reeves method with inexact line search. IMA Journal of Numerical Analysis, 5, 121-124.
Al-Namat, F.N. and Al-Naemi, G.M. (2020). Global convergence property with inexact line search for a new hybrid conjugate gradient method. Open Access Library Journal, 7: e6048. https://doi.org/10.4236/oalib.1106048
Andrei, N. (2008). An unconstrained optimization test function. Advanced Modeling and Optimization. An Electronic International Journal,10, 172-182.
Andrei, N. (2017). A Dai-Liao nonlinear conjugate gradient algorithm. Numer. Algor., DOI 10.1007/s11075-017-0362-5.
Babaie-Kafaki, S. (2011). A modified BFGS algorithm based on a hybrid secant equation. Science China Mathematics,58, 315-331.
Babaie-Kafaki, S. (2013). A hybrid conjugate gradient method based on quadratic relaxation of Dai-Yuan hybrid conjugate gradient parameter. Journal of Mathematical Programming and Operation Research, 62(7), 929-941.
Babaie-Kafaki, S. & Mahdavi-Amiri, N. (2013). Two hybrid conjugate gradient methods based on hybrid secant equation. Mathematical Modeling and Analysis,18(1):32-52.
Babaie-Kafaki, S. & Ghanbari, R. (2014c). Two hybrid nonlinear conjugate gradient methods based on a modified secant equation. Journal of Mathematical Programming and Operation Research,63(7), 1027-1042.
Babaie-Kafaki, S., & Ghanbari, R. (2014a). The Dai-Liao nonlinear conjugate gradient method with optimal parameter choices. Eur. J. Oper. Res. 234(3), 625-630.
Babaie-Kafaki, S., & Ghanbari, R. (2014b). A descent family of Dai-Liao conjugate gradient methods. Optim. Methods Softw., 29(3): 583-591.
Babaie-Kafaki, S., & Ghanbari, R. (2015). Two optimal Dai-Liao conjugate gradient methods. Optimization, 64(1): 2277-2287.
Babaie-Kafaki, S. (2015). On optimality of two adaptive choices for the parameter of Dai-Liao method. Optim. Lett., DOI 10.1007/s11590-015-0965-5.
Babaie-Kafaki, S., & Ghanbari, R. (2017). Two adaptive Dai-Liao nonlinear conjugate gradient methods. Iran J. Sci. Technol. Trans. Sci., DOI 10.1007/s40995-017- 027 1-4.
Dai, Y.H. & Yuan, Y. (1999). A nonlinear conjugate gradient method with a strong global convergence property. SIAM J. Optim.,10, 177-182.
Dai, Y.H & Liao, L.Z. (2001). New conjugacy conditions and related nonlinear conjugate gradient methods, Appl. Math. Optim., 43, 87-101.
Dai, Y.H. & Yuan, Y. (2001). An efficient hybrid conjugates gradient method for unconstrained optimization. Annals of Operations Research, 103, 33-47.
Djordjevic, S.S. (2016). New hybrid conjugate gradient method as a convex combination of FR and PRP methods. Published by faculty of science and mathematics. University of Nis, Serbia, Filomat, 31, 3083-3100. https://doi.org/10.2298/FIL16083D
Djordjevic, S.S. (2017). New hybrid conjugate gradient methods as a convex combination of LS and CD methods. Published by faculty of science and mathematics. University of Nis, Serbia,31(6), 1813-1825.
Djordjevic, S.S. (2018). New hybrid conjugate gradient method as a convex combination of HS and FR conjugate gradient methods. Journal of Applied Mathematics and Computation, 2, 366-378. https://doi.org/10.26855/jamc.2018.09.002
Djordjevic, S.S. (2019). New hybrid conjugate gradient method as a convex combination of LS and FR conjugate gradient methods. Acta Mathematica Scientia, 39, 214-228.
Dolan, E.D. & More´, J.J. (2002). Benchmarking optimization software with performance profiles. Journal of Math. Program, 91(2), 201-213.
Esmaeili H., Rostami M., & Kimiaei M. (2018). Extended Dai–Yuan conjugate gradient strategy for large-scale unconstrained optimization with applications to compressive sensing. Published by faculty of sciences and mathematics, University of Nis, Serbia, Filomat, 32(6), 2173–2191. Available at: http://www.pmf.ni.ac.rs/filomat
Fletcher, R., & Reeves, C. (1964). Function minimization by conjugate gradients. Computational Journal,7, 149-154.
Fletcher, R. (1987). Practical methods of optimization, (vol. 1), Unconstrained optimization. New York: John Wiley & Sons.
Gould, N.I.M., Orban, D. & Toint, P.L. (2003). CUTEr: a constrained and unconstrained testing environment, revisited. ACM Transactions on Mathematical Software,29(4), 373-394.
Guo, J., & Wan, Z. (2019). A modified spectral PRP conjugate gradient projection method for solving large-scale monotone equations and its application in compressed sensing. Hindawi Mathematical Problems in Engineering. Volume 2019, Article ID 5261830, 17 pages. https://doi.org/10.1155/2019/5261830
Hager, W.W. & Zhang, H. (2006). A survey of nonlinear conjugate gradient methods. Pacific J. Optim., 2, 35-58.
Hestenes, M.R., & Stiefel, E.L. (1952). Methods of conjugate gradients for solving linear systems. Journal Res. Nat. Bur. Stand., 49, 409-436
Ibrahim, A.H., Kumam, P., Abubakar, A. B., Jirakitpuwapat, W., & Abubakar, J. (2020). A hybrid conjugate gradient algorithm for constrained monotone equations with application in compressive sensing. Heliyon.e03466. https://doi.org/10.1016/j.heliyon.2020.e03466
Li, D.H., & Fukushima, A. M. (2001). modified BFGS method and its global convergence in nonconvex minimization, J. Comput. Appl. Math. 129(1) 15-35.
Liu, K., & Du, S. (2019). Modified three-term conjugate gradient method and its applications. Hindawi Mathematical Problems in Engineering. Volume 2019, Article ID 5976595, 9 pages. https://doi.org/10.1155/2019/5976595
Liu, J., Du, S., & Chen, Y. (2020). A sufficient descent nonlinear conjugate gradient method for solving M-tensor equations. Journal of Computational and Applied Mathematics, 371112709
Liu, Y., & Storey, C. (1991). Efficient generalized conjugate gradient algorithms, part 1: Theory. Journal of optimization theory and applications. 69, 129-137.
Lotfi, M., & Hosseini, S.M. (2019). An efficient Dai–Liao type conjugate gradient method by reformulating the CG parameter in the search direction equation. Journal of Computational and Applied Mathematics, 371. 112708. https://doi.org/10.1016/j.cam.2019.112708
Mohammed, N. S., Mustafa M., Mohd R. & Shazlyn M. S. (2020). A new hybrid coefficient of conjugate gradient method. Indonesian Journal of Electrical Engineering and Computer Science, Vol. 18, No. 3, June 2020, pp. 1454-1463.
Perry, A. (1978). A modified conjugate gradient algorithm. Oper. Res. Tech. Notes,26(6): 1073-1078.
Polak, B.T. (1969). The conjugate gradient method in extreme problems. USSR Comput. Math. Math. Phys. 4, 94-112.
Polyak, B.T. (1967). A general method of solving extremal problems. Soviet Math. Doklady, 8, 14-29.
Powell, M.J.D. (1984). Nonconvex minimization calculations and the conjugate gradient method, Numerical Analysis (Dundee, 1983), D.F. Griffiths, ed., Lecture Notes in Mathematics, Vol. 1066, Springer, Berlin. 122-141.
Salihu, N., Odekunle, M., Waziri, M. & Halilu, A. (2020). A new hybrid conjugate gradient method based on secant equation for solving large scale unconstrained optimization problems. Iranian Journal of Optimization, 12(1): 33-44.
Waziri, M. Y., Ahmed, K., & Sabi’u, J. (2019). A Dai-Liao conjugate gradient method via modified secant equation for system of nonlinear equations. Arab. J. Math. https://doi.org/10.1007/s40065-019-0264-6.
Xue, W., Ren, J., Zheng, X., Liu, Z., & Liang, Y. (2018). A New DY conjugate gradient method and applications to image denoising. IEICE TRANS. INF. & SYST., VOL.E101–D, NO.1 2, December 2018. DOI: 10.1587/transinf.2018EDP7210
Yao, S., Feng, Q., Li, L., & Xu, J. (2019). A class of globally convergent three-term Dai-Liao conjugate gradient methods. Applied Numerical Mathematics, 151 354-366. https://doi.org/10.1016/j.apnum.2019.12.026.
Zoutendijk, G. (1970). Nonlinear programming computational methods, in integer and nonlinear programming. J. Abadie ed., North-Holland, Amsterdam 37-86.
Al-Baali, M. (1985). Descent property and global convergence of the Fletcher Reeves method with inexact line search. IMA Journal of Numerical Analysis, 5, 121-124.
Al-Namat, F.N. and Al-Naemi, G.M. (2020). Global convergence property with inexact line search for a new hybrid conjugate gradient method. Open Access Library Journal, 7: e6048. https://doi.org/10.4236/oalib.1106048
Andrei, N. (2008). An unconstrained optimization test function. Advanced Modeling and Optimization. An Electronic International Journal,10, 172-182.
Andrei, N. (2017). A Dai-Liao nonlinear conjugate gradient algorithm. Numer. Algor., DOI 10.1007/s11075-017-0362-5.
Babaie-Kafaki, S. (2011). A modified BFGS algorithm based on a hybrid secant equation. Science China Mathematics,58, 315-331.
Babaie-Kafaki, S. (2013). A hybrid conjugate gradient method based on quadratic relaxation of Dai-Yuan hybrid conjugate gradient parameter. Journal of Mathematical Programming and Operation Research, 62(7), 929-941.
Babaie-Kafaki, S. & Mahdavi-Amiri, N. (2013). Two hybrid conjugate gradient methods based on hybrid secant equation. Mathematical Modeling and Analysis,18(1):32-52.
Babaie-Kafaki, S. & Ghanbari, R. (2014c). Two hybrid nonlinear conjugate gradient methods based on a modified secant equation. Journal of Mathematical Programming and Operation Research,63(7), 1027-1042.
Babaie-Kafaki, S., & Ghanbari, R. (2014a). The Dai-Liao nonlinear conjugate gradient method with optimal parameter choices. Eur. J. Oper. Res. 234(3), 625-630.
Babaie-Kafaki, S., & Ghanbari, R. (2014b). A descent family of Dai-Liao conjugate gradient methods. Optim. Methods Softw., 29(3): 583-591.
Babaie-Kafaki, S., & Ghanbari, R. (2015). Two optimal Dai-Liao conjugate gradient methods. Optimization, 64(1): 2277-2287.
Babaie-Kafaki, S. (2015). On optimality of two adaptive choices for the parameter of Dai-Liao method. Optim. Lett., DOI 10.1007/s11590-015-0965-5.
Babaie-Kafaki, S., & Ghanbari, R. (2017). Two adaptive Dai-Liao nonlinear conjugate gradient methods. Iran J. Sci. Technol. Trans. Sci., DOI 10.1007/s40995-017- 027 1-4.
Dai, Y.H. & Yuan, Y. (1999). A nonlinear conjugate gradient method with a strong global convergence property. SIAM J. Optim.,10, 177-182.
Dai, Y.H & Liao, L.Z. (2001). New conjugacy conditions and related nonlinear conjugate gradient methods, Appl. Math. Optim., 43, 87-101.
Dai, Y.H. & Yuan, Y. (2001). An efficient hybrid conjugates gradient method for unconstrained optimization. Annals of Operations Research, 103, 33-47.
Djordjevic, S.S. (2016). New hybrid conjugate gradient method as a convex combination of FR and PRP methods. Published by faculty of science and mathematics. University of Nis, Serbia, Filomat, 31, 3083-3100. https://doi.org/10.2298/FIL16083D
Djordjevic, S.S. (2017). New hybrid conjugate gradient methods as a convex combination of LS and CD methods. Published by faculty of science and mathematics. University of Nis, Serbia,31(6), 1813-1825.
Djordjevic, S.S. (2018). New hybrid conjugate gradient method as a convex combination of HS and FR conjugate gradient methods. Journal of Applied Mathematics and Computation, 2, 366-378. https://doi.org/10.26855/jamc.2018.09.002
Djordjevic, S.S. (2019). New hybrid conjugate gradient method as a convex combination of LS and FR conjugate gradient methods. Acta Mathematica Scientia, 39, 214-228.
Dolan, E.D. & More´, J.J. (2002). Benchmarking optimization software with performance profiles. Journal of Math. Program, 91(2), 201-213.
Esmaeili H., Rostami M., & Kimiaei M. (2018). Extended Dai–Yuan conjugate gradient strategy for large-scale unconstrained optimization with applications to compressive sensing. Published by faculty of sciences and mathematics, University of Nis, Serbia, Filomat, 32(6), 2173–2191. Available at: http://www.pmf.ni.ac.rs/filomat
Fletcher, R., & Reeves, C. (1964). Function minimization by conjugate gradients. Computational Journal,7, 149-154.
Fletcher, R. (1987). Practical methods of optimization, (vol. 1), Unconstrained optimization. New York: John Wiley & Sons.
Gould, N.I.M., Orban, D. & Toint, P.L. (2003). CUTEr: a constrained and unconstrained testing environment, revisited. ACM Transactions on Mathematical Software,29(4), 373-394.
Guo, J., & Wan, Z. (2019). A modified spectral PRP conjugate gradient projection method for solving large-scale monotone equations and its application in compressed sensing. Hindawi Mathematical Problems in Engineering. Volume 2019, Article ID 5261830, 17 pages. https://doi.org/10.1155/2019/5261830
Hager, W.W. & Zhang, H. (2006). A survey of nonlinear conjugate gradient methods. Pacific J. Optim., 2, 35-58.
Hestenes, M.R., & Stiefel, E.L. (1952). Methods of conjugate gradients for solving linear systems. Journal Res. Nat. Bur. Stand., 49, 409-436
Ibrahim, A.H., Kumam, P., Abubakar, A. B., Jirakitpuwapat, W., & Abubakar, J. (2020). A hybrid conjugate gradient algorithm for constrained monotone equations with application in compressive sensing. Heliyon.e03466. https://doi.org/10.1016/j.heliyon.2020.e03466
Li, D.H., & Fukushima, A. M. (2001). modified BFGS method and its global convergence in nonconvex minimization, J. Comput. Appl. Math. 129(1) 15-35.
Liu, K., & Du, S. (2019). Modified three-term conjugate gradient method and its applications. Hindawi Mathematical Problems in Engineering. Volume 2019, Article ID 5976595, 9 pages. https://doi.org/10.1155/2019/5976595
Liu, J., Du, S., & Chen, Y. (2020). A sufficient descent nonlinear conjugate gradient method for solving M-tensor equations. Journal of Computational and Applied Mathematics, 371112709
Liu, Y., & Storey, C. (1991). Efficient generalized conjugate gradient algorithms, part 1: Theory. Journal of optimization theory and applications. 69, 129-137.
Lotfi, M., & Hosseini, S.M. (2019). An efficient Dai–Liao type conjugate gradient method by reformulating the CG parameter in the search direction equation. Journal of Computational and Applied Mathematics, 371. 112708. https://doi.org/10.1016/j.cam.2019.112708
Mohammed, N. S., Mustafa M., Mohd R. & Shazlyn M. S. (2020). A new hybrid coefficient of conjugate gradient method. Indonesian Journal of Electrical Engineering and Computer Science, Vol. 18, No. 3, June 2020, pp. 1454-1463.
Perry, A. (1978). A modified conjugate gradient algorithm. Oper. Res. Tech. Notes,26(6): 1073-1078.
Polak, B.T. (1969). The conjugate gradient method in extreme problems. USSR Comput. Math. Math. Phys. 4, 94-112.
Polyak, B.T. (1967). A general method of solving extremal problems. Soviet Math. Doklady, 8, 14-29.
Powell, M.J.D. (1984). Nonconvex minimization calculations and the conjugate gradient method, Numerical Analysis (Dundee, 1983), D.F. Griffiths, ed., Lecture Notes in Mathematics, Vol. 1066, Springer, Berlin. 122-141.
Salihu, N., Odekunle, M., Waziri, M. & Halilu, A. (2020). A new hybrid conjugate gradient method based on secant equation for solving large scale unconstrained optimization problems. Iranian Journal of Optimization, 12(1): 33-44.
Waziri, M. Y., Ahmed, K., & Sabi’u, J. (2019). A Dai-Liao conjugate gradient method via modified secant equation for system of nonlinear equations. Arab. J. Math. https://doi.org/10.1007/s40065-019-0264-6.
Xue, W., Ren, J., Zheng, X., Liu, Z., & Liang, Y. (2018). A New DY conjugate gradient method and applications to image denoising. IEICE TRANS. INF. & SYST., VOL.E101–D, NO.1 2, December 2018. DOI: 10.1587/transinf.2018EDP7210
Yao, S., Feng, Q., Li, L., & Xu, J. (2019). A class of globally convergent three-term Dai-Liao conjugate gradient methods. Applied Numerical Mathematics, 151 354-366. https://doi.org/10.1016/j.apnum.2019.12.026.
Zoutendijk, G. (1970). Nonlinear programming computational methods, in integer and nonlinear programming. J. Abadie ed., North-Holland, Amsterdam 37-86.
Downloads
Published
2021-02-24
How to Cite
Salihu, N., Odekunle, M. R., Saleh, A. M., & Salihu, S. (2021). A dai-liao hybrid hestenes-stiefel and fletcher-revees methods for unconstrained optimization. International Journal of Industrial Optimization, 2(1), 33–50. https://doi.org/10.12928/ijio.v2i1.3054
Issue
Section
Articles
License
Copyright (c) 2021 Nasiru Salihu, Mathew Remilekun Odekunle, Also Mohammed Saleh, Suraj Salihu
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
License and Copyright Agreement
In submitting the manuscript to the journal, the authors certify that:
- They are authorized by their co-authors to enter into these arrangements.
- The work described has not been formally published before, except in the form of an abstract or as part of a published lecture, review, thesis, or overlay journal. Please also carefully read the International Journal of Industrial Optimization (IJIO) Author Guidelines at http://journal2.uad.ac.id/index.php/ijio/about/submissions#onlineSubmissions
- That it is not under consideration for publication elsewhere,
- That its publication has been approved by all the author(s) and by the responsible authorities tacitly or explicitly of the institutes where the work has been carried out.
- They secure the right to reproduce any material that has already been published or copyrighted elsewhere.
- They agree to the following license and copyright agreement.
Copyright
Authors who publish with the International Journal of Industrial Optimization (IJIO) agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License (CC BY-SA 4.0) that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
1.png)
