A Dai-Liao Hybrid Hestenes-Stiefel and Fletcher-Revees Methods for Unconstrained Optimization

Some problems have no analytical solution or too difficult to solve by scientists, engineers, and mathematicians, so the development of numerical methods to obtain approximate solutions became necessary. Gradient methods are more efficient when the function to be minimized continuously in its first derivative. Therefore, this article presents a new hybrid Conjugate Gradient (CG) method to solve unconstrained optimization problems. The method requires the first-order derivatives but overcomes the steepest descent method’s shortcoming of slow convergence and needs not to save or compute the second-order derivatives needed by the Newton method. The CG update parameter is suggested from the Dai-Liao conjugacy condition as a convex combination of Hestenes-Stiefel and Fletcher-Revees algorithms by employing an optimal modulating choice parameterto avoid matrix storage. Numerical computation adopts an inexact line search to obtain the step-size that generates a decent property, showing that the algorithm is robust and efficient. The scheme converges globally under Wolfe line search, and it’s like is suitable in compressive sensing problems and M-tensor systems.


Introduction
Conjugate Gradient (CG) method was initially proposed for solving linear systems and unconstrained minimization. The method is an excellent choice for solving optimization problems by scientists, engineers, and mathematicians. The method has the following form ( ), (1) ∈ ℝ where : ℝ → ℝ is a smooth nonlinear function. The iterative scheme of the method is computed by 0 ∈ ℝ (2) +1 = + ,
Some problems have no analytical solution or too difficult to solve by scientists, engineers, and mathematicians, so the development of numerical methods to obtain approximate solutions became necessary. Gradient methods are more efficient when the function to be minimized continuously in its first derivative. Therefore, this article presents a new hybrid Conjugate Gradient (CG) method to solve unconstrained optimization problems. The method requires the first-order derivatives but overcomes the steepest descent method's shortcoming of slow convergence and needs not to save or compute the second-order derivatives needed by the Newton method. The CG update parameter is suggested from the Dai-Liao conjugacy condition as a convex combination of Hestenes-Stiefel and Fletcher-Revees algorithms by employing an optimal modulating choice parameter to avoid matrix storage. Numerical computation adopts an inexact line search to obtain the step-size that generates a decent property, showing that the algorithm is robust and efficient. The scheme converges globally under Wolfe line search, and it's like is suitable in compressive sensing problems and M-tensor systems.
in which > 0is a step length obtained by a suitable exact or inexact line search. However, is usually generated by an inexact line search, such as the standard Wolfe line search Or using strong Wolfe condition, which consists of (3) and where 0 < < < 1 and is the search direction given by where is a scalar called CG (update) parameter (Babaie -Kafaki, 2011). Liu & Du (2019) proposed a CG method by transforming -tensor system to general unconstrained minimization problem and solving a kind of nonsmooth optimization problems with 1norm, the given numerical experiments show the efficiency of the suggested method due to simplicity, low storage and nice convergence properties of the CG methods. Esmaeili, Rostami & Kimiaei (2018) employed a new CG method to solve compressive sensing problems that play an important role in medical and astronomical imaging, file restoration, image, video coding, etc applications. The method is characterized by 0minimization that is NP-hard in general. Hence, replaced the 0 -norm by the closest convex norm, which is the 1 -norm leads to the minimization problem. Similarly, Guo & Wan (2019) developed a CG algorithm to solve an engineering problem originated from compressed sensing of sparse signals. Numerical tests and preliminary application in recovering sparse signals indicate that the established algorithm outperforms similar algorithms in the literature, especially for solving largescale problems and singular ones. It was shown that the compressed sensing of sparse signals does not involve computing the Jacobian matrix or its approximation, both information storage and computational cost of the algorithm are lower. Recently, Liu, Du & Chen (2020) suggested a kind of important tensor optimization problem with higher-order nonlinear equations, widely used in engineering and economics. The algorithm is concerned with solving -tensor equations by transforming the equations to nonlinear unconstrained optimization problems. The effectiveness of the proposed nonlinear conjugate gradient method was compared with the three-term conjugate gradient method and Newton method. Numerical results show that the proposed nonlinear conjugate gradient method is potentially efficient.  (Dai & Yuan, 2001). Numerical experiments show that the FR, DY and CD conjugate schemes are characterized by strong global convergence properties and have poor practical performances due to jamming. On the contrast, LS, HS and PRP have better practical performances, but may not always be convergent (Babaie-Kafaki & Ghanbari, 2014c). To improve these schemes' behavior and avoid numerical uncertainty, researchers were interested in combining CG schemes of the two groups (Babaie-Kafaki & Mahdavi-Amiri, 2013).
There are some strengths and weaknesses in the theory of CG schemes. The first global convergent property of FR method with exalt line was proved by Zoutendijk (1970); where Al-Baali (1985) extended this result to an inexact line search and show that the method generates sufficient descent direction under the strong Wolfe conditions using the constraint < 1 2 . However, the HS and PRP schemes possess an automatic approximate restart feature that addresses a jamming problem that makes them numerically efficient (Babaie-Kafaki, 2013). Here this research considers a new convex combination of HS and FR conjugate gradient methods. The corresponding conjugate gradient parameters are and = ‖ +1 ‖ 2 ‖ ‖ 2 The structures of CG update parameters were obtained from conjugacy condition and secant equation which depends on the exact line search. These procedures require computation and storage of the Hessian matrix, respectively. However, the practical numerical analysis adopts inexact line searches instead of exact line searches to obtain the step-size. To address these drawbacks, this article, therefore, presents a hybrid method from Dai-Liao conjugacy condition so that, if the modulating parameter t = 1, then it reduces to a method that uses pure conjugacy condition. The numerical performance of the Dai-Liao CG method depends on the parameter t. The best choice of t remains subject of consideration (Babaie-Kafaki & Ghanbari, 2017).
The article aims to modify the CG methods using classical HS and FR method by employing optimal choice of the parameter t for solving large scale unconstrained optimization problems. This paper is organized as follows: Section 2 presents the proposed method. Convergence results are presented in Section 3. Some numerical results are reported in Section 4. Finally, conclusions are made in Section 5.

Literature Review
A large number of hybrid conjugate gradient techniques were proposed. The idea is to combine different conjugate algorithms to use the projection to form a new hybrid convex-combination algorithm to avoid jamming and improve the convergence analysis (Mohammed, et al., 2020). Djordjevic (2016;2018) proposed hybrid conjugate gradient algorithms. The conjugate gradient parameters are computed as a convex combination of the hybrid parameter , where they are computed in such a way that the conjugacy condition(s) are satisfied using strong Wolfe line search conditions, which has the following formulas for respectively On the other hand; Djordjevic (2019), Al-Namat & Al-Naemi (2020) and Salihu et al. (2020) recently derived new hybrid schemes for solving large scale unconstrained optimization algorithms. The hybrid schemes satisfy the sufficient descent condition in such a way that Newton directions are employed, global convergence analysis were proved under the same conditions above, and the algorithms are characterized by following .
Numerical comparisons show that these algorithms behave better than some known methods. Based on the modified BFGS method proposed by Li & Fukushima (2001), Lotfi & Hosseini (2019) presented a new value of the parameter in Dai-Liao CG scheme. The proposed method's global convergence property was established, and numerical results illustrated the computational efficiency of the new method. Considerable efforts have recently been made to extend CG methods to solve monotone nonlinear equations, Abubakar et-al. (2019) presented a modification of the FR CG method for constrained monotone nonlinear equations. The method possesses sufficient descent property, and its global convergence was proved. Numerical experiments show efficiency of the proposed method using some benchmark test problems while applying the method in signal and image recovery problems arising from compressive sensing. Also, Xue et-al. (2018) suggested DY CG method for solving largescale unconstrained optimization problems, which possesses a spectral CG parameter in which the search direction generated at each iteration is independent of any line search. Global convergence of the method is also established using strong Wolfe conditions. Finally, comparison experiments on impulse noise removal are reported and demonstrated the effectiveness of the method.
These new methods are based on secant equations or conjugacy condition, for nonlinear conjugate gradient methods, the conjugacy condition is given by Perry (1978) extended the result in (15) by exploiting the following secant condition of quasi-Newton scheme +1 = and quasi-Newton search direction given by +1 +1 = − +1 , where +1 is a square matrix of the Hessian approximation; as which implies that (16) holds for exact line search, where − +1 = 0, but practical numerical computations normally adopt inexact line search; that is, − +1 ≠ 0. For this reason, Dai and Liao (2001) replaces (16) with a condition called extended conjugacy condition: Due to the simpler structure and low memory requirements of Dai-Liao conjugate gradient methods, Yao et-al. (2019) (2017) using (17) to access and combine the CG update parameters' strength.

Research Methodology
In this section, this research combines the CG update parameters proposed by Hestenes & Stiefel (1952) with Fletcher & Reeves (1964) conjugate descent based on Dai-Liao conjugacy condition as a convex combination as follows: From relations (7) and (8), it obtains = (1 − ) ( +1 ) + ( where is the hybridization scalar parameter satisfying ∈ [0,1]. It is obvious that if ≤ 0, set = 0, then = and if ≥ 0, set = 1, then = . On the other hand, if0 < < 1, then is a proper convex combination of and . Therefore, from relation (6) and by taking the inner product with the vector it obtains Applying on +1 = −∇ 2 ( +1 ) −1 +1 and equating with (21), lead to the following hybridization parameter of Djordjevic (2018), which imply that +1 satisfies Newton direction: Here, this research uses Dai-Liao conjugacy condition (17) on (21), and after some algebra, this research proposes another hybridization parameter as follows: However, for large-scale problems, the update parameter choices that do not require evaluation of the Hessian matrix are often required. Therefore, to have an algorithm for solving large-scale problems, this research computes the modulating parameter t from optimal choice obtain in Babaie-Kafaki and Ghanbari (2015) and Andrei (2017). * = . (24)

Convergence Analysis
In this section, the convergence result of the hybrid CG method is analyzed base on strong Wolfe condition, an algorithm has to possess both sufficient descent condition and global convergence properties to be convergent.

Sufficient Descent Condition
and also satisfies sufficient descent condition if and only if where c is positive constant.

Convergence Analysis
In this section, this research applies the following theorems to illustrate the global convergence of DHF method. It is necessary to show that and * are bounded. Therefore, the following basic assumptions are: Assumption (i). The level set S = { ∈ ℝ ∶ ( ) ≤ ( 0 )}, is bounded from below. That is, there exists a positive constant B such that‖ ‖ ≤ B, ∀ ∈ S.
Assumption (ii). In a neighborhood , the objective function is continuously differentiable and its gradient ( )is Lipschitz continuous on that is, there exist a constant > 0such that International Journal of Industrial Optimization Vol. 2, No. 1, February 2021, pp. 33-50 P-ISSN 2714-6006 E-ISSN 2723 Salihu et al. 41 Under Assumptions (i) and (ii) on , there exist a constant such that For any conjugate gradient method with a strong Wolfe line search, the convergence holds. But, for general function, only a weak form of the Zoutendijk condition is needed (Dai & Liao, 2001).
A CG method converges globally if = 0 for some or (38) holds.
Theorem 3.2. Consider the iterative method, defined by DHF method. Let +1 be a descent direction, then either = 0, for some , or The proof is using contradiction, that theorem (3.1) is not true.
Proof: Let ≠ 0, for all . Then it has to prove (39). Suppose, on the contrary, that (39) does not hold, which means the gradient is bounded away from zero. Then there exists a constant > 0, such that Let be the diameter of the level set , then Because the descent condition holds for DHF method. Since it has +1 ≠ 0, it is sufficient to prove that +1 is bounded above, so from relation (20), it has Applying Lipschitz condition‖ ‖ ≤ ‖ ‖ and (24) implies that It shows that is bounded using (23), so that This is a contradiction of (38), so it has been proved (39).

Results and Discussion
In this section, we present computational performance of DHF method and compare with Hybrid Hestenes-Stiefel and Fletcher-Revees (HHSFR) of Djordjevic (2018) method. To implement the hybridize CG parameters, the codes were written in Matlab 9.2 (R2018a) and run on a personal computer 2.20 GHz CPU processor and 3.0 GB RAM memory. The test problems are the unconstrained problems from (Andrei, 2008) and (Gould, Orban & Toint, 2003). Since CG schemes are mainly designed to solve large-scale unconstrained optimization, we select 24 unconstrained optimization problems and tested them on a gradually increasing number of dimension(s) from 100 to 1000000 as shown in Table 1. The stopping criterion is set to‖ ‖ ∞ ≤ 10 −4 . Numerical results were compared based on the performance profile of Dolan and Mor´e (2002) and shown graphically in figures 1-2.
Benchmark results are generated by running a solver on a set of problems and recording information of interest such as the number of iterations and the computing time. A solver has higher efficiency when its value of ( ) is higher. The ( )from the performance, the profile is the fraction of the problem with a high ratio performance . In a set of problem and a set of optimization solver , a performance comparison of problem ∈ by a particular algorithm ∈ is measured. Let, , be the number of iterations or CPU time required when solving a problem ∈ with solver ∈ . The performance ratio is defined by , = , min { , : ∈ } . From this expression, it is assumed that , ∈ [1, ], where ≥ , and , = only when problem is not solved by the solver. Then, graphically, a graph of ( ) versus ∈ [1, ] is plotted. In a graph of performance profile, the smallest performance ratio is 1, and it will be located at the most left of -axis hence, the top curve represents the most efficient method. In particular, if the set of problems is suitably large and representative of problems that are likely to occur in applications, then solver with large probability ( ) are to be preferred.  Digital image processing plays an important role in medical sciences, biological engineering, and other science and engineering areas. Ibrahim et al. (2020) combined Solodov and Svaiter method with the Liu-Storey and Fletcher-Reeves conjugate gradient algorithm of Djordjevic unconstrained minimization problems to propose a hybrid conjugate gradient algorithm and extend the result to solve convex constrained nonlinear monotone equations. The global convergence established and applied to solve the 1 -norm regularized problems to restore sparse signal and image in compressive sensing. Numerical comparisons of the algorithm with some sparse signal reconstruction and image restoration in compressive sensing CG algorithms show that the proposed scheme is computationally more efficient and robust than the compared schemes. Ibrahim et al. (2020) utilized HLSFR Algorithm of Djordjevic (2019) in the restoration of one-dimensional sparse signal and image restoration using mean squared error (MSE). The performance of HLSFR won and proves to be more efficient in decoding sparse signals in compressive sensing by a lesser number of iterations, computing time, and lesser MSE by repeated experiment on 10 different noise samples. The HLSFR algorithm is similar to the HHSFR of Djordjevic (2018) and DHS algorithms here.

Conclusion
This paper has presented a new hybrid conjugate hybrid algorithm in which the CG parameter is computed as a convex combination of and from Dai-Liao conjugacy condition by employing an optimal choice of the modulating parameter . Numerical computation adopts inexact line search, which is compared with HHSFR conjugate gradient coefficient proposed by Djordjevic. The method requires the first-order derivatives but overcomes the steepest descent method's shortcoming of slow convergence and needs not to save or compute the second-order derivatives needed by the Newton method. Numerical results show that DHR coefficient outperforms the HHSFR scheme and suitable in compressed sensing. The algorithm converges globally using strong Wolfe conditions. International Journal of Industrial Optimization Vol. 2, No. 1, February 2021, pp. 33-50 P-ISSN 2714-6006 E-ISSN 2723