TY - JOUR

T1 - Analysis of limited-memory BFGS on a class of nonsmooth convex functions

AU - Asl, Azam

AU - Overton, Michael L.

N1 - Publisher Copyright:
© 2020 The Author(s) 2020. Published by Oxford University Press on behalf of the Institute of Mathematics and its Applications. All rights reserved.

PY - 2021/1/1

Y1 - 2021/1/1

N2 - The limited-memory BFGS (Broyden-Fletcher-Goldfarb-Shanno) method is widely used for large-scale unconstrained optimization, but its behavior on nonsmooth problems has received little attention. L-BFGS (limited memory BFGS) can be used with or without 'scaling'; the use of scaling is normally recommended. A simple special case, when just one BFGS update is stored and used at every iteration, is sometimes also known as memoryless BFGS. We analyze memoryless BFGS with scaling, using any Armijo-Wolfe line search, on the function f(x) = a|x^{(1)}| + \sum _{i=2}^{n} x^{(i)}, initiated at any point x_0 with x_0^{(1)}\not = 0. We show that if a\ge 2\sqrt{n-1}, the absolute value of the normalized search direction generated by this method converges to a constant vector, and if, in addition, a is larger than a quantity that depends on the Armijo parameter, then the iterates converge to a nonoptimal point \bar x with \bar x^{(1)}=0, although f is unbounded below. As we showed in previous work, the gradient method with any Armijo-Wolfe line search also fails on the same function if a\geq \sqrt{n-1} and a is larger than another quantity depending on the Armijo parameter, but scaled memoryless BFGS fails under a weaker condition relating a to the Armijo parameter than that implying failure of the gradient method. Furthermore, in sharp contrast to the gradient method, if a specific standard Armijo-Wolfe bracketing line search is used, scaled memoryless BFGS fails when a\ge 2 \sqrt{n-1}regardless of the Armijo parameter. Finally, numerical experiments indicate that the results may extend to scaled L-BFGS with any fixed number of updates m, and to more general piecewise linear functions.

AB - The limited-memory BFGS (Broyden-Fletcher-Goldfarb-Shanno) method is widely used for large-scale unconstrained optimization, but its behavior on nonsmooth problems has received little attention. L-BFGS (limited memory BFGS) can be used with or without 'scaling'; the use of scaling is normally recommended. A simple special case, when just one BFGS update is stored and used at every iteration, is sometimes also known as memoryless BFGS. We analyze memoryless BFGS with scaling, using any Armijo-Wolfe line search, on the function f(x) = a|x^{(1)}| + \sum _{i=2}^{n} x^{(i)}, initiated at any point x_0 with x_0^{(1)}\not = 0. We show that if a\ge 2\sqrt{n-1}, the absolute value of the normalized search direction generated by this method converges to a constant vector, and if, in addition, a is larger than a quantity that depends on the Armijo parameter, then the iterates converge to a nonoptimal point \bar x with \bar x^{(1)}=0, although f is unbounded below. As we showed in previous work, the gradient method with any Armijo-Wolfe line search also fails on the same function if a\geq \sqrt{n-1} and a is larger than another quantity depending on the Armijo parameter, but scaled memoryless BFGS fails under a weaker condition relating a to the Armijo parameter than that implying failure of the gradient method. Furthermore, in sharp contrast to the gradient method, if a specific standard Armijo-Wolfe bracketing line search is used, scaled memoryless BFGS fails when a\ge 2 \sqrt{n-1}regardless of the Armijo parameter. Finally, numerical experiments indicate that the results may extend to scaled L-BFGS with any fixed number of updates m, and to more general piecewise linear functions.

KW - BFGS

KW - nonsmooth optimization

UR - http://www.scopus.com/inward/record.url?scp=85104023522&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85104023522&partnerID=8YFLogxK

U2 - 10.1093/imanum/drz052

DO - 10.1093/imanum/drz052

M3 - Article

AN - SCOPUS:85104023522

VL - 41

SP - 1

EP - 27

JO - IMA Journal of Numerical Analysis

JF - IMA Journal of Numerical Analysis

SN - 0272-4979

IS - 1

ER -