Λεπτομέρειες βιβλιογραφικής εγγραφής
| Τίτλος: |
On Averaging and Extrapolation for Gradient Descent. |
| Συγγραφείς: |
Luner, Alan1 (AUTHOR) aluner1@jhu.edu, Grimmer, Benjamin1 (AUTHOR) grimmer@jhu.edu |
| Πηγή: |
Mathematics of Operations Research (INFORMS). Aug2026, Vol. 51 Issue 3, p2158-2188. 31p. |
| Θεματικοί όροι: |
*Mathematical optimization, Extrapolation, Optimization algorithms, Convex programming, Benchmark problems (Computer science) |
| Περίληψη: |
This work considers the effect of averaging, and more generally extrapolation, of the iterates of gradient descent in smooth convex optimization. After running the method, rather than reporting the final iterate, one can report either a convex combination of the iterates (averaging) or a generic combination of the iterates (extrapolation). For several common stepsize sequences, including recently developed accelerated periodically long stepsize schemes, we show averaging cannot improve gradient descent's worst-case performance and is, in fact, strictly worse than simply returning the last iterate. In contrast, we prove a conceptually simple and computationally cheap extrapolation scheme strictly improves the worst-case convergence rate: when initialized at the origin, reporting (1+1/16Nlog(N))xN rather than xN improves the best possible worst-case performance by the same amount as conducting O(N/log(N)) more gradient steps. Our analysis and characterizations of the best possible convergence guarantees are computer-aided, using performance estimation problems. Numerically, we find similar (small) benefits from such simple extrapolation for a range of gradient methods. Funding: This work was supported by the Air Force Office of Scientific Research [Grant FA9550-23-1-0531]. [ABSTRACT FROM AUTHOR] |
|
Copyright of Mathematics of Operations Research (INFORMS) is the property of INFORMS: Institute for Operations Research & the Management Sciences and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) |
| Βάση Δεδομένων: |
Business Source Index |
