TextGrad: Automatic "Differentiation" via TextMert Yuksekgonul et al.
Momentum
TEXTGRAD supports the use of Momentum in the Textual Gradient Descent. In standard SGD momentum uses a linear combination of past gradients and the most recent one to define a new gradient to update a variable. Similarly, TEXTGRAD keeps track of past iterations of the variable. This postfix is appended to the prompt for the optimizer.
1 optimizer = TextualGradientDescent(parameters=[solution], momentum_window=3)
Code Snippet 6: How to enable momentum using 3 previous steps in the TextualGradientDescent optimizer.
Momentum prompt
Here are the past iterations of this variable: <PAST_ITERATIONS> {past_values}</PAST_ITERATIONS>