Optimization Techniques
Automatic differentiation is a strong analogy for TEXTGRAD and provides conceptual support for optimization techniques implemented in the framework. Below, we describe some examples.
Batch Optimization: We implement stochastic minibatch gradient descent [18, 34] for prompt optimization. Specifically, after doing a forward pass with multiple instances in a batch and evaluate individual loss terms, we use the tg.sum function to sum the losses (akin to torch.sum). In the backward pass, gradients on variables through individual loss terms are concatenated, mirroring backpropagating through addition.
Constrained Optimization: We use natural language constraints, building on constrained optimization as an analogy [35]. In particular, we use natural language constraints (e.g., such as 'The last line of your response should be of the following format: 'Answer: \*SELECT' where LETTER is one of ABCD.'') to guide the behavior of the optimizer. We observe that thanks to instruction- tuning [36, 37], language models can follow these simple constraints, although their reliability can reduce with too many constraints [38, 39].
Momentum: We use the analogy to momentum in gradient descent [40, 41]. When optimizing a variable, the TGD optimizer can optionally see the earlier iterations of the variable when making the update. For more details on optimization techniques that are implemented in TEXTGRAD, see Appendix B.