Skip to main content
TextGrad: Automatic "Differentiation" via TextMert Yuksekgonul et al.

Batch Optimization

In batch optimization, we use the tg.sum function described above. In particular, gradients propagating from multiple instances are concatenated together, thus the optimizer sees all of the feedback to a variable coming from multiple sources.

The syntax is as simple as the following:

1 losses == [loss_fn(answer, model(question)) for question, answer in batch] 2 total_loss == tg.sum(losses) 3 total_loss.backward()

Code Snippet 4: An example use for batch optimization for question answering.