3.3 Prompt optimization for reasoning
While LLMs demonstrate an impressive performance in reasoning tasks, their performance can be sensitive to the prompt used to guide their behavior. In particular, with the right choice of a prompt, their performance can be significantly improved [49]. In prompt optimization, the goal is to find a prompt or an instruction to guide the behavior of an LLM, such that it performs well on a given task. In particular, we often have a computation graph like the following:
| Dataset | Method | Accuracy (%) |
| Google-proof QA [27] | CoT [46, 47] | 51.0 |
| Best reported [48] | 53.6 | |
| TEXTGRAD | 55.0 | |
| MMLU-Machine Learning [45] | CoT [46, 47] | 85.7 |
| TEXTGRAD | 88.4 | |
| MMLU-College Physics [45] | CoT [46, 47] | 91.2 |
| TEXTGRAD | 95.1 |
where we have a Question for the task, an Answer to the question, and an Evaluation Metric indicating the quality of the output given the ground truth answer. For instance, for a question- answering task, the evaluation metric would be the accuracy of the answer, evaluated e.g. using string- based metrics.
Here, given a handful of training examples to optimize a Prompt, the goal is to maximize the performance of an LLM on the given task. In our experiments, our goal is to improve the performance of a weaker and cheaper model (e.g., gpt- 3.5- turbo- 0125) using the feedback generated by stronger models (e.g., gpt- 4o). This is useful in practice because by paying a fixed cost to optimize a Prompt, the prompt- optimized weaker model can be used with cheaper inference costs instead of using the strong and more expensive model.
1 # Initialize the system prompt 2 system_prompt = tg.Variable("You are a helpful language model. Think step by step.", 3 requires_grad=True, 4 role_description="system prompt to the language model") 5 6 # Set up the model object 'parameterized by' the prompt. 7 model = tg.BlackboxLLM(system_prompt=system_prompt) 8 9 # Optimize the system prompt 10 optimizer = tg.TextualGradientDescent(parameters=[system_prompt]) 11 12 for iteration in range(max_iterations): 13 batch_x, batch_y = next(train_loader) 14 optimizer.zero_grad() 15 # Do the forward pass 16 responses = model(batch_x) 17 losses = [loss_fn(response, y) for (response, y) in zip(responses, batch_y)] 18 total_loss = tg.sum(losses) 19 # Perform the backward pass and compute gradients 20 total_loss.backward() 21 # Update the system prompt 22 optimizer.step()
Code Snippet 2: A simple implementation of prompt optimization with TEXTGRAD.
Here, we use TEXTGRAD in a minibatch stochastic gradient descent setting [18, 34]. In particular, at each iteration, we use a few training examples to run the forward pass in Equation 16. A pseudocode and short implementation can be found in the snippet above. Full details of prompt optimization can be found in Appendix E. Unlike instance optimization, where TEXTGRAD tries to optimize each individual solution, the goal here is to optimize a single prompt that works well across all the questions in a benchmark.
Tasks: We explore prompt optimization in multiple datasets, including the two standard reasoning tasks (Object Counting and Word Sorting) from Big Bench Hard (randomly split into 50/100/100 train/validation/test samples) [50, 51], GSM8k grade- school math problem solving [52] (using train/validation/test splits from DSPy). For GSM8k and Object Counting, we use a string- based exact match metric to quantify accuracy (i.e. whether the final number provided in the response is the same as the ground truth answer). For Word Sorting, we use an LLM to compare the response and the ground truth answer. We give more details about the tasks, prompts, and example queries in Appendix E.1.
.
.
.
Methods: We explore improving the performance of gpt- 3.5- turbo- 0125 using gpt- 4o [48] to provide feedback during backpropagation. In particular, while the forward model that performs the reasoning is gpt- 3.5- turbo- 0125, we use gpt- 4o to provide the feedback and improve the prompt. We use a batch size of 3 with 12 iterations, i.e., the model sees 36 training examples in total, sampled randomly with replacement. After each iteration, we run a validation loop with the validation set of the datasets, and if the performance is better than the previous iteration we update the Prompt.
Baselines: We have two main baselines:
1. Zero-shot Chain-of-Thought (CoT) [46, 47]: We initialize all prompts as zero-shot CoT prompt, where a model is instructed to 'Think step-by-step' to explain its reasoning before giving its answer. This strategy is well-known to be a strong baseline for prompting.
2. DSPy is a state-of-the-art language model programming and prompt optimization framework [10], thus we use it as the reference baseline. We instantiate DSPy's BootstrappedFewShotRandomSearch (BFSR) optimizer with 10 candidate programs and 8 few-shot examples. This optimizer identifies demonstrations to include in the prompt as few-shot examples. This is done through generating traces of LLM inputs and outputs that individually pass the metric (in this case, accuracy) and includes CoT reasoning. It then applies random search over subsets of up to size eight shots with these demonstrations.
| Dataset | Method | Accuracy (%) |
| Object Counting [50, 51] | CoT (0-shot) [46, 47] | 77.8 |
| DSPy (BFSR, 8 demonstrations) [10] | 84.9 | |
| TExTGRAD (instruction-only, 0 demonstrations) | 91.9 | |
| Word Sorting [50, 51] | CoT (0-shot) [46, 47] | 76.7 |
| DSPy (BFSR, 8 demonstrations) [10] | 79.8 | |
| TExTGRAD (instruction-only, 0 demonstrations) | 79.8 | |
| GSM8k [52] | CoT (0-shot) [46, 47] | 72.9 |
| DSPy (BFSR, 8 demonstrations) [10] | 81.1 | |
| TExTGRAD (instruction-only, 0 demonstrations) | 81.1 |
Example: TextGrad optimized prompt for gpt- 3.5- turbo- 0125
Prompt at initialization (GSM8k Accuracy ):
You will answer a mathematical reasoning question. Think step by step. Always conclude the last line of your response should be of the following format: 'Answer: VALUE' where VALUE is a numerical value."
Prompt after 12 iterations with batch size 3 (GSM8k Accuracy ):
You will answer a mathematical reasoning question. Restate the problem in your own words to ensure understanding. Break down the problem into smaller steps, explaining each calculation in detail. Verify each step and re-check your calculations for accuracy. Use proper mathematical notation and maintain consistency with the context of the question. Always conclude with the final answer in the following format: 'Answer: VALUE' where VALUE is a numerical value.
Results: Across all three tasks, TExTGRAD improves the performance of the 0- shot prompt significantly. It performs similarly to DSPy [10] for Word Sorting and GSM8k, and improves over DSPy by for Object Counting. While the 8 demonstrations in the context can help guide the behavior of the LLM, it can increase the cost of inference. Interestingly, the DSPy optimizer and TExTGRAD make complementary adjustments—the former adds in- context demonstration examples and latter optimizes the system prompt. Adding the examples selected by DSPy to TExTGRAD's optimized prompt could further improve perfor mance (for GSM8k, directly combining the demonstrations from DSPy with the instruction from TextGrad increases the accuracy to ), suggesting that a fruitful direction is to combine both approaches.