E.1 Tasks
Below, we provide an example query for each of the tasks in the prompt optimization section.
Example Query for Word Sorting
Sort the following words alphabetically: List: oakland seaborg jacobi membrane trapezoidal allis marmot toggle anthology.
Example Query for Object Counting
I have a couch, a bed, a car, a fridge, two tables, an oven, a toaster, and a chair. How many objects do I have?
Example Query for GSM8k
Amber, Micah, and Ahito ran 52 miles in total. Amber ran 8 miles. Micah ran 3.5 times what Amber ran. How many miles did Ahito run?
For word sorting and object counting, we obtain the datasets from the BBH repository, and we randomly split examples into 50 (training)/100 (validation)/100 (test) samples. For GSM8k, we use the splits provided in DSPy [10] which has 200 (training)/300 (validation)/1319 (test) samples.
Evaluation: For object counting and GSM8k, we use the string- based exact match metric, which looks at the last numerical value provided in the answer, and compares it to the ground truth answer. For word sorting, we prompt gpt- 4o to compare the ground truth list to the response provided in the answer, through the following prompt:
Evaluation system prompt for Word Sorting evaluation
System Prompt: Below is a question from a question- answering task, the ground truth answer, and reasoning with the final prediction. Is the final prediction correct, i.e. the same as the ground truth answer? Say only 1 (yes) or 0 (no). Return your response within tags. e.g. or .
Example prompt: \*\*Question for the task:\*\* {question} \*\*Ground truth answer:\*\* {answer} \*\*Reasoning and prediction from the language model:\*\* {prediction}