D.1 Methodology
For the CoT 0- shot prediction, we use the question template and system prompt released with GPT- 4o in the simple- evals repository. In particular, to closely match their evaluations, we use the ChatGPT system prompt: You are ChatGPT, a large language model trained by OpenAI, based on the GPT- 4 architecture. \n Knowledge cutoff: 2023- 12 \n Current date: 2024- 04- 01". Further, we use the following template:
Multiple Choice Question Answering Template
Answer the following multiple choice question. The last line of your response should be of the following format: 'Answer: \(LETTER' (without quotes) where LETTER is one of ABCD. Think step by step before answering.
{Question}
A) {A}
B) {B}
C) {C}
D) {D}
During optimization, we provide the constraint to the optimizer that the prediction should conclude with an answer, following the simple- evals repository, through the following constraint description: The last line of your response should be of the following format: 'Answer: \(LETTER' (without quotes) where LETTER is one of ABCD.
Evaluation: Similarly, using the practice in the simple- evals repository, we perform string matching to find the final answer, which is one of the letters ABCD, and compare it to the ground truth answer. GPQA Diamond subset contains 198 questions. MMLU Machine Learning subset contains 112 questions, and College Physics subset contains 92 questions. At each iteration of optimization, we make 1 call to gpt- 4o to evaluate the test time loss, 1 call to collect gradients, and 1 call to update the solution.