Skip to main content
TextGrad: Automatic "Differentiation" via TextMert Yuksekgonul et al.

C.1 Methodology

Baseline Details. We re- run Reflexion [26] using the code by the authors currently available online. We had to make minor changes to ensure it ran correctly in our setting but we also contacted the original authors and ask feedback on our edits to ensure our evaluation was consistent. Moreover, we had to re- extract the LeetCodeHard [26] dataset using the authors' pipeline; this means that it is likely that the dataset we are using in this manuscript is not the same dataset that was used in the Reflexion paper. While this dataset contains a set of simple tests to check if the code works as expected, the real evaluation in this context is given by passing the tests on the LeetCode platform.

Reflexion is run with a one shot prompt that is meant to instruct the model on how to provide feedback. The pipeline run by Reflexion is as follows: given in input a prompt to generate code, a language model generates a first solution. If this solution passes the local tests, it is then submitted to the LeetCode platform to be evaluated on harder tests. However, if the solution does not pass the tests, we ask for feedback through Reflexion and optimize the code. Once again, if the new solution passes the local tests, we submit it to the LeetCode platform. We do this optimization for 5 iterations.

We ran the experiment 5 times with 5 different seeds and we averaged the results. At each iteration of optimization, TEXTGRADmakes 1 call to gpt- 4o to evaluate the test time loss, 1 call to collect gradients, and 1 call to update the code snippet. The number of coding problems in LeetCodeHard is 39.

Example Query for LeetCode Hard

def minimumTime(grid: List[List[int]]) -> int: You are given a 'm x n' matrix 'grid' consisting of non- negative integers where 'grid[row][col]' represents the minimum time required to be able to visit the cell ('row, col'), which means you can visit the cell ('row, col') only when the time you visit it is greater than or equal to 'grid[row][col]'. You are standing in the top- left cell of the matrix in the '0th' second, and you must move to any adjacent cell in the four directions: up, down, left, and right. Each move you make takes 1 second. Return the minimum time required in which you can visit the bottom- right cell of the matrix. If you cannot visit the bottom- right cell, then return '- 1'. Example 1: Input: grid = [[0,1,3,2],[5,1,2,5],[4,3,8,6]] Output: 7 Explanation: One of the paths that we can take is the following:- at t = 0, we are on the cell (0,0).- at t = 1, we move to the cell (0,1). It is possible because grid[0][1] <= 1.- at t = 2, we move to the cell (1,1). It is possible because grid[1][1] <= 2.- at t = 3, we move to the cell (1,2). It is possible because grid[1][2] <= 3.- at t = 4, we move to the cell (1,1). It is possible because grid[1][1] <= 4.- at t = 5, we move to the cell (1,2). It is possible because grid[1][2] <= 5.- at t = 6, we move to the cell (1,3). It is possible because grid[1][3] <= 6.- at t = 7, we move to the cell (2,3). It is possible because grid[2][3] <= 7. The final time is 7. It can be shown that it is the minimum time possible. Example 2: Input: grid = [[0,2,4],[3,2,1],[1,0,4]] Output: - 1 Explanation: There is no path from the top left to the bottom- right cell. Constraints:\*'m == grid.length'\*'n == grid[i].length'\*'2 <= m, n <= 1000'\*'4 <= m * n <= 105'\*'0 <= grid[i][j] <= 105'\*'grid[0][0] == 0'