Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More
Large language models (LLMs) may have changed software development, but enterprises will need to think twice about entirely replacing human software engineers with LLMs, despite OpenAI CEO Sam Altman’s claim that models can replace “low-level” engineers.
In a new paper, OpenAI researchers detail how they developed an LLM benchmark called SWE-Lancer to test how much foundation models can earn from real-life freelance software engineering tasks. The test found that, while the models can solve bugs, they can’t see why the bug exists and continue to make more mistakes.
The researchers tasked three LLMs — OpenAI’s GPT-4o and o1 and Anthropic’s Claude-3.5 Sonnet — with 1,488 freelance software engineer tasks from the freelance platform Upwork amounting to $1 million in payouts. They divided the tasks into two categories: individual contributor tasks (resolving bugs or implementing features), and management tasks (where the model roleplays as a manager who will choose the best proposal to resolve issues).
“Results indicate that the real-world freelance work in our benchmark remains challenging for frontier language models,” the researchers write.
The test shows that foundation models cannot fully replace human engineers. While they can help solve bugs, they’re not quite at the level where they can start earning freelancing cash by themselves.
Benchmarking freelancing models
The researchers and 100 other professional software engineers identified potential tasks on Upwork and, without changing any words, fed these to a Docker container to create the SWE-Lancer dataset. The container does not have internet access and cannot access GitHub “to avoid the possible of models scraping code diffs or pull request details,” they explained.
The team identified 764 individual contributor tasks, totaling about $414,775, ranging from 15-minute bug fixes to weeklong feature requests. These tasks, which included reviewing freelancer proposals and job postings, would pay out $585,225.
The tasks were added to the expensing platform Expensify.
The researchers generated prompts based on the task title and description and a snapshot of the codebase. If there were additional proposals to resolve the issue, “we also generated a management task using the issue description and list of proposals,” they explained.
From here, the researchers moved to end-to-end test development. They wrote Playwright tests for each task that applies these generated patches which were then “triple-verified” by professional software engineers.
“Tests simulate real-world user flows, such as logging into the application, performing complex actions (making financial transactions) and verifying that the model’s solution works as expected,” the paper explains.
Test results
After running the test, the researchers found that none of the models earned the full $1 million value of the tasks. Claude 3.5 Sonnet, the best-performing model, earned only $208,050 and resolved 26.2% of the individual contributor issues. However, the researchers point out, “the majority of its solutions are incorrect, and higher reliability is needed for trustworthy deployment.”
The models performed well across most individual contributor tasks, with Claude 3.5-Sonnet performing best, followed by o1 and GPT-4o.
“Agents excel at localizing, but fail to root cause, resulting in partial or flawed solutions,” the report explains. “Agents pinpoint the source of an issue remarkably quickly, using keyword searches across the whole repository to quickly locate the relevant file and functions — often far faster than a human would. However, they often exhibit a limited understanding of how the issue spans multiple components or files, and fail to address the root cause, leading to solutions that are incorrect or insufficiently comprehensive. We rarely find cases where the agent aims to reproduce the issue or fails due to not finding the right file or location to edit.”
Interestingly, the models all performed better on manager tasks that required reasoning to evaluate technical understanding.
These benchmark tests showed that AI models can solve some “low-level” coding problems and can’t replace “low-level” software engineers yet. The models still took time, often made mistakes, and couldn’t chase a bug around to find the root cause of coding problems. Many “low-level” engineers work better, but the researchers said this may not be the case for very long.
