Even the best artificial intelligence agents are fairly hopeless at online freelance work, according to an experiment that challenges the idea of AI replacing office workers en masse.
The Remote Labor Index, a new benchmark developed by researchers at data annotation company Scale AI and the Center for AI Safety (CAIS), a nonprofit, measures the ability of frontier AI models to automate economically valuable work.
The researchers gave several leading AI agents a range of simulated freelance work and found that even the best could perform less than 3 percent of the work, earning $1,810 out of a possible $143,991. The researchers looked at several tools and found the most capable to be Manus from a Chinese startup of the same name, followed by Grok from xAI, Claude from Anthropic, ChatGPT from OpenAI, and Gemini from Google.
“I should hope this gives much more accurate impressions as to what’s going on with AI capabilities,” says Dan Hendrycks, director of CAIS. He adds that while some agents have improved significantly over the past year or so, that does not mean that this will continue at the same rate.
Spectacular AI advances have led to speculation about AI soon surpassing human intelligence and replacing vast numbers of workers. In March, Dario Amodei, CEO of Anthropic, suggested that 90 percent of coding work would be automated within a matter of months.
Previous waves of AI have inspired misplaced predictions about job displacement, for example concerning the imminent replacement of radiologists with AI algorithms.
The researchers generated a range of freelance tasks through verified Upwork workers. The tasks span a range of work including graphic design, video editing, game development, and administrative chores like scraping data. They combined a description of each job with a directory of files needed to perform the work and an example of a finished project produced by a human.
Hendrycks says that while AI models have gotten better at coding, math, and logical reasoning in recent years, they still struggle to use different tools and to perform complex tasks that involve numerous steps. “They don’t have long-term memory storage and can’t do continual learning from experiences. They can’t pick up skills on the job like humans,” he says.
The analysis offers a counterpoint to a benchmark of economic work offered in September by OpenAI called GDPval, which purports to measure economically valuable work. According to GDPval, frontier AI models such as GPT-5 are approaching human abilities on 220 tasks across a range of office jobs. OpenAI did not provide a comment.
“We have debated AI and jobs for years, but most of it has been hypothetical or theoretical,” adds Bing Liu, director of research at Scale AI.
Liu and Hendrycks concede that the new benchmark is not a perfect yardstick for AI’s economic impact. Many professions include tasks not covered by the measure. In reality, many freelancers are also likely to use AI as a tool in a way that amplifies their productivity.
The idea that AI is already taking jobs is gaining momentum however. This week Amazon announced that it would cut 14,000 jobs in a move that it partly blamed on the rapid rise of generative artificial intelligence. “This generation of AI is the most transformative technology we’ve seen since the Internet,” Beth Galetti, senior vice president of people experience and technology at Amazon, wrote in a publicly shared memo. “It’s enabling companies to innovate much faster than ever before (in existing market segments and altogether new ones).”
If the Remote Labor Index is any indication, however, AI is unlikely to be stepping into any of these vacated roles.
Are you worried about AI taking your job? Let me know by sending an email to ailab@wired.com.
This is an edition of Will Knight’s AI Lab newsletter. Read previous newsletters here.
