AI beats human accountants at monthly close, transaction categorization
- Key insight: Here’s why two new studies pitting human accountants against AI yielded highly contrasting results.
- What’s at stake: Human accountants fighting to maintain their roles as advanced AI models outperform them.
- Supporting data: 0.4 percentage points separating the top one-shot AI models from human performance.
Two different studies recently pitted human CPAs against AI models in tests of accounting acumen. While their respective results varied dramatically, both agreed that computers were still no substitute for flesh and blood accountants, at least for now.
Processing Content
The Mercor benchmark
Each participant was given a simulated company’s working files and had to dig through those files to find the right numbers, figure out which calculation approach the company’s rules called for, do the math and then deliver a completed summary table of the results. To reduce pure search time, the human accountants received a file map describing at a high level what each file contained.
The humans performed the tasks on virtual machines meant to replicate the feel of logging onto one’s desktop at work, with all files living locally on the desktop. Each participant had a three-hour window per task and was paid for the full three hours no matter how much time they took, so there was no incentive for the participants to artificially prolong their time spent on each task. Mercor also offered a bonus for each task criterion they satisfied. Aden Barton, econ researcher at Mercor, said they focused on junior accountants specifically because of the way firms usually deploy AI.
“We initially focused on junior accountants (with an average of approximately five years of accounting experience), as AI is often thought to excel at junior-level tasks,” he said in an email. “While we don’t have plans to expand the study to senior accountants at this time, we may expand the study design to other occupations to observe differences in model behavior across domains.”
For the AI models, Mercor tested the models as tool-using ReAct agents over an API gateway and capped them at 250 steps. They matched everything (grader prompts, rubrics, file system, etc.) in the task setup and grading between the human alone and AI alone tasks.
On two of the tasks, the human accountants also had access to Claude Cowork and training on how to use it in an engagement. The remaining two they had to do on their own.
The submissions were graded on rubric items also written by accountants, ranging from 13 to 22 total items depending on the task. One item on a hotel task, for example, required the response to identify that $4,000 of parking revenue had been miscoded as room revenue. The initial grades were produced by an LLM judge (also Claude Opus 5) but to ensure things were calibrated correctly, Mercor also had eight task submissions hand graded — two for each of the four tasks — by the task authors; their grades showed a 98% agreement with the LLM judge.
The result was the AI soundly defeated the junior accountants on all tasks. AI, working on the tasks alone, scored perfectly every time. Barton said they ran the AI through the procedure multiple times, with Claude making 20 attempts across the four tasks and scored 100% on every attempt, each in under 10 minutes.
The humans did not do as well. Without AI assistance, humans generally scored below 50% for accuracy, with only a fraction of attempts scoring above 75%. None achieved a perfect score at any point in the experiment. Further, while the AI only took 10 minutes to complete all tasks, the humans needed 30 minutes at best, but more often took two, sometimes three hours.
The researchers found that, despite strong predictions otherwise, the humans actually took even longer using AI than without it, though they were more accurate (but still not as accurate as AI alone.)
“Accountants working with Claude took about 15X longer per task than Claude working alone,” said the report. “And, interestingly, in three of the four AI-assisted sessions that did not receive a perfect score, Claude had the correct figure at some point, but it didn’t make it into the final submission. In two, Claude revised away from it and the participant deferred to the new answer; in the third, the participant overrode Claude’s initial correct advice.”
The Digits benchmark
The humans did notably better in another
“Categorization is where the judgment is,” he said in an email. “It’s high-volume, happens daily and depends on each business’ own conventions. That’s exactly where automation pays off and where general-purpose models struggle. It has a checkable ground truth.”
The experiment, which compared AI models against each other in addition to humans, utilized a dataset comprising 2,000 financial transactions collected from four businesses that use Digits, which reflected typical transaction patterns with an 88% to 12% split between debits and credits, mirroring the natural distribution commonly observed in accounting systems. The split was skewed toward debit transactions because most transactions’ credit side can be defined by their source (e.g., a given bank account). The number of categories in the charts of accounts varied significantly across these businesses, ranging from 15 to 128.
Digits assessed the performance of LLMs in a single-turn interaction to perform transaction classification. Each model was provided with a uniform system and user prompt, directing it to produce a response in a structured JSON format. All models also received the standardized system prompt: “You are an expert bookkeeper with deep knowledge of accrual-accounting and an eye for detail.”
The user prompt template was dynamically populated with transaction-specific data, including the transaction description, monetary amount, the known counter-entry originating from the ingestion source, and the business’s complete chart of accounts excluding the category associated with the known counter-entry. For example, one prompt said:
1: Description: UBER *TRIP
2: Merchant name: Uber
3: Amount: $44.80
4: Given that this will be recorded as a credit to Mercury Credit (0000)- 1
5: Which category should receive the debit side?
6: “Software & Apps”
7: “Travel”
8: “Meals”
9: ⟨other categories removed from publication for privacy reasons⟩
10: * I want you to think of the most likely category from the list above for the described transaction and amount
11: * Think of a single sentence description of why
12: * Double check that the category is actually in the list
13: NOTE: Do not provide explanations. Only provide the most relevant category.
14: Return them as JSON with the schema: {“category”: ⟨string⟩
To derive a human baseline, Digits employed 12 outsourced accountants, ranging from seasoned accountants with decades of experience to college seniors in accounting, to perform the same tasks. They were organized into four teams of three. Each team was assigned 500 transactions from a single business within Digits’ benchmark dataset for independent categorization. To determine the final classification for each sample, they used a majority-vote system within each three-person group.
All outsourced accountants received information sets identical to those provided to the LLMs being evaluated, ensuring consistency in available data across both human and artificial intelligence classification approaches.
Against the GAAP accountant-reviewed ground-truth dataset, the outsourced accountants achieved an aggregate accuracy of 79.1%. They were beaten by Google Gemini 3.8 Flash (medium effort), which scored 79.15%; GPT-6 Luna (high effort), which scored 79.2%; GPT-6 Luna (low effort), which scored 79.3%; GPT-6 Luna (medium effort), which scored 79.5%; and Digits’ own Automated General Ledger, which scored 97.8%.
However, the humans beat literally dozens of other AI models, including various other versions of Gemini, ChatGPT, DeepSeek, Muse and Claude as well as other models like Kimi, Jev and GLM. Overall, the AIs scored an average 75.9% accuracy, meaning that while the humans technically did lose to the more advanced models, they still stood head and shoulders above most other AIs; Meta’s Muse Glimmer 30B (high effort) scored the worst at 60.35%.
With respect to efficiency, the human accountants required an average of four hours and 43 minutes to complete their allocation of 500 transactions, corresponding to approximately 34 seconds per transaction. Under identical, context-free conditions, the best one-shot LLM configurations surpassed this baseline, though by at most 0.4 percentage points, a margin within sampling error.
The experiment also saw the AI models fitted with an agentic harness, which was intentionally kept lightweight in scope, with the primary objective of equipping reasoning models with targeted retrieval tools capable of surfacing contextual information relevant to transaction booking decisions. Agent runs were capped at four turns where each turn could yield parallel function calls or simply invoking the LLM, after which the model must produce a final classification. In most cases, the harness increased model accuracy, but seldom by more than 10 percentage points, save DeepSeek 4.1 (flash), which went from 68% to 86.7% accuracy, and Meta’s Muse Glimmer 30B (low effort), which went from 66.6% to 79.15% accuracy. Overall, agents with harnesses also used far, far more tokens.
Overall, Digits concluded that one-shot LLM classification has plateaued, as the results were actually slightly worse than the last time it benchmarked public models. The seven most accurate one-shot configurations lay within 0.4 percentage points of each other, a spread well within sampling error. The best one-shot accuracy in this revision is also 1.2 percentage points below the best reported in their previous revision (80.7%)
The failures of the top models cluster on a small number of recurring merchants and on categories whose meaning is set by the business rather than by accounting textbooks: near-duplicate categories, departmental expense categories and payment-processor clearing accounts. The 10 most frequent merchant clusters account for 40–56% of the top models’ errors.
This means, according to Digits, that real progress will come from specialized systems like its own. In an email, CEO Jeff Seibert attributed his product’s success relative to other AI models to their ensemble model architecture that consults a sequence of specialized models from the most specific to the most general, providing business-specific context for the entity in question that allows it to achieve a much higher rate of accuracy.
Don’t count humans out just yet
Despite AIs doing better than humans in both experiments, neither thought this was proof they could replace living accountants.
Digits noted in its report that while the strongest AIs did score higher than the human accountants in one-shot configurations, it was not by that much, 0.4 percentage points, a margin within sampling error. And while the agent harness did manage to exceed the human baseline by a more significant degree, between 3.85 to 9.8 percentage points, the report noted that the agents had an advantage in that it could access the business’s historical bookings, information the outsourced accountants did not have.
Meanwhile, the Mercor report was careful to note that, in its experiment, there were many ways that the task-taking diverged from reality. One of the biggest was lack of context: In a real job, people would normally have months or years of experience with the client and so would have had background information making the task easier. Further, there was no ability to ask questions or clarify ambiguities; in real-world situations, the accountants would be able to call or email the client to ask questions, or they could even turn to a coworker for help, but there was no opportunity for that in this task. Finally, Mercor noted that although the tasks could all be solved with foundational accounting knowledge, they tested specific concepts that may not come up regularly, depending on an accountant’s specialization, such as how a nonprofit splits shared costs.
“The result was that we ended up testing precisely what the models are best at: being detail oriented, searching over the whole file set, and following instructions very closely, i.e., not defaulting to heuristics that work 80% of the time,” said the report. “Humans, by contrast, often rely on context and colleagues not only to catch what they miss but also to brainstorm with.”
Introductory bullet points created by AI with editorial review.
