Researchers gave AI assistants to 5,000 customer support agents and tracked how many issues each agent resolved per hour.
On average, output increased by 14%. That’s a solid result. But when Erik Brynjolfsson and his co-authors looked at the data by skill level, the story changed. Novices, or those new to the job, resolved 34% more issues. Experienced agents saw almost no improvement, maybe just a slight boost.
Everyone used the same software, worked in the same place, and handled the same types of customer issues. Yet, the tool that transformed one person’s day made little difference for the person right next to them.
Remember this, as it clarifies a larger question.
The forecasts and the ledger don’t match
In March, Goldman Sachs released a report saying AI could boost global GDP by 7%, or about $7 trillion, and add 1.5 percentage points per year to US labor productivity growth.
Three months later, McKinsey estimated generative AI could add $2.6 to $4.4 trillion in value each year. They were just as enthusiastic, but their productivity estimate was much lower, only 0.1 to 0.6 percentage points per year through 2040. That’s much less than Goldman’s projection. Both are respected firms, looking at the same technology, just a few months apart.
But here’s what really happened: US labor productivity dropped by 1.7% in 2022, the biggest yearly decline since 1974. It went up by 3.5% in the second quarter this year, which sounds good until you look closer. Output grew by 1.9%, but hours worked fell by 1.5%. So, productivity rose partly because people worked less, not because they produced much more.
Last month, Gartner said generative AI is at the peak of inflated expectations. They predict it will bring major benefits in two to five years.
Around one-third of organizations say they use generative AI in at least one part of their business. But only 22% of individuals say they use it in their own jobs.
This has a name, and it’s older than most of us
Robert Solow reviewed a book for the New York Times Book Review in 1987 and left behind a line that outlasted the review by decades: “You can see the computer age everywhere but in the productivity statistics.” Corporations were buying computers by the truckload. Growth had slowed from around 3.3% a year in the postwar decades to roughly 1.5%.
In 1993, Brynjolfsson gave this issue a name and suggested four explanations that people still use today:
- Mismeasurement: The statistics were built for factories and miss what software changes.
- Redistribution: One firm’s gain is a rival’s loss, so the aggregate stays flat.
- Time lags: The payoff shows up years later, once organizations rebuild how they work.
- Mismanagement: Buying the tool and getting value from it are different projects.
He revisited the topic in 2017 with Daniel Rock and Chad Syverson in a paper called “Artificial Intelligence and the Modern Productivity Paradox.” They argued that while AI kept beating human benchmarks, measured productivity growth had slowed by half. They concluded that most of the gap was due to delays in putting new technology to use.
The dynamo got there first
Long before chatbots were a concern, economist Paul David looked for a similar example in history and found it in electricity.
Edison opened his first central power stations in 1881. By 1899, eighteen years later, only 3% of American homes had electric lighting, and electric motors made up less than 5% of factory drive power. Manufacturing productivity didn’t really increase until the early 1920s, almost forty years later.
The delay wasn’t because of the technology itself. The real issue was how buildings were set up.
For about twenty years, factories used what David called group drive: they swapped the steam engine for an electric motor, but kept the same overhead shafts and belts to run clusters of machines. The power source changed, but the factory layout stayed the same. What’s more, the old shafts, belts, and engines were often kept as backup. This meant more money was spent, but output didn’t increase, so productivity actually went down.
The breakthrough came with unit drive: putting a motor on each machine. This made the old overhead shafts unnecessary, so they could be removed. Without shafts, buildings didn’t need heavy ceiling supports and could be lighter. Factories could now be single-story, since multistory designs were mainly to keep shafts short. On one floor, machines could be arranged to match the actual workflow.
About half of the increase in US manufacturing productivity during the 1920s came from this change.
David anticipated the irony. He wrote, “In 1900, contemporary observers well might have remarked that the electric dynamos were to be seen ‘everywhere but in the productivity statistics!’”
So what counts as a productivity tool now?
The definition of productivity tools has expanded over time. It used to cover just four things: documents, spreadsheets, project management, and communication. Tools like Word, Excel, Slack, and Zoom helped make existing tasks easier.
AI assistants complement these tools rather than replace them, working hand in hand to enhance what we do. Now, the term “AI productivity tool” includes products that are very different from each other. For example, Copilot is built into a whole suite with your organization’s permissions, while ChatGPT is a standalone tool that works independently. They share a label, but they’re not the same.
What the coding numbers actually say
In February, Microsoft and GitHub researchers did a controlled test. Developers using Copilot built an HTTP server in JavaScript 55.8% faster than those without it. That’s impressive, but it’s important to note this was a new project with no existing code to consider, an ideal situation for pattern-matching tools.
Compare that to a Stanford study from late 2022. People using an AI assistant wrote code that was actually less secure, but they were more confident in its safety. An NYU team found that about 40% of the 1,689 programs Copilot generated in 89 scenarios had a vulnerability.
Developers seem aware of the issues. In a June Stack Overflow survey, 44% said they use AI tools and 77% had a positive view of them, but only 3% highly trust the results. Twice as many strongly distrust them.
On paper, none of this costs much. Copilot is $10 a month for individuals and $19 per user for businesses. CodeWhisperer’s individual plan is free. When the price is this low, it seems easy to justify the cost, so most people don’t consider the other side of the equation.
Where I think this goes
What follows is just my opinion, not a proven fact. There’s no data yet to confirm I’m correct.
I think the same pattern seen in call centers appears in engineering, but even more strongly. If AI mainly helps weaker performers adopt the habits of stronger ones, its value drops as people become more skilled. For example, a senior engineer who knows a codebase inside out might see little or no benefit. In mature systems, I suspect the gains are close to zero, and could even turn negative when you factor in the time spent reading, checking, and fixing code.
We often tend to misjudge this, and the Stanford study really highlights why. It shows that even when people do work that’s not as good, they might feel more confident about it. Feeling fast and actually being fast are not the same, and only one is easy to measure.
The bottleneck is shifting, too. David pointed this out in 1990: when it costs nothing to share information, people send everything, making it expensive to review. Soon, generating code will be very cheap, but reading and reviewing it won’t be. That extra work falls on reviewers, and I don’t think most teams realize they’re about to become the bottleneck.
There’s also the group-drive problem, which concerns me most. Many teams are adding AI to workflows that assume writing code is the slowest step. The process stays the same, just with faster typing. If we learn from David’s factories, this approach leads to more code, more duplication, and extra maintenance work that no one planned for. It’s like leaving the old belts in place while adding new motors.
Brynjolfsson and Hitt found something in 2000 that I keep thinking about: companies that adopted computers without changing how they worked didn’t just miss out on gains; they sometimes fell behind, because the new tools clashed with old habits. Partial adoption was worse than not adopting at all. I expect many AI projects will quietly run into this problem and be labeled as failures of the tool, when the real issue was how the organization used it.
This brings up an idea from Daron Acemoglu and Pascual Restrepo that has stuck with me: so-so technology. This is automation that really works and is widely used, but the gains are too small to show up in productivity statistics. It’s not a failure; it’s just a smaller success than most people expect.
The truth is, we’re in a situation like 1899, debating what will happen by 1920. Electricity was real in 1899, and the dynamos were running. But no one had rethought how to design buildings yet, and that ended up making all the difference.
