Podcast discusses contradictory AI productivity studies and measurement challenges
Summary
The speaker introduces the topic of AI productivity, noting that empirical findings are contradictory and less conclusive than expected. Three specific studies are cited: experienced software developers were 19% slower but believed they were 20% faster; customer service agents saw a 15% productivity increase, especially novices; and 640 Kenyan entrepreneurs saw no significant business performance improvement from AI access. An AI research agent compiled 82 records, including 29 analytical research entries and 53 company accounts. Erica Groshen, former Commissioner of the Bureau of Labor Statistics, noted over 1,000 studies in 2026 with varying findings. The speaker identifies three factors for divergence: what is measured, task suitability, and user judgment.
Key takeaways
- 1 Experienced software developers were 19% slower but believed they were 20% faster when using AI.
Unidentified speaker (speaker_0) · 0:22
- 2 And in an experiment involving 640 six hundred forty Kenyan entrepreneurs, access to AI. produced no significant improvement in average business performance.
Unidentified speaker (speaker_0) · 0:41
- 3 In a randomized experiment by researchers at Stanford and MIT, accountants using AI to classify financial transactions. improved their accuracy by 17.5 percentage points. But total completion time did not decline significantly.
Unidentified speaker (speaker_0) · 2:16
- 4 AI improved both speed and quality on assignments within its capabilities, but performance deteriorated when consultants relied on it for problems beyond those capabilities.
Unidentified speaker (speaker_0) · 2:36
Episode sections
- 0:00 – 0:21
Introduction to Contradictory AI Productivity Data
The speaker introduces the topic of AI productivity, noting that empirical findings are contradictory and less conclusive than expected. This raises questions for policymakers regarding the actual impact of AI on work speed and quality.
- 0:21 – 0:58
Specific Study Results: Developers, Agents, and Entrepreneurs
The speaker cites three specific studies: experienced software developers were 19% slower but believed they were 20% faster; customer service agents saw a 15% productivity increase, especially novices; and 640 Kenyan entrepreneurs saw no significant business performance improvement from AI access.
- 0:58 – 1:36
Research Inventory and Methodology
On September 15, the speaker launched an AI research agent to monitor empirical studies and company experiences. The agent compiled 82 records, including 29 analytical research entries and 53 company accounts, covering experiments, observational studies, and surveys.
- 1:36 – 2:16
Expert Interview and Divergence Factors
In early October, the speaker interviewed Erica Groshen, former Commissioner of the Bureau of Labor Statistics, who noted over 1,000 studies in 2026 with varying findings. The speaker identifies three factors for divergence: what is measured, task suitability, and user judgment.
- 2:16 – 2:36
Measurement Differences: Speed vs. Accuracy
A Stanford and MIT experiment showed accountants using AI improved accuracy by 17.5 percentage points but did not significantly reduce completion time. The speaker argues that counting hours saved fails to capture the economic value of higher quality work.
- 2:36 – 3:17
Task Suitability and User Judgment
A Boston Consulting Group experiment found AI improved speed and quality within its capabilities but deteriorated performance when consultants relied on it for problems beyond those capabilities. Additionally, novice customer service agents benefited most from AI-embedded knowledge.
- 3:17 – 4:16
Kenyan Experiment and Developer Overhead
In the Kenyan experiment, initially weaker businesses performed worse with AI, while stronger ones improved. A developer experiment using early 2025 AI tools found AI increased completion time for experienced programmers, likely due to the overhead of reviewing AI suggestions on familiar codebases.
- 4:16 – 8:25
Foundation Sponsorship and Personal AI Experience
The speaker discusses Foundation's Envoy and Key OS releases, which allow users to add external entropy for seed creation. The speaker then shares personal experience, noting AI helps process information and identify relationships but requires significant effort to verify and correct errors.
Claims worth verifying
Experienced software developers using AI completed work 19% slower but believed they were 20% faster.
Unidentified speaker (speaker_0) ·0:22
How to verify: Search for the specific study cited in the transcript (likely the GitHub Copilot study by Noy and Zhang, or similar) and verify the reported speed reduction and self-reported speed increase percentages in the original paper or press release. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
Customer service agents using AI increased productivity by 15%, with largest gains among less experienced workers.
Unidentified speaker (speaker_0) ·0:22
How to verify: Locate the study on customer service agents and AI productivity (e.g., Harvard Business Review or academic journal articles on AI in call centers) and verify the 15% productivity gain and the correlation with experience levels. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
An experiment involving 640 Kenyan entrepreneurs found no significant improvement in average business performance from AI access.
Unidentified speaker (speaker_0) ·0:22
How to verify: Search for the specific randomized controlled trial involving Kenyan entrepreneurs and AI (likely published in a journal like Science or Nature, or by a major development organization) and verify the sample size of 640 and the null result on business performance. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
The speaker launched an AI research agent on September 15 to monitor AI productivity studies.
Unidentified speaker (UNKNOWN) ·0:58
How to verify: Check the speaker's Substack homepage (exploringprosperity.com) or social media posts for announcements dated around September 15, 2026, regarding the launch of this research tool or inventory. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
The resulting inventory contains 82 records: 29 analytical research entries and 53 company accounts.
Unidentified speaker (UNKNOWN) ·1:18
How to verify: Access the inventory linked on the speaker's Substack homepage (exploringprosperity.com) and count the total entries, as well as the breakdown between research entries and company accounts. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
Erica Groshen, former Commissioner of the Bureau of Labor Statistics, stated there were more than 1,000 studies in 2026 examining AI's effects on labor.
Unidentified speaker (UNKNOWN) ·1:36
How to verify: Verify Erica Groshen's tenure at the BLS via the BLS website. Search academic databases (e.g., Google Scholar, SSRN) for studies on AI and labor published in 2026 to assess the volume of literature, though the specific '1,000' count is a claim best verified via the speaker's cited source or Groshen's own statements. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
A Stanford and MIT experiment found accountants using AI improved accuracy by 17.5 percentage points but did not significantly reduce completion time.
Unidentified speaker (speaker_0) ·2:16
How to verify: Search for joint Stanford-MIT research on AI and accounting (e.g., studies involving AI tools for financial transaction classification) and verify the specific 17.5 percentage point accuracy improvement and the lack of significant time reduction. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
A Boston Consulting Group (BCG) experiment found AI improved speed and quality within capabilities but deteriorated performance beyond them.
Unidentified speaker (speaker_0) ·2:36
How to verify: Search for the Boston Consulting Group study on AI and consultants (likely published in Harvard Business Review or BCG's own research portal) and verify the findings regarding performance within vs. beyond AI capabilities. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
Primary sourceFoundation's Envoy and Key OS releases allow users to add external entropy for seed creation.
Unidentified speaker (speaker_0) ·4:16
How to verify: Visit foundation.xyz and check the documentation or release notes for the Envoy and Key OS products to confirm the feature allowing users to add external entropy for seed generation. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
The speaker spent 27 years on Wall Street.
Unidentified speaker (speaker_0) ·4:35
How to verify: Check the speaker's biography on their Substack homepage (exploringprosperity.com) or professional profiles (LinkedIn) to verify the duration of their Wall Street career. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
Do your own research
- Verify the specific methodology of the Stanford and MIT experiment regarding accuracy vs. time metrics.
- Check the Bureau of Labor Statistics archives for the 1,000+ studies referenced by Erica Groshen.
- Review the Boston Consulting Group report on AI performance deterioration beyond capability boundaries.
- Examine the Kenyan entrepreneur study details to understand why weaker businesses performed worse.
- Analyze the developer overhead hypothesis for experienced programmers using AI tools.
- Look for counter-arguments to the claim that counting hours saved fails to capture economic value.
- Verify the claims made by Foundation regarding their Envoy and Key OS releases.
- Check independent reviews of AI tools used in the early 2025 developer experiment.
Full transcriptexpand
- 0:00
- Unidentified speaker (speaker_0): Is AI really making us more productive? I've been examining the research, and the findings are more contradictory and less conclusive than I expected. What does that mean for policymakers? Here's what I found: Faster, slower, or better. Why studies of AI productivity disagree, and what
- 0:20
- Unidentified speaker (speaker_0): that means for the people using it?
- 0:21
- Unidentified speaker (UNKNOWN): The
- 0:22
- Unidentified speaker (speaker_0): evidence on AI productivity is surprisingly contradictory. Artificial intelligence. made experienced software developers 19% slower completing their work, yet those same developers believed it had made them 20% faster. In another study, customer service agents using AI increased their productivity by 15%, with the
- 0:41
- Unidentified speaker (speaker_0): largest gains among the less experienced workers. And in an experiment involving 640 six hundred forty Kenyan entrepreneurs, access to AI. produced no significant improvement in average business performance. So what explains such different results?
- 0:58
- Unidentified speaker (UNKNOWN): On
- 0:58
- Unidentified speaker (speaker_0): september fifteenth, I launched an AI research agent to monitor empirical studies and company experiences, examining AI's impact on productivity. I wanted to understand how AI affects the speed and quality of work, employment, entrepreneurship. and the economics of businesses of different sizes.
- 1:18
- Unidentified speaker (UNKNOWN): The
- 1:18
- Unidentified speaker (speaker_0): agent had assembled 82 records, 29 analytical research entries, and 53 company accounts. The research includes experiments, observational studies, and surveys. Company accounts offer evidence of how AI is being deployed, Employed, although their claims are generally harder to verify.
- 1:36
- Unidentified speaker (UNKNOWN): In
- 1:36
- Unidentified speaker (speaker_0): early October, I interviewed Erica. Groshen, former Commissioner of the Bureau of Labor Statistics. She described more than 1,000 studies in 2026 examining AI's effects on labor, with widely varying findings. My smaller collection, which includes early research, is beginning to
- 1:56
- Unidentified speaker (speaker_0): reveal why drawing broad conclusions is so difficult. Three factors appear to explain. some of the divergence. What researchers measure, the suitability of AI for the task, and the judgment of the person using it. First, studies measure different outcomes. Some examine speed,
- 2:16
- Unidentified speaker (speaker_0): others accuracy, quality, sales, or profits. In a randomized experiment by researchers at Stanford and MIT, accountants using AI to classify financial transactions. improved their accuracy by 17.5 percentage points. But total completion time did not decline significantly. They produce more
- 2:36
- Unidentified speaker (speaker_0): accurate work without finishing faster. Counting hours saved won't tell us how much additional economic value better work creates. Second, AI's usefulness depends on the task in an experiment involving boss. Boston Consulting Group consultants, AI improved both speed and quality on assignments
- 2:55
- Unidentified speaker (speaker_0): within its capabilities, but performance deteriorated when consultants relied on it for problems beyond those capabilities. Third, the user's judgment influences the outcome. Although the relationship is complicated, novice customer service agents gained the most, apparently benefiting from knowledge. embedded in the
- 3:15
- Unidentified speaker (speaker_0): AI system.
- 3:16
- Unidentified speaker (UNKNOWN): In
- 3:17
- Unidentified speaker (speaker_0): the Kenyan experiment, initially weaker businesses performed worse with AI access, while stronger businesses showed signs of improvement.
- 3:25
- Unidentified speaker (UNKNOWN): The
- 3:25
- Unidentified speaker (speaker_0): developer experiment offers another perspective. Conducted with early 2025 AI tools on experienced programmers' own repositories, it found that AI assistance assistants increased increase completion time. The researchers suspect that the overhead of using AI may be particularly costly when developers already
- 3:45
- Unidentified speaker (speaker_0): know their code bases exceptionally well. Access to powerful technology alone does not ensure better performance. Perhaps AI helps most when it supplies knowledge the user lacks, and the user can recognize which recommendations deserve trust. The study suggests this possibility without
- 4:05
- Unidentified speaker (speaker_0): establishing a general rule.
- 4:16
- Unidentified speaker (speaker_0): Foundation's latest Envoy and Key OS releases now let users add their own entropy when creating a seed, introducing randomness from outside the device itself. It's a feature Foundation customers have been asking for, and another example of putting control and more ability to verify security into the hands of the
- 4:35
- Unidentified speaker (speaker_0): user. Visit them at foundation.xyz. I spent 27 years on Wall Street, working alongside exceptionally intelligent people making consequential decisions. Yet, even in that environment, we were constrained by how much information we could gather, how effectively we could analyze. it, and how clearly we could communicate our conclusions. I contributed my
- 4:55
- Unidentified speaker (speaker_0): share of lengthy, confusing explanations, emails, and powerpoints.
- 5:00
- Unidentified speaker (UNKNOWN): My
- 5:00
- Unidentified speaker (speaker_0): experience with AI makes me optimistic about its economic implications. It can process extraordinary quantities of information, identify relationships that might otherwise escape notice, and communicate findings with remarkable clarity and consistency. Of course, its polished explanations can also conceal errors.
- 5:20
- Unidentified speaker (speaker_0): Better information and analysis could reduce costly mistakes, uncover opportunities, and improve the allocation of capital. Faster decisions could accelerate subsequent decisions, allowing improvements to compound throughout organizations and supply chains. These benefits may take years to appear. in conventional economic
- 5:40
- Unidentified speaker (speaker_0): statistics. I see some of this in my own work. AI allows me to investigate questions much more thoroughly, test hypotheses, and reorganize data. Increasingly, I can work with entire ideas rather than labor over individual sentences.
- 5:56
- Unidentified speaker (UNKNOWN): But
- 5:56
- Unidentified speaker (speaker_0): the developers in the opening experiment also believed AI was helping them. How do I know my own experience is
- 6:02
- Unidentified speaker (UNKNOWN): not a good idea?
- 6:03
- Unidentified speaker (speaker_0): different?
- 6:03
- Unidentified speaker (UNKNOWN): I
- 6:04
- Unidentified speaker (speaker_0): don't claim that AI saves me time. I spend considerable effort challenging its conclusions, checking evidence, and correcting mistakes. I believe the finished work expresses my ideas far more clearly and effectively than I could previously manage, although that judgment is subjective too. Quality depends heavily on the effort I put into
- 6:24
- Unidentified speaker (UNKNOWN): it.
- 6:24
- Unidentified speaker (speaker_0): Even my research agent illustrates the need. for scrutiny, its initial inventory mixed research studies with company accounts, requiring closer audit to distinguish the evidence from the anecdotes. My collection remains limited, the technology is changing rapidly, and many experiments examine narrow tasks over short periods. The evidence is
- 6:44
- Unidentified speaker (speaker_0): useful, but the broader economic consequences remain uncertain. This uncertainty worries me because the reason research offers plenty of ammunition for both AI optimists and skeptics. Each find credible studies supporting what they already believe. Policymakers face the same temptation, with potentially greater consequences.
- 7:04
- Unidentified speaker (speaker_0): Policies based on premature conclusions about AI's effects on jobs, businesses, or economic growth could produce unintended consequences, whether those conclusions are overly optimistic. or pessimistic. My own experience suggests that AI is likely to substantially improve the quality and speed of decisions
- 7:24
- Unidentified speaker (speaker_0): throughout the economy. The studies suggest that those benefits will be uneven. Understanding AI's effects requires measuring the value of better work, identifying the tasks where AI helps, and learning how people develop the judgment to use. it effectively. How broadly people develop that judgment may determine
- 7:43
- Unidentified speaker (speaker_0): much of AI's economic impact.
- 7:46
- Unidentified speaker (UNKNOWN): You
- 7:46
- Unidentified speaker (speaker_0): can find the full inventory of these AI studies on my Substack homepage at exploring prosperity dot com.
- 7:55
- Unidentified speaker (UNKNOWN): I
- 7:55
- Unidentified speaker (speaker_0): believe the next generation can live better than any before it.
- 7:59
- Unidentified speaker (UNKNOWN): But
- 7:59
- Unidentified speaker (speaker_0): only if we stay curious, keep learning, and refuse to accept
- 8:02
- Unidentified speaker (UNKNOWN): the.
- 8:03
- Unidentified speaker (speaker_0): decline as inevitable. If that message resonates with you, please hit subscribe
- 8:09
- Unidentified speaker (UNKNOWN): and
- 8:10
- Unidentified speaker (speaker_0): share this video with someone who's curious as
- 8:13
- Unidentified speaker (UNKNOWN): well. And
- 8:14
- Unidentified speaker (speaker_0): if you'd like to go deeper, you'll find essays and research on my Substack. The link is below. Until next time, keep exploring.