All Else Equal: Making Better Decisions

David Romer explains why NFL teams delayed adopting aggressive fourth-down analytics due to behavioral biases

37 min Listen to the original Episode page
NFL AnalyticsBehavioral EconomicsDynamic ProgrammingOrganizational CultureDecision MakingSports StrategySuper Bowl 52

Summary

Hosts Jules Van Binsbergen and Jonathan Burke discuss David Romer's research on suboptimal decision-making in the NFL, specifically regarding fourth-down choices. Romer, a Berkeley economics professor, explains that his dynamic programming analysis showed teams were consistently punting when going for it was mathematically superior. Despite the clear evidence, adoption was delayed for years due to behavioral biases like risk aversion and psychological costs, rather than just agency problems. The shift eventually occurred after competitive pressure, the rise of analytics culture, and a high-profile success by the Philadelphia Eagles in Super Bowl 52. Romer notes that experts often resist data-driven insights due to a loss of control or status, not monetary cost.

Episode sections

  1. 0:05 – 5:46

    Introduction and NFL Rules

    Jules Van Binsbergen and Jonathan Burke introduce the topic of suboptimal decision-making in the NFL, using David Romer's research as a case study. Jules explains the basic rules of American football, including downs, scoring, and the fourth-down decision between punting, kicking a field goal, or going for it.

  2. 5:46 – 8:32

    Romer's Dynamic Programming Analysis

    Jonathan Burke explains that David Romer used dynamic programming to calculate expected scoring probabilities from any yard line. The analysis showed that teams were consistently making suboptimal decisions, particularly on fourth down, by being too conservative compared to the mathematically optimal strategy.

  3. 8:32 – 11:29

    Specific Suboptimal Scenarios

    The speakers highlight two obvious cases where teams should go for it: near the opponent's end zone (to avoid giving the ball back in a dangerous position) and in the middle of the field (where punting yields minimal field position advantage). Despite the clear logic, teams largely ignored these insights for years.

  4. 11:29 – 12:56

    Coach Incentives and Agency Problems

    The discussion turns to why teams didn't optimize immediately. Jonathan Burke suggests an agency problem where coaches prioritize their reputation and avoid looking foolish to the audience over maximizing team winning probability. David Romer expresses skepticism about this, noting that external pressure existed for decades.

  5. 12:56 – 19:34

    David Romer's Introduction and Initial Predictions

    David Romer is introduced as a Berkeley economics professor. He reveals that his two initial predictions were wrong: teams became aware of his paper within months via journalists, but they did not change their behavior immediately despite the compelling evidence.

  6. 19:34 – 22:45

    Reasons for the Delayed Adoption of Analytics

    Romer attributes the eventual change in strategy to three factors: long-term competitive pressure, the general rise of analytics across sports, and a specific precipitating event: the Philadelphia Eagles' aggressive fourth-down decisions in Super Bowl 52 (2018), which led to a victory.

  7. 22:45 – 25:55

    Behavioral Biases vs. Agency Theory

    Romer leans towards the explanation that strong behavioral biases (risk aversion) blocked optimization for decades, rather than an agency problem where coaches knew the right answer but feared backlash. He notes that coaches are competitive and smart, making the 'fear of looking foolish' explanation less plausible.

  8. 25:55 – 28:57

    Implications for AI and Business Optimization

    The speakers discuss how AI could help overcome behavioral biases in well-defined decision-making contexts. Romer concludes that if firms fail to optimize in a simple, high-stakes environment like the NFL, they likely fail even more in complex, ambiguous business decisions.

  9. 28:57 – 29:58

    Behavioral Biases and Organizational Rigidity

    David Romer recounts a high school coach's anecdote about parents misapplying life-or-death stakes from war to football decisions, illustrating deep-seated behavioral biases. He questions why NFL organizations remained resistant to analytical insights despite decades of evidence suggesting they should be more aggressive on fourth downs.

  10. 29:58 – 31:39

    MBA Education and Organizational Culture

    The host suggests that business schools fail to teach the importance of analytical insight and openness to error. Romer agrees, proposing a research question to identify which organizational traits—such as analytics, structure, or culture—enable nimbleness and efficient optimization.

  11. 31:39 – 32:21

    The 'Moneyball' Effect and Marketing Analytics

    The discussion highlights how the movie 'Moneyball' changed the cultural perception of analytics in sports, making it 'cool' and motivating younger generations. Romer suggests this demonstrates that the tools are valuable but may have been poorly marketed to traditionalists.

  12. 32:21 – 33:30

    Editorial Influence and Academic Focus

    Romer reveals that Stephen Levitt, editor at the University of Chicago, pushed him to remove frivolous elements from his paper to focus on whether competitive forces cause firms to optimize. This editorial guidance helped sharpen the paper's core argument.

  13. 33:30 – 35:04

    Psychological Costs of Analytics vs. Monetary Costs

    Romer argues that the reluctance to adopt dynamic programming is not due to monetary cost, which has dropped to near zero, but rather psychological cost. Experts resist relinquishing control and admitting that intuition may be inferior to computer models.

  14. 35:04 – 37:01

    NFL Teams' Reaction to the Paper

    Romer notes that NFL teams did not initially adopt his suggestions. Instead, their reaction was to find flaws in the analysis to justify ignoring it, rather than rigorously testing whether those flaws overturned the results. He states that time has since proven his analysis correct.

Claims worth verifying

David Romer used dynamic programming to calculate the probability of expected scoring from any yard line and any down.

Unidentified speaker ·5:46

How to verify: Check David Romer's academic paper 'Do Firms Maximize? Evidence from Professional Football' (2006) for the methodology section describing dynamic programming.

If a team is tackled in its own end zone, the opposing team gets two points and the ball (a safety).

Unidentified speaker ·6:22

How to verify: Verify against the official NFL Rulebook, specifically the section on scoring plays and safeties.

In the early 2000s, NFL teams almost always punted or kicked a field goal on fourth down, ignoring Romer's findings.

Unidentified speaker ·8:00

How to verify: Review historical NFL play-by-play data from the early 2000s or cite Romer's 2006 paper which documents this pre-adoption behavior.

The probability of converting a fourth-and-one is approximately 65-75%, not 90% as commonly assumed.

Unidentified speaker ·17:47

How to verify: Check NFL historical conversion rate statistics for fourth-and-one situations, available on sites like Pro-Football-Reference or NFL.com.

The probability of converting a fourth-and-ten is approximately 30%, not 5% as commonly assumed.

Unidentified speaker ·17:55

How to verify: Check NFL historical conversion rate statistics for fourth-and-ten situations, available on sites like Pro-Football-Reference or NFL.com.

There was essentially no change in NFL fourth-down aggression from the time of Romer's paper until 2017.

Unidentified speaker ·20:19

How to verify: Analyze NFL fourth-down attempt rates from 2006 to 2017 using historical play-by-play data.

The Philadelphia Eagles went for it twice on fourth downs in unconventional situations in Super Bowl 52 in February 2018.

Unidentified speaker ·20:37

How to verify: Review the official box score and play-by-play for Super Bowl LII (Eagles vs. Patriots, Feb 4, 2018).

David Romer is the Hermann Royer Professor of Political Economy at the University of California, Berkeley.

Unidentified speaker ·12:59

How to verify: Check the UC Berkeley Department of Economics faculty directory or Romer's official academic profile.

A 1971 paper on fourth-down decisions was published in Operations Research, with one author being an NFL backup quarterback.

Unidentified speaker ·16:07

How to verify: Search the Operations Research journal archives for 1971 publications on football strategy and verify the authors' affiliations.

Behavioral biases cause teams in other sports to be too conservative, such as being slow to take three-point shots in basketball.

Unidentified speaker ·28:23

How to verify: Review sports analytics literature on the adoption of the three-point shot in the NBA and its impact on team efficiency.

David Romer received an email from a high school coach who faced parental pushback for going for it on fourth downs, with parents claiming it was a 'life or death' situation.

Unidentified speaker ·28:57

How to verify: Search for David Romer's interviews or articles from 2026-11-05 or earlier where he discusses the reception of his paper and includes this specific anecdote about the Korean War comparison.

Stephen Levitt, an economist at the University of Chicago, was the journal editor who handled David Romer's paper and advised him to remove frivolous content to focus on whether competitive forces cause firms to optimize.

Unidentified speaker ·32:36

How to verify: Check the publication history and acknowledgments of David Romer's 2004 paper 'Should Football Teams Go for It on Fourth Down?' in the Journal of Political Economy, specifically looking for Stephen Levitt as the handling editor.

The cost of performing dynamic programming analysis has dropped to near zero due to new technologies.

Unidentified speaker ·34:20

How to verify: Review technical literature on computational complexity and software tools for dynamic programming in sports analytics or business optimization from 2026 to confirm current computational costs versus historical costs.

NFL teams initially reacted to Romer's paper by identifying minor omissions to justify ignoring the analysis, rather than rigorously testing if those omissions overturned the results.

Unidentified speaker ·35:04

How to verify: Review post-publication commentary, interviews with NFL coaches or analysts from 2004-2010, or Romer's own subsequent writings on the adoption of his fourth-down strategy to verify the described reaction pattern.

There is now a consensus in the NFL that teams should be more aggressive on fourth downs, validating Romer's original paper.

Unidentified speaker ·35:22

How to verify: Analyze NFL fourth-down conversion rates and decision-making data from 2020-2026 compared to pre-2004 data, and review NFL coaching manuals or widely accepted strategic guides to confirm the consensus.

The movie 'Moneyball' changed the cultural perception of analytics in sports, making it 'cool' and motivating young people.

Unidentified speaker ·31:39

How to verify: Review media coverage and cultural commentary surrounding the release of the movie 'Moneyball' (2011) and its impact on sports analytics adoption in subsequent years.

David Romer recently attended his 50th high school reunion.

Unidentified speaker ·33:11

How to verify: Check David Romer's birth year (approx. 1960s) and recent biographical updates or social media posts from late 2026 to confirm the attendance of his 50th reunion.

Do your own research

  • Verify David Romer's original paper 'Do Firms Optimize? Evidence from Professional Football' to review the dynamic programming methodology and expected scoring probability calculations.
  • Check NFL game logs from Super Bowl 52 (2018) to confirm the Philadelphia Eagles' fourth-down decision sequences and their impact on the game outcome.
  • Review academic literature on 'agency problems' in sports management to compare Romer's behavioral bias explanation against traditional reputation-based theories.
  • Search for post-2018 NFL analytics studies to verify if teams have sustained the aggressive fourth-down strategies identified by Romer.
  • Investigate the role of the University of Chicago editorial team, specifically Stephen Levitt, in shaping the scope and arguments of Romer's published research.
  • Look for data on the cost of analytics software in sports organizations to verify claims that monetary costs are negligible compared to psychological barriers.
Full transcriptexpand
0:05
Unidentified speaker: Welcome to the Lauder Institute at the University of Pennsylvania.
0:09
Unidentified speaker: I'm Jules Van Binsbergen, Director of the Institute and a finance professor at the Wharton School.
0:14
Unidentified speaker: And I'm Jonathan Burke, a finance professor at the Graduate School of Business at Stanford University.
0:22
Unidentified speaker: This is the All Else Equal Podcast.
0:31
Unidentified speaker: Welcome back, everybody.
0:33
Unidentified speaker: Today we're going to look at something that has always fascinated me, which is in a competitive market, most economists would say that the competition is so strong that if firms don't correctly optimize, they get weeded out.
0:49
Unidentified speaker: And so you would expect close to optimal behavior of firms.
0:55
Unidentified speaker: And I have to say I buy into this.
0:57
Unidentified speaker: But today we're going to look at a case where it's, I think, very clear where you have a competitive market and firms are not optimizing.
1:05
Unidentified speaker: And that is the NFL.
1:07
Unidentified speaker: So before we get into it, let me first turn it over to Jules, because not every listener understands football.
1:15
Unidentified speaker: And Jules, I think, is one of them.
1:19
Unidentified speaker: You're so right about that, Jonathan.
1:21
Unidentified speaker: So just for our listeners in Europe, the NFL stands for the National Football League.
1:25
Unidentified speaker: And of course, in the United States, football means something entirely different than it does in Europe.
1:30
Unidentified speaker: And so we're talking here about the sport that's called American football.
1:34
Unidentified speaker: And so the reason why, indeed, this is an interesting topic to study is that doing an experiment in a firm where you really can observe all the possible decisions that can be made and where you have a good sense of what the competition really looks like is actually pretty hard.
1:54
Unidentified speaker: And so, therefore, and our guest today, David Romer, wrote an academic paper that we are going to be discussing together.
2:00
Unidentified speaker: He essentially found a very nice competitive setting where we can really rationally model what a competitive player would or should do.
2:10
Unidentified speaker: And then we can compare that with the actual behavior that we are observing.
2:15
Unidentified speaker: And indeed, for our listeners to understand what we're going to be discussing next, it is probably a good idea to lay out the basic rules of football.
2:22
Unidentified speaker: So let me try to do that as in simple possible terms as I can, Jonathan, which is indeed not an easy feat.
2:28
Unidentified speaker: Now, for our listeners that are intimately familiar with football, we suggest that they skip the next three minutes and go straight to the five minute and 30-second mark.
2:38
Unidentified speaker: For those listeners that are not that familiar with football, I hope that the next explanation will be useful.
2:44
Unidentified speaker: So you have two teams, we have a 100-yard field, and you score points by carrying or catching the ball into what's called the end zone, which is all the way on the other side of where you start, so on the opponent's side.
2:56
Unidentified speaker: And if you get it there, then you can score in a number of ways.
3:01
Unidentified speaker: One way, which is the most spectacular way, if you will, is what's called a touchdown, where you touch the ball down on the other side, which is worth six points.
3:10
Unidentified speaker: And then after that, you get an automatic extra point due to kicking the ball, which is so effectively you get seven points for a touchdown.
3:18
Unidentified speaker: Just for our European audience, think of American football, of course, as being much closer to rugby than it is to soccer.
3:25
Unidentified speaker: Now, the team with the ball gets four attempts to try to progress on the field 10 yards towards the opponent.
3:35
Unidentified speaker: And each of those attempts are also called downs.
3:39
Unidentified speaker: And so it's important that in four attempts you make that 10-yard progress.
3:44
Unidentified speaker: If you keep on succeeding, then you get four new downs.
3:48
Unidentified speaker: And so eventually, if you keep proceeding 10 yards every time within your four attempts, then you reach the end zone the other side of the field.
3:56
Unidentified speaker: But if you fail and the other team takes the ball exactly where you were stopped, so the other team will take over the ball and will start from that point where you ended up in your attempts.
4:09
Unidentified speaker: So every possession is a sequence of, I would say, small bets.
4:14
Unidentified speaker: And the question that we're going to be really talking about is that what is the best thing to do on the fourth attempt?
4:20
Unidentified speaker: So you've already failed three times to advance 10 yards.
4:23
Unidentified speaker: Now you're on your fourth attempt.
4:25
Unidentified speaker: And what should you do with the last one?
4:27
Unidentified speaker: Okay, Jules, there's an important thing you left out.
4:30
Unidentified speaker: At any point a team can elect to kick the ball through the goal posts.
4:36
Unidentified speaker: And if they manage to do that, they get three points.
4:40
Unidentified speaker: So there is a part of the field, usually from the 40-yard line onwards, where teams have a chance of kicking the ball through the goal posts.
4:49
Unidentified speaker: And so when the fourth down decision comes, they can go for it on fourth down, and then if they don't make it, they lose the ball and the other team gets the ball.
4:58
Unidentified speaker: Or they can try to kick the ball through the goal posts, and if they make it, they get three points, then the other team gets the ball.
5:06
Unidentified speaker: So now the crucial thing to understand is it's automatic to at approximately 20 yards.
5:13
Unidentified speaker: If you kick it, you'll get three points for sure.
5:16
Unidentified speaker: But between 20 and 40 or 50 yards, it becomes less and less likely.
5:21
Unidentified speaker: And I would say at about 45 yards, it's impossible to get to three points.
5:25
Unidentified speaker: That's a crucial part of this thing.
5:27
Unidentified speaker: And the last thing I would add for somebody who's never seen the game of football is it's very much a possession game.
5:33
Unidentified speaker: When one team has the ball, it is extremely hard for the other team to get the ball so that who has possession is a big advantage.
5:42
Unidentified speaker: I think that's enough to understand the rules.
5:46
Unidentified speaker: So what our guest David Roman did was he used a technique which is ubiquitous in optimization and in economics called dynamic programming to calculate the probability from any yard line and any down of expected scoring.
6:03
Unidentified speaker: And the first thing that is kind of surprising about the result of that calculation is that for yard lines that are very close to your own end zone.
6:12
Unidentified speaker: So let's say they're one yard line, it's actually negative.
6:16
Unidentified speaker: The likelihood is the other team will score.
6:18
Unidentified speaker: And I think the reason for that is, first of all, there's a concept of a safety.
6:22
Unidentified speaker: If they tackle you in your own end zone, the other team gets two points and the ball.
6:27
Unidentified speaker: But also, if the other team gets the ball close to your own end zone, then the likelihood of scoring is very high.
6:36
Unidentified speaker: And so I think for those two reasons, on the one yard line, you actually have a negative expected amounts of points.
6:43
Unidentified speaker: But of course, that goes up.
6:45
Unidentified speaker: And when you're on your own 99-yard line, you have a very high expected number of points.
6:51
Unidentified speaker: What David was wondering about is the decision on fourth down.
6:55
Unidentified speaker: So the decision on fourth down is you could go for it.
6:59
Unidentified speaker: If you miss, the other team takes over where you are.
7:02
Unidentified speaker: If you make it, you get four more downs.
7:05
Unidentified speaker: Or you could try to kick the ball through the posts.
7:08
Unidentified speaker: And then the last point was you could also give up the ball to the other team voluntarily by what's called punting.
7:16
Unidentified speaker: But when you punt, you kick the ball way down the field.
7:20
Unidentified speaker: And so the other team gets the ball, but on their side of the field.
7:24
Unidentified speaker: And when you kick the ball, Jonathan, you generally kick at 30 to 40 yards or something.
7:29
Unidentified speaker: So you set them back by quite a bit, but definitely not all the way.
7:33
Unidentified speaker: Well, depending on where you are at that point in time, you don't set them all the way back to the first yard.
7:38
Unidentified speaker: Basically, any well-managed team is not going to find themselves inside the 20-yard line.
7:43
Unidentified speaker: So they will always take over at the 20, and usually in football, somewhere between the 20 and the 40-yard line, when they take over.
7:51
Unidentified speaker: Okay, so that's the decision people are facing.
7:54
Unidentified speaker: When should the team uh go for fourth down and when should they punt or kick a field goal?
8:00
Unidentified speaker: And in football, uh, when this paper was written in the early 2000s, teams almost always either punted or kicked a field goal.
8:10
Unidentified speaker: What David showed was that this decision was clearly suboptimal.
8:15
Unidentified speaker: The paper he wrote is really an outstanding paper.
8:19
Unidentified speaker: He thought about every possible reason why his result could be overturned and looked at it and did not find any good reason why the analysis wasn't correct.
8:32
Unidentified speaker: Now it's a complicated analysis, but there are two cases where it's entirely obvious.
8:38
Unidentified speaker: The first case is if you've got one yard to go to get a touchdown and it's fourth down, he does a calculation and it's in expectation, it's about the same.
8:47
Unidentified speaker: If you kick the field goal, there you'll get three for sure.
8:50
Unidentified speaker: If instead you go for the touchdown, you get seven, the probability of making it is about 40% or something, meaning in expectation, it's three points.
9:00
Unidentified speaker: So from an expectation standpoint, at that point you're indifferent.
9:04
Unidentified speaker: But David's point is that if you fail on fourth down, the other team takes over on the one yard line.
9:12
Unidentified speaker: And that, as I've told you, is a negative number for that.
9:16
Unidentified speaker: Whereas if you kick three points, the other team takes over on the 20, 30, or 40 yard line.
9:23
Unidentified speaker: And so you're indifferent about the number of points being scored, but you certainly aren't indifferent in where they take over.
9:29
Unidentified speaker: And because of that, it doesn't make any sense to kick a field goal in that position.
9:36
Unidentified speaker: And even today, we don't see teams do that.
9:40
Unidentified speaker: So, Jonathan, just as a clarification, also for our European listeners, right?
9:44
Unidentified speaker: When we say 40 or 30 yard line, does that mean that from your side you've gone 30 yards, or does it mean that you have 30 yards to go?
9:53
Unidentified speaker: You say you're on your own 30 yard line, it means you have 70 yards to go.
9:57
Unidentified speaker: If you say you're on your opponent's 30-yard line, then you have 30 yards to go.
10:02
Unidentified speaker: Yes.
10:03
Unidentified speaker: If you're all the way close to the end zone and you try to fourth down and you fail, the opposing team has to essentially 100 yards to cover, which is way more than an average play, than an average kickoff.
10:15
Unidentified speaker: And so that is a big advantage.
10:17
Unidentified speaker: And since you're indifferent on the number of points, that swings it.
10:21
Unidentified speaker: That swings it.
10:22
Unidentified speaker: So that means that you should always try it.
10:25
Unidentified speaker: When you're really close to the end, you should go for the play, you should not punt, and you should not kick the field goal.
10:30
Unidentified speaker: That's one of the obvious cases.
10:32
Unidentified speaker: And the other obvious case is if you've got fourth down in the middle of the field.
10:37
Unidentified speaker: So let's say you're fourth down at the 50 yard line.
10:40
Unidentified speaker: It's too far to kick a field goal.
10:43
Unidentified speaker: But if you punt, I told you they have a kind of an option to start on the 20-yard line.
10:48
Unidentified speaker: So at most, you can get something like 20 or 30 yards with the punt.
10:52
Unidentified speaker: Yeah.
10:52
Unidentified speaker: Okay.
10:53
Unidentified speaker: But if you make it on that fourth down, you are on the 50 yard line.
10:58
Unidentified speaker: And so that's another case where you should always go for it.
11:02
Unidentified speaker: And what happened when they wrote the paper is there was no change in strategy.
11:07
Unidentified speaker: But over the years, that strategy has changed.
11:11
Unidentified speaker: Today in football, uh, teams generally go for it in that second situation.
11:17
Unidentified speaker: And so the question on the table is why in such a competitive world with a strategy that's obviously better, do teams not adopt the strategy?
11:29
Unidentified speaker: A number of questions come to mind immediately.
11:31
Unidentified speaker: The first one is when we analyze organizations, the first thing that you and I always discuss, and we've discussed many times on the podcast, is what are the incentives of all the different players that are at stake here?
11:43
Unidentified speaker: And so there's a coach, the coach makes a decision, right?
11:46
Unidentified speaker: Is that fair?
11:47
Unidentified speaker: Is it to say that the coach is really the one who makes the decision on what happens?
11:51
Unidentified speaker: Yeah.
11:51
Unidentified speaker: So therefore, obviously the incentives of the coach could matter.
11:55
Unidentified speaker: That's one explanation.
11:56
Unidentified speaker: I actually think that there's a deeper explanation, and we'll see what David thinks of this.
12:00
Unidentified speaker: But don't you think that the reputation of the coach also depends on what the audience thinks he's supposed to be doing?
12:08
Unidentified speaker: And that foregoing three points for sure by doing that play, and then the play fails, you look like such a fool that you want to avoid that scenario under all costs, despite the fact that, in a dynamic programming sense, for the team and for the winning of the team, you would do better.
12:27
Unidentified speaker: Yeah, Jules, I think that's certainly a possible explanation.
12:30
Unidentified speaker: It's stunning to me.
12:31
Unidentified speaker: The paper got a lot of press when it came out in the early 2000s, and the football teams knew about the paper, and they essentially ignored the paper.
12:42
Unidentified speaker: With the rise of analytics in general, they began to pay some attention.
12:46
Unidentified speaker: But as I said, it's still not fully optimal.
12:49
Unidentified speaker: They still do not go enough on fourth down.
12:52
Unidentified speaker: I think, Jules, this is the right time for us to introduce our guest today.
12:56
Unidentified speaker: Our guest today is David Rohmer.
12:59
Unidentified speaker: David is the Hermann Royer Professor of Political Economy at the University of California in Berkeley and the author of the standard textbook and graduate macroeconomics, as well as many influential papers in the area of what's called New Keynesian economics.
13:12
Unidentified speaker: The Nobel Prize was just given for that field.
13:15
Unidentified speaker: I think that in macro, David is very famous, but I think that in the world, David is probably the most famous for this paper, this paper that ended up changing the field of football.
13:26
Unidentified speaker: It took a long time, but eventually I think he had major influence in this field.
13:30
Unidentified speaker: And this was probably not a field that David expected to have a large influence in, but I think it's a phenomenal study of optimal decision making under uncertainty.
13:40
Unidentified speaker: And I think a place where we can learn a lot about how firms operate too.
13:44
Unidentified speaker: David, welcome to the show.
13:46
Unidentified speaker: Thank you for having me.
13:48
Unidentified speaker: It's great to have you, David.
13:49
Unidentified speaker: So, David, I'm maybe South African, but I'm a somewhat of a football fan.
13:54
Unidentified speaker: I understand the rules of football well enough to understand the basic insights in your paper and how incontrovertible they are.
14:01
Unidentified speaker: When you wrote this paper, what did you think was going to happen to in football?
14:05
Unidentified speaker: And how did you expect the NFL teams to react to your findings?
14:10
Unidentified speaker: I had two predictions.
14:11
Unidentified speaker: Both of them were completely wrong.
14:13
Unidentified speaker: The first was that it would be hard to get teams to be aware of my research.
14:18
Unidentified speaker: It's not part of the job description and work in the National Football League to read economics journals or working papers.
14:24
Unidentified speaker: Within months of when my paper came out, every team knew about it.
14:28
Unidentified speaker: And the channel I'd missed was journalists who were always looking for something new to write about, partly, you know, to drive readership and partly out of boredom.
14:36
Unidentified speaker: And so they all found my paper and talked to coaches and so on.
14:42
Unidentified speaker: My second prediction was once they knew about it, of course they would change their behavior because they want to win games.
14:48
Unidentified speaker: And I thought the evidence was compelling, and that did not happen.
14:53
Unidentified speaker: You know, this is the question that I think is the interesting question for our listeners and the decision-making podcast.
14:59
Unidentified speaker: You have a very competitive market, you know, seeming incontrovertible evidence that the current strategy can be improved upon.
15:09
Unidentified speaker: And yet, at least immediately, you just saw no change in people's strategy.
15:14
Unidentified speaker: I think there's a prior question, which is maybe more important and more interesting, which was why before my paper weren't they doing this, right?
15:22
Unidentified speaker: The arguments that competition, competitive pressures, drive firms to efficiency, they're so powerful, they're so logical.
15:31
Unidentified speaker: If you're not competitive, if you're a manager of a firm doing a bad job, you'd be fired.
15:36
Unidentified speaker: If the firm continues to do a bad job, it'll either be outcompeted or be taken over by people who can do it better.
15:44
Unidentified speaker: And that wasn't happening.
15:45
Unidentified speaker: If I had, you know, uncovered some loophole in the rules that no one ever thought of.
15:49
Unidentified speaker: Well, sure, that might have been something new.
15:51
Unidentified speaker: But you have to make a decision on fourth downs.
15:54
Unidentified speaker: People didn't always do exactly the same thing.
15:58
Unidentified speaker: There was some variation.
15:59
Unidentified speaker: People knew it was a decision.
16:01
Unidentified speaker: There have been people making this argument back.
16:04
Unidentified speaker: The first paper I found was from 1971.
16:07
Unidentified speaker: One of the authors was the backup quarterback of an NFL team.
16:11
Unidentified speaker: It was published in Operations Research.
16:13
Unidentified speaker: It was written up, I think, in Sports Illustrated.
16:16
Unidentified speaker: So our argument is not competition leads to efficiency if an egghead professor points out the right strategy.
16:24
Unidentified speaker: Our argument is supposed to be competition leads to efficiency, period.
16:27
Unidentified speaker: And that didn't happen.
16:28
Unidentified speaker: That wasn't happening even before I wrote the paper.
16:32
Unidentified speaker: And then it sort of adds to the puzzle that, you know, when I wrote my paper, which got a lot of attention, I think it had stronger evidence than what came before.
16:40
Unidentified speaker: It did get a lot of attention.
16:42
Unidentified speaker: I think that's a follow-up question.
16:43
Unidentified speaker: Why wasn't that enough to push it up?
16:45
Unidentified speaker: But I do think the prior question is the bigger one.
16:48
Unidentified speaker: So, David, if we take a step back, how did you get the idea to write the paper?
16:53
Unidentified speaker: Was it already the case that you were watching games and every time you saw it as happening, you were starting to get increasingly frustrated?
16:59
Unidentified speaker: Or was it that you thought, let's see what comes out of this if I do the dynamic programming?
17:04
Unidentified speaker: Because only after you've computed the probabilities and all the different plays can you really make the case really strongly, right?
17:11
Unidentified speaker: So did you already know before you started that this was pretty much going to be the outcome?
17:16
Unidentified speaker: Or was what you found quite surprising?
17:19
Unidentified speaker: It was largely what I knew.
17:21
Unidentified speaker: There are cases that I walked through some of them in the paper where you don't need the dynamic programming, you don't need the math.
17:26
Unidentified speaker: You can do fairly simple examples, and it's compelling based on that.
17:32
Unidentified speaker: So I was pretty sure what I would find.
17:35
Unidentified speaker: The one thing I didn't know, I thought that if it's very short, it was fourth and one, fourth and two, the probability of making it would be very high, and fourth and 10, it would be very low.
17:47
Unidentified speaker: It turns out that's wrong.
17:49
Unidentified speaker: Fourth and one, I don't remember if it's 70% or 75 or 65, but it's not 90.
17:55
Unidentified speaker: And fourth and 10, it's not 5%, it's 30% or something like that.
18:00
Unidentified speaker: So particularly in the heart of the field where neither punting nor trying to feel gold are very attractive, one thing that comes out is actually you should go for it with quite substantial distances to go.
18:10
Unidentified speaker: That was unexpected to me.
18:12
Unidentified speaker: But the overall pattern was not.
18:15
Unidentified speaker: So, David, in the subsequent years, and correct me if I'm wrong, I think there are two obvious points you can make without dynamic programming, without really understanding dynamic programming.
18:27
Unidentified speaker: One was what to do in what you call the dead zone in the middle of the field, and what to do with fourth and one.
18:33
Unidentified speaker: Now, correct me if I'm wrong.
18:35
Unidentified speaker: The dead zone, the strategy did change, right?
18:38
Unidentified speaker: Over the last 20 years.
18:40
Unidentified speaker: Strategies have changed a lot over the last 20 years.
18:43
Unidentified speaker: So you just look at the fraction of times on fourth and one, fourth and two over the whole field that teams go for it.
18:50
Unidentified speaker: It's much higher than before.
18:53
Unidentified speaker: My impression, I haven't redone the analysis, my impression is they're not as aggressive as they should be, particularly in their own territory, when you can sort of it's very palpable that if you didn't make it, the other team would be in great field position.
19:07
Unidentified speaker: But there's been a very big change.
19:08
Unidentified speaker: And it's really been the last decade that that's happened.
19:12
Unidentified speaker: So a dual question.
19:14
Unidentified speaker: One is what do you think eventually caused the teams to adopt a better strategy, given that they hadn't for so many years?
19:23
Unidentified speaker: And what about the fact that you very rarely see them going in, you know, fourth term goal, which is one of the other points of your paper, that they should go more often there?
19:34
Unidentified speaker: I think in terms of what caused the change, I think three things.
19:39
Unidentified speaker: One is the underlying forces that we sort of started with competition should produce efficiency.
19:46
Unidentified speaker: You wait long enough, maybe you will get it without anything else.
19:50
Unidentified speaker: The second is the general rise of analytics.
19:52
Unidentified speaker: So across all sports, including football, it and the idea that they should look at data, be analytical.
19:58
Unidentified speaker: Every team, I think, now has an analytics, not just person, but staff.
20:02
Unidentified speaker: And you see that obviously in baseball, but you see it in basketball.
20:05
Unidentified speaker: I think you see it in soccer and you know, just every sport.
20:08
Unidentified speaker: And then the third, which is kind of fun, is there was a precipitating event, I think, which is if you look at the data from before I wrote my paper till 2017, there's essentially no change.
20:19
Unidentified speaker: It kind of bounces around a little bit.
20:22
Unidentified speaker: Starting in 2018, there's a discreet step up, and there's been a trend ever since then to going for it more aggressively.
20:29
Unidentified speaker: The intervening event was the Super Bowl 52 in February 2018, the Eagles against the Patriots.
20:37
Unidentified speaker: The Eagles went for it twice on fourth downs in unconventional situations.
20:43
Unidentified speaker: One was a very memorable trick play on fourth and goal.
20:47
Unidentified speaker: And the other, which I think is more important from the point of view of my analysis, was they were down by a little bit late-ish in the game, but not super late.
20:56
Unidentified speaker: They had fourth and one deep in their own territory.
20:59
Unidentified speaker: They went for it, they made it, they drove down, they scored a touchdown, that was the margin of victory.
21:04
Unidentified speaker: As an aside, this led to a tweet that I just enjoyed absolutely enormously from Bill Easterly, who's a very distinguished development economist.
21:13
Unidentified speaker: It was something like, you know, who says that economists are useless?
21:16
Unidentified speaker: David Romer's been arguing for years, teams should be more aggressive on smackdowns.
21:21
Unidentified speaker: Someone finally listened to him and they won the Super Bowl as a result.
21:24
Unidentified speaker: So that was fun.
21:25
Unidentified speaker: Anyway, it got a lot of attention, really got notice.
21:28
Unidentified speaker: I think it sort of teams woke up to the fact that, gee, it can make a difference.
21:32
Unidentified speaker: And so we should question what we've been doing and think things through.
21:36
Unidentified speaker: So, David, there are really, in my opinion, two interpretations of your paper.
21:41
Unidentified speaker: And also to put into context what happened after, I think there are two interpretations, right?
21:46
Unidentified speaker: And so, one is there is a lack of rationality or a lack of analytics, a lack of insights.
21:52
Unidentified speaker: People do not understand what they're supposed to be doing.
21:55
Unidentified speaker: As the technology progresses, it's possible for them to start optimizing.
21:59
Unidentified speaker: Once the optimization becomes easier, that's what they do.
22:02
Unidentified speaker: The second interpretation is, which I think is one way to read your paper, is an agency problem.
22:07
Unidentified speaker: There's the objective of winning, and there is the reputation of the coach.
22:11
Unidentified speaker: The reputation of the coach is very much dependent on what other people think the right thing to do is, regardless of whether that's right or not.
22:20
Unidentified speaker: And once it becomes clear that the other people understand what the right thing to do is, that allows the coach to do what the right thing to do is, and therefore the agency problem, if you will, is resolved because the incentives are properly aligned.
22:31
Unidentified speaker: Should I read your paper more in the context of an agency problem that is resolved, or should I read it more as a behavioral phenomenon that data analytics has resolved for us, and therefore we get to more rational decisions, or some combination of the two?
22:45
Unidentified speaker: I think my best answer is I don't know.
22:47
Unidentified speaker: Those are complicated hypotheses and it's hard to get definitive evidence on them.
22:52
Unidentified speaker: A very common reaction to my paper from people who were persuaded by the logic was yeah, you don't want to be the coach who does this.
23:02
Unidentified speaker: If you make an unconventional decision and it doesn't work, and you know, this is about, you know, there's an upside and downside whenever you try this, you'll get pilloryed and so on.
23:11
Unidentified speaker: It doesn't ring true to me, but this is not strong evidence.
23:14
Unidentified speaker: It's really saying coaches knew, at least in the back of their minds, what the right thing to do was, and they chose not to do it.
23:21
Unidentified speaker: That just doesn't seem right, particularly if you think about the people who are coaches in the NFL who have a lot of intensely smart, competitive, willing to defy things.
23:32
Unidentified speaker: The other thing about it is if you're coaching a bad team or you're a mid-level college coach who has ambitions of doing more, doing something that defies the conventional wisdom and works, and on average, you know, it's a good strategy, that has a huge payoff.
23:51
Unidentified speaker: So it's not just you did well, you did well because you were willing to buck the tide.
23:57
Unidentified speaker: So I'm skeptical of the agency theory.
23:59
Unidentified speaker: The other reason I'm skeptical of the agency theory is the pressure to do this, it wasn't the owners telling the coaches to do this.
24:09
Unidentified speaker: It was internal to the workings of things.
24:12
Unidentified speaker: And there had been people on the outside yelling at them again, you know, for decades that they were doing the wrong thing.
24:17
Unidentified speaker: So it's not like I gave them cover to do something that they knew was right.
24:22
Unidentified speaker: That's just not plausible.
24:23
Unidentified speaker: So I'm much more on the side.
24:25
Unidentified speaker: Again, I don't know, and I think that's very important, that their behavioral biases, they're very, very, very strong.
24:33
Unidentified speaker: They can block for decades these powerful competitive forces.
24:37
Unidentified speaker: But, you know, in some cases, when the evidence is clear enough, things do change.
24:42
Unidentified speaker: So I guess I lean towards your technology explanation, but again, I don't know.
24:46
Unidentified speaker: Although, David, the fact is, you know, I'm not that avid watcher of football, but I don't see them going uh fourth and one very often.
24:56
Unidentified speaker: I mean, normally they kick when it's fourth and one, right?
24:59
Unidentified speaker: So I don't watch much football either.
25:02
Unidentified speaker: I think I have to put in as an aside that something makes me uncomfortable about talking about football is that it's a sport that probably shouldn't be played at all given the costs it imposes on the players.
25:14
Unidentified speaker: And that sort of diminished my interest.
25:16
Unidentified speaker: But it's certainly clear they haven't gone all the way to the optimal strategy.
25:22
Unidentified speaker: I think that's pretty clear based on casual observation.
25:24
Unidentified speaker: I think what I don't know is whether they've sort of taken up the vast majority of the low-hanging fruit.
25:30
Unidentified speaker: So it's, you know, we talk about a dead weight loss from a triangle or an inefficiency from the triangle.
25:36
Unidentified speaker: And as you work down the bigger chunks of the triangle, even if you're not going most of the distance, you are getting most of the gains.
25:43
Unidentified speaker: I don't know if they've gotten a large majority of the gains or just a decent chunk of them.
25:47
Unidentified speaker: I think someone would need to redo my paper with modern data and modern evidence about what teams do to answer that one.
25:55
Unidentified speaker: If the explanation that you're providing is indeed the technology explanation, would that also mean that you think that the arrival of AI could lead to huge productivity increases?
26:06
Unidentified speaker: Because if this would really help people get over all their behavioral biases and they're so prevalent, as your paper suggests, that must mean that introducing more rational decision making should make a big difference, no?
26:18
Unidentified speaker: I think for things like this, yes.
26:22
Unidentified speaker: But what's notable about this is it's a very well-defined decision.
26:27
Unidentified speaker: So that's why it was great to study.
26:30
Unidentified speaker: And it's also a reason that finding failures of optimization there is really compelling.
26:36
Unidentified speaker: This is a setting where you would be particularly likely to find optimization if it were happening, right?
26:42
Unidentified speaker: I mean, very clear decisions.
26:44
Unidentified speaker: The analytics are relative to how should we position our brand in the marketplace or should we have an assembly line or team production?
26:52
Unidentified speaker: This is a vastly simpler decision.
26:54
Unidentified speaker: I think for decisions like this, AI, absolutely.
26:57
Unidentified speaker: I could imagine AI getting to the point of dear AI, please improve the productivity of my firm or my team's winning performance, and it comes back with your 10 things that you could do.
27:10
Unidentified speaker: The bigger picture things I don't know.
27:13
Unidentified speaker: Dear AI, how should we position our brand in the global marketplace?
27:17
Unidentified speaker: Is AI really gonna know that better?
27:19
Unidentified speaker: It doesn't have a lot of data to go on.
27:23
Unidentified speaker: The payoffs are very long term.
27:25
Unidentified speaker: So I think that's a hard one.
27:27
Unidentified speaker: David, what do you think the general implications are?
27:29
Unidentified speaker: I mean, I would take away from your study that in a very clear-cut, obvious case, we observe businesses in a highly competitive market not acting optimally.
27:42
Unidentified speaker: I would conclude from that, well, in a case where it's not clear-cut and the competition isn't as intense, you're gonna see large departures from optimality.
27:52
Unidentified speaker: To me, that makes complete sense.
27:54
Unidentified speaker: Yeah.
27:54
Unidentified speaker: So that's sort of what I was saying a moment ago.
27:56
Unidentified speaker: If they're not optimizing here, if they can't get this one right, there's every reason to think that they're getting other things that are harder, even more wrong.
28:06
Unidentified speaker: So that's one lesson.
28:07
Unidentified speaker: I think a second lesson is behavioral biases are really important.
28:11
Unidentified speaker: And but if you think about it, the departures from optimality were in a particular direction.
28:15
Unidentified speaker: They were towards the cautious, the conservative thing to do.
28:19
Unidentified speaker: And if you look certainly other sports contexts, that's what we see.
28:23
Unidentified speaker: Teams were very slow to figure out that three-point shots were a good thing to be doing.
28:27
Unidentified speaker: In baseball, teams were much too invested in scoring one run at a time rather than playing for the big rally and you know, the three-run home run.
28:36
Unidentified speaker: In hockey, my understanding is teams are too slow to pull the goalie if they're behind late.
28:41
Unidentified speaker: So it's behavioral, right?
28:43
Unidentified speaker: It's not there was randomly an error that they got wrong and then stuck with it.
28:48
Unidentified speaker: It's we're probably hardwired to be risk averse because you know, being eaten by a lion is very, very costly.
28:55
Unidentified speaker: And that gets misapplied.
28:57
Unidentified speaker: I remember getting an email once from uh someone who coached high school who said he went for it a lot on fourth downs, and the parents would come up and say, but you know, it's a life or death situation.
29:07
Unidentified speaker: And he would say, I fought in Korea.
29:10
Unidentified speaker: I know what a life and death situation is.
29:12
Unidentified speaker: Going forward in fourth down is not a life or death situation, and they shouldn't carry over the messages from Korea to coaching this football team.
29:21
Unidentified speaker: So behavioral biases are really big.
29:23
Unidentified speaker: I think another message is this question of why, given that the evidence was there, people were suggesting this, what is it about the organizations that made them, when it was sort of presented to them for decades, why didn't they respond?
29:40
Unidentified speaker: What makes an organization have the flexibility, the openness, and so on to be open to the idea, gee, maybe we're making a mistake.
29:50
Unidentified speaker: Maybe this professor at Berkeley, who's clearly never played a football game in his life, maybe there's something there worth looking into.
29:58
Unidentified speaker: You know, David, the implication for the economy as a whole certainly must be that we're not doing a good enough job in teaching MBAs and undergraduates the importance of analytical insight, right?
30:15
Unidentified speaker: I mean, if it is a behavioral bias where people think it's a life and death situation when it isn't, the defense is an analytical approach, and yet we don't see that.
30:26
Unidentified speaker: I agree.
30:27
Unidentified speaker: I mean, I think there are a couple of things that I don't teach in a business school, so I I'm not gonna criticize MBA education, but I think yes, the value of analytics, but also the value of just figuring out ways of being open to mistakes.
30:44
Unidentified speaker: So I'll turn it into a research question.
30:46
Unidentified speaker: I have the sense that there's a body of research, certainly a stack of sort of top management books.
30:53
Unidentified speaker: I assume they're MBA classes on how to run a company in a way that makes it nimble, that makes it respond to new ideas, makes it overcome mistakes.
31:03
Unidentified speaker: I would think it would be really interesting to sit down, say with all those books and say, what are 20 hypotheses about what makes an organization be efficient, be optimizing?
31:14
Unidentified speaker: Is it about analytics?
31:15
Unidentified speaker: Is it about the structure?
31:17
Unidentified speaker: Is it about the culture, willingness to admit failure, and then say, how do those 20 hypotheses fit with explaining why no one did this for so long?
31:28
Unidentified speaker: What were the teams that picked up on it first, and so on?
31:31
Unidentified speaker: And I'd be fascinated to know if some of those hypotheses were like really strong successes and others were not.
31:39
Unidentified speaker: Because I think that also from a cultural point of view, it should not be underestimated.
31:43
Unidentified speaker: If I look at my own kids and their generation, what Moneyball, also the movie, what that did for the reputation of analytics within a sports context.
31:52
Unidentified speaker: Suddenly it's actually cool to do this.
31:55
Unidentified speaker: And I think it was a very motivating thing for young people to find out that using these tools could actually make you win a sports game.
32:02
Unidentified speaker: I don't think it had that reputation before.
32:04
Unidentified speaker: So I think that acceleration of that insight is good.
32:07
Unidentified speaker: So it may also just mean that we are still failing at marketing the tools that we teach, that they're actually way more valuable than what is coming across.
32:16
Unidentified speaker: And it just requires Moneyball to really make that case from a marketing point of view.
32:21
Unidentified speaker: No, I agree.
32:22
Unidentified speaker: When I wrote the paper, I think I had a bunch of motivations in mind, and the journal editor handled it really pushed me, I think, completely correctly in the direction of focus on this issue of do competitive forces cause firms to optimize?
32:36
Unidentified speaker: The editor who handled the paper was Stephen Levitt at the University of Chicago.
32:40
Unidentified speaker: And his main comment was this is an intellectually serious paper.
32:45
Unidentified speaker: You should take out anything that seems frivolous or cute or off topic and just focus on the core issue.
32:55
Unidentified speaker: And that was great advice.
32:57
Unidentified speaker: But one of the motivations was this is just fun.
33:00
Unidentified speaker: This is just shows you you can use dynamic programming and linear algebra and things like that to get insights into a question where 99 people out of a hundred would think that's irrelevant.
33:11
Unidentified speaker: I recently went to my 50th high school reunion, and I would say half the people there knew about my work, about what I'd done on this.
33:21
Unidentified speaker: And rather than being the boring nerd who comes back, it was like, oh, there's this cool kid who did something on fourth down strategy.
33:28
Unidentified speaker: So that was nice.
33:30
Unidentified speaker: But you know, what makes the paper so interesting is precisely as you point out, it wasn't just a seat-of-the-pants analysis.
33:38
Unidentified speaker: And although you in the introduction, there's a lot of intuition, really the analysis is applying dynamic programming in a very useful way.
33:47
Unidentified speaker: The problem with applying dynamic programming in a useful way is it takes a lot of effort.
33:52
Unidentified speaker: It's a lot easier just to do a seat of the pants analysis, right?
33:56
Unidentified speaker: And so I wonder how much of the reticence is that you don't want to put the effort in.
34:01
Unidentified speaker: So it's easy to say, oh no, come on, that's bullshit.
34:04
Unidentified speaker: I know the game of football, I don't need to do dynamic programming.
34:07
Unidentified speaker: The same as in a business.
34:09
Unidentified speaker: I'm an expert, I've sold shoes for 30 years.
34:12
Unidentified speaker: I know how to sell shoes.
34:13
Unidentified speaker: I don't need somebody to tell me with a computer program how to sell shoes better.
34:18
Unidentified speaker: Although that cost has calmed down a lot, right?
34:20
Unidentified speaker: Particularly now with the arrival of new technologies, the cost of doing dynamic programming has become close to zero.
34:26
Unidentified speaker: Maybe.
34:26
Unidentified speaker: Depends if you understand it.
34:28
Unidentified speaker: I think there are two issues.
34:30
Unidentified speaker: I don't know which one is right.
34:32
Unidentified speaker: One is it's too costly, and so I'll do it the old way.
34:36
Unidentified speaker: The other is that means that if you're the person who's made the decisions by experience, intuition, seat of the pants, and to say, well, dynamic programming might do it better is relinquishing control, right?
34:49
Unidentified speaker: It's saying I'm not the expert here.
34:51
Unidentified speaker: And so it may be a psychological cost rather than a monetary cost, right?
34:55
Unidentified speaker: The cost to a football team of hiring someone to do dynamic programming is a drop in the bucket compared to what they do.
35:03
Unidentified speaker: And agreed.
35:04
Unidentified speaker: One of the things we talked about beforehand was, you know, how did teams react?
35:08
Unidentified speaker: And they didn't react by adopting what I suggested.
35:12
Unidentified speaker: And regardless of how strong you find the evidence in my paper, the passage of time has shown that there's now a consensus that for better or worse, I was right.
35:22
Unidentified speaker: Teams now things have changed dramatically without a big change in the costs and benefits of it.
35:28
Unidentified speaker: So the way teams reacted, I think the closest that teams got to taking it seriously would be someone to engage in the paper long enough to identify something that was left out.
35:39
Unidentified speaker: And then the next step was, so we can ignore this.
35:43
Unidentified speaker: Whereas I think if you had been reading the paper and you said something was left out, the next step would be let me think about whether that thing that's left out, first of all, which direction it goes in.
35:56
Unidentified speaker: And second, could it be strong enough to overturn the results?
35:59
Unidentified speaker: And third, you know, how would I get evidence about that?
36:01
Unidentified speaker: And no team, as far as I know, got beyond the, oh, the analysis isn't perfect, therefore we can ignore it.
36:08
Unidentified speaker: That was the general reaction.
36:10
Unidentified speaker: Well, David, I have to say thank you so much for coming on.
36:13
Unidentified speaker: I think this is such a good example, a clear-cut example of the limits of competition.
36:20
Unidentified speaker: That I think it's an excellent paper.
36:22
Unidentified speaker: It really is.
36:23
Unidentified speaker: Thank you for having me.
36:24
Unidentified speaker: It's been great to talk.
36:26
Unidentified speaker: Thank you, David.
36:29
Unidentified speaker: Thanks for listening to the All Else Equal Podcast.
36:32
Unidentified speaker: Please leave us a review at Apple Podcasts.
36:35
Unidentified speaker: We'd love to hear from our listeners.
36:37
Unidentified speaker: Also be sure to catch our next episode by subscribing or following our show wherever you listen to your podcasts.
36:43
Unidentified speaker: For more information and episodes, visit all else equal podcast.com or follow us on LinkedIn.
36:50
Unidentified speaker: The All Else Equal Podcast is a joint production of Stanford University's Graduate School of Business and the Lauder Institute at the University of Pennsylvania.
36:59
Unidentified speaker: It is produced by University FM.
This page was produced automatically: the audio was transcribed with nvidia/parakeet-tdt-0.6b-v3 and interpreted by Qwen3.6-35B-A3B-GGUF:UD-Q4_K_XL. Transcription and interpretation can contain errors; always confirm against the original episode. Nothing here is investment advice.
All insights