Han Lee and Speaker_2 discuss AI content homogenization, engineering bottlenecks, and evaluation frameworks.
Summary
Han Lee argues that AI-generated content now comprises over half of online material, leading to 'mode collapse' and homogenized voices. He notes that AI has shifted software engineering bottlenecks from code writing to planning and review, while high token costs and maintenance burdens favor buying established solutions over building in-house. Speaker_2 emphasizes that result evaluation and data trustworthiness are more critical than model selection, advocating for a Total Cost of Ownership comparison against human labor. Both speakers stress the need for domain expertise and practical AI literacy to navigate pilot failures and production challenges.
Episode sections
- 0:00 – 3:03
Intro and AI Content Volume
The podcast introduces the topic of AI hype and Han Lee's perspective on the volume of AI-generated content, noting that over half of online content may now be AI-produced, which can lead to a homogenized voice.
- 3:03 – 12:25
Preserving Human Voice and Creativity
Lee discusses the 'AI slop paradox' and 'mode collapse,' where AI reverts to common linguistic patterns, urging users to maintain distinct human voices and use AI only for verification or transcription, not core writing.
- 12:25 – 14:36
Shift in Engineering Bottlenecks
AI has removed code writing as the primary bottleneck in software engineering, shifting the constraint to planning, coordination, and review, which impacts team structures and project management methodologies.
- 14:36 – 18:48
Token Costs and Agent Behavior
High token bills have made users cost-conscious; Lee notes that AI agents are becoming more 'eager' and conversational, potentially over-scoping tasks, requiring human oversight to prevent architectural errors.
- 18:48 – 24:31
Domain Expertise and Gellman Amnesia
Lee warns against 'Gellman Amnesia' in AI marketing, where tools appear perfect to non-experts but lack domain-specific rigor; he emphasizes the need for trusted data and evaluation in compliance contexts.
- 24:31 – 27:15
Build vs. Buy and Total Cost of Ownership
While generating code is cheap, maintaining it is expensive; Lee argues that for enterprise compliance, buying established solutions is often lower total cost of ownership than building in-house due to data and operational needs.
- 27:15 – 30:44
Pilot Failure Rates and Production Effort
Lee confirms that most AI pilots fail to reach production, noting that while demos are easy to create (10-20% effort), moving to production requires 80% of the effort due to evaluation and integration challenges.
- 30:44 – 32:27
Primacy of Trusted Data
Lee identifies data as the most critical component of AI solutions, advising practitioners to first verify the source and trustworthiness of the data before evaluating the AI model itself.
- 32:27 – 35:07
Evaluating AI Agents and Total Cost of Ownership
Speaker_2 argues that evaluating AI results (accuracy, time, cost, similarity to human experts) is more critical than selecting specific models or dashboards. Speaker_1 adds that Total Cost of Ownership (TCO) must be compared against human labor costs to determine if AI is economically viable.
- 35:07 – 38:59
Abstraction of Technical Complexity for End Users
Speaker_2 contends that non-technical practitioners do not need to understand underlying technical terms like MCPs, harnesses, or context windows. He asserts that good product design should abstract these complexities away, allowing users to focus on outcomes rather than implementation details.
- 38:59 – 42:14
Practitioner Resources and AI Literacy
Speaker_1 asks for recommended resources. Speaker_2 promotes his upcoming book 'AI Evaluation and Alignment' and his personal blog/LinkedIn. He emphasizes that users should develop a 'taste' for AI by using it to understand its boundaries rather than obsessing over rapid technical jargon changes.
Claims worth verifying
Han Lee states that nowadays at least half of things written online are produced by AI.
Unidentified speaker (speaker_2) ·1:57
How to verify: Survey current web content using AI detection tools or conduct a statistical sample of recent blog posts and social media posts to estimate the percentage of AI-generated text. For recent events, verify 2026-08-03 through 2026-11-03. Use another period only when the speaker explicitly names it.
AI models tend to output the most common word used, reverting to the mean, a phenomenon called mode collapse.
Unidentified speaker (UNKNOWN) ·3:09
How to verify: Review machine learning literature on 'mode collapse' in generative models and analyze output variance from large language models with different prompts. For recent events, verify 2026-08-03 through 2026-11-03. Use another period only when the speaker explicitly names it.
AI coding agents are removing human writing code as the bottleneck in software engineering.
Unidentified speaker (speaker_2) ·12:59
How to verify: Review industry reports on software development productivity metrics and AI adoption rates in coding tasks from sources like GitHub or McKinsey. For recent events, verify 2026-08-03 through 2026-11-03. Use another period only when the speaker explicitly names it.
Users have seen token bills of $5,000 to $10,000 from AI services.
Unidentified speaker (speaker_2) ·14:36
How to verify: Search for public anecdotes or case studies of enterprise AI token costs on tech news sites or cloud provider billing dashboards. For recent events, verify 2026-08-03 through 2026-11-03. Use another period only when the speaker explicitly names it.
95% of AI projects or agentic solutions are dying after the pilot stage.
Unidentified speaker (speaker_1) ·27:26
How to verify: Verify this statistic against the source event mentioned (hosted by a major model provider and systems integrator) or search for industry reports on AI pilot failure rates. For recent events, verify 2026-08-03 through 2026-11-03. Use another period only when the speaker explicitly names it.
It takes 10-20% of effort to create a demo, but 80% to move to production in the AI era.
Unidentified speaker (speaker_2) ·29:34
How to verify: Compare this claim to historical software engineering effort estimates (e.g., Boehm's model) and current industry surveys on AI project lifecycles. For recent events, verify 2026-08-03 through 2026-11-03. Use another period only when the speaker explicitly names it.
Han Lee uses Obsidian for note-taking and converts content into markdown notes.
Unidentified speaker (speaker_2) ·5:36
How to verify: Check Han Lee's LinkedIn profile or website for mentions of Obsidian as a tool in his workflow. For recent events, verify 2026-08-03 through 2026-11-03. Use another period only when the speaker explicitly names it.
Moody's builds agent services with a focus on evaluation, monitoring, and auditability.
Unidentified speaker (speaker_2) ·21:03
How to verify: Review Moody's official website or press releases regarding their AI and data products, specifically looking for mentions of evaluation frameworks. For recent events, verify 2026-08-03 through 2026-11-03. Use another period only when the speaker explicitly names it.
AI models are adopting a social media tone and asking more questions to keep users engaged.
Unidentified speaker (speaker_1) ·15:31
How to verify: Observe the behavior of current LLM interfaces (e.g., ChatGPT, Gemini) and check for updates in model cards or documentation regarding conversational design. For recent events, verify 2026-08-03 through 2026-11-03. Use another period only when the speaker explicitly names it.
Han Lee is the Senior Director of Data and AI at Moody's.
Unidentified speaker (UNKNOWN) ·1:03
How to verify: Check Moody's corporate website leadership page or Han Lee's LinkedIn profile for his current title. For recent events, verify 2026-08-03 through 2026-11-03. Use another period only when the speaker explicitly names it.
Speaker_2 states that the second most important factor in AI adoption is how to evaluate the result.
Unidentified speaker (speaker_2) ·32:27
How to verify: Transcript text at [32:27]. For recent events, verify 2026-08-03 through 2026-11-03. Use another period only when the speaker explicitly names it.
Speaker_2 claims that before ChatGPT, people sold software, not decisions or solutions.
Unidentified speaker (UNKNOWN) ·32:38
How to verify: Transcript text at [32:38]. For recent events, verify 2026-08-03 through 2026-11-03. Use another period only when the speaker explicitly names it.
Speaker_2 lists accuracy, time, token cost, and similarity to human experts as key metrics for evaluating AI agents.
Unidentified speaker (UNKNOWN) ·33:14
How to verify: Transcript text at [33:14]. For recent events, verify 2026-08-03 through 2026-11-03. Use another period only when the speaker explicitly names it.
Speaker_2 asserts that result evaluation and data trust are the two most important things in AI.
Unidentified speaker (speaker_2) ·34:13
How to verify: Transcript text at [34:13]. For recent events, verify 2026-08-03 through 2026-11-03. Use another period only when the speaker explicitly names it.
Speaker_1 suggests that if AI costs twice as much as a junior employee, the junior employee might still be the better option.
Unidentified speaker (speaker_1) ·34:23
How to verify: Transcript text at [34:23]. For recent events, verify 2026-08-03 through 2026-11-03. Use another period only when the speaker explicitly names it.
Speaker_2 states he does not care about technical terms like MCP, harnesses, or context for personal use.
Unidentified speaker (speaker_2) ·36:03
How to verify: Transcript text at [36:03]. For recent events, verify 2026-08-03 through 2026-11-03. Use another period only when the speaker explicitly names it.
Speaker_2 compares the ease of using ChatGPT to using Siri or Alexa, stating it is much easier.
Unidentified speaker (UNKNOWN) ·37:28
How to verify: Transcript text at [37:28]. For recent events, verify 2026-08-03 through 2026-11-03. Use another period only when the speaker explicitly names it.
Speaker_2 mentions his book 'AI Evaluation and Alignment, The Seminal Papers' is up and coming with 'men's Minutes publishing'.
Unidentified speaker (speaker_2) ·39:35
How to verify: Transcript text at [39:35]. Verify publisher name via Manning Publications website or book listings; 'men's Minutes' is likely a transcription error for 'Manning'. For recent events, verify 2026-08-03 through 2026-11-03. Use another period only when the speaker explicitly names it.
Do your own research
- Verify current estimates of AI-generated content volume against recent studies from Pew Research or similar bodies.
- Review Han Lee's published works or LinkedIn posts for detailed examples of 'mode collapse' and linguistic patterns.
- Calculate Total Cost of Ownership (TCO) for AI implementations by comparing token costs and maintenance hours against equivalent human labor rates.
- Check for independent evaluations of AI agents against the four metrics proposed: accuracy, time, cost, and similarity to human experts.
- Investigate the publication details of 'AI Evaluation and Alignment' to confirm publisher accuracy and content scope.
- Assess internal pilot failure rates and production effort ratios to validate the claim that 80% of effort is post-demo.
- Review enterprise compliance requirements to determine if 'buying' established solutions is indeed lower TCO than in-house builds.
- Look for counter-arguments to 'Gellman Amnesia' by examining domain-specific AI tools with expert-led evaluations.
Full transcriptexpand
- 0:00
- Unidentified speaker (UNKNOWN): By
- 0:01
- Unidentified speaker (speaker_0): downloading or listening to this podcast, you are agreeing to Moody's legal terms and conditions found at Moody's.com slash disclaimer, including that the information provided is not investment or financial advice, and that Moody's will not be liable for losses arising from your use of the information.
- 0:27
- Unidentified speaker (speaker_1): This is Risk Reframed.
- 0:30
- Unidentified speaker (UNKNOWN): The
- 0:31
- Unidentified speaker (speaker_1): show where we bring you the leading risk experts from Moody's, our partners, and the wider ecosystem to discuss the risks organizations face,
- 0:39
- Unidentified speaker (UNKNOWN): how
- 0:40
- Unidentified speaker (speaker_1): they're interconnected,
- 0:41
- Unidentified speaker (UNKNOWN): and
- 0:42
- Unidentified speaker (speaker_1): ultimately what you, the risk professional, can do about
- 0:45
- Unidentified speaker (UNKNOWN): them.
- 0:51
- Unidentified speaker (speaker_1): Is the hype about AI just hype?
- 0:54
- Unidentified speaker (UNKNOWN): To
- 0:54
- Unidentified speaker (speaker_1): figure this out, we're joined by Han Lee. Han is an AI machine learning leader, author, and currently serving as senior director of data and AI at Moody's.
- 1:03
- Unidentified speaker (UNKNOWN): Han,
- 1:03
- Unidentified speaker (speaker_1): how are you doing?
- 1:05
- Unidentified speaker (speaker_2): Good. Thank you uh for having
- 1:07
- Unidentified speaker (UNKNOWN): me.
- 1:08
- Unidentified speaker (speaker_1): No, of course. I um I had mentioned to you before we hit record that uh part of this is because I've been really enjoying your writing. both on LinkedIn, but also some the things you put on your website, et cetera, which I think's kind of um brought a bit of reality to my views around AI and what's going on in the world when there's a lot of noise. And as you know, I was
- 1:28
- Unidentified speaker (speaker_1): thinking about calling this episode 10 things I hate about AI, just because I quite like the ring of that.
- 1:33
- Unidentified speaker (UNKNOWN): But
- 1:33
- Unidentified speaker (speaker_1): hate is a really strong word, 10 is a lot. So if we can bring the reality check that I think I've been getting to the wider audience, that isn't necessarily a list of things, uh, we'll be doing well. So uh thanks for joining and thanks for the writing you are already doing. Um as a point of interest, Han, what what percentage of things that you've read today or listened to
- 1:53
- Unidentified speaker (speaker_1): or seen do you think were produced by AI?
- 1:57
- Unidentified speaker (speaker_2): That's that's a very good question. I think initially, right, when ChatGPT just got started. it was relatively uh small percentage of things are visibly AI, right? And nowadays it's getting to a point where every single turn, like at least half of the things are written by AI. And it's
- 2:17
- Unidentified speaker (speaker_2): it's getting to a point that is uh annoying.
- 2:22
- Unidentified speaker (speaker_1): You're telling me, I I think today I've been above 60, 70%. And some of them have been good uses, right? Like, don't get me wrong. Like people have pulled together quite large strategy reports based on real data, based on things they've turned it around quicker. So I've been able to read that, digest that, understand it. And I can put the fact that it reads in a
- 2:41
- Unidentified speaker (speaker_1): particular tone, typical of a LLM, to one side. But then when I see stuff that's like marketing marked in or that, you know, LinkedIn side is like it's actually, I don't want every single person to have the same voice. It's um, as you say, it can be tad annoying and it kind of loses some of the message power, um, or power of the message, I should
- 3:01
- Unidentified speaker (speaker_1): say, when you do that.
- 3:03
- Unidentified speaker (speaker_2): Right. It's it's sort of a paradox where for a lot of analytical work,
- 3:09
- Unidentified speaker (UNKNOWN): right?
- 3:10
- Unidentified speaker (speaker_2): AI can create awesome great analytical work. It can come connect like all of the dots between data, observations together. But once in a while, a lot of the time when we write, we want to hear people's voice instead of just something that's that was a
- 3:29
- Unidentified speaker (speaker_2): voice of AI, right? So
- 3:31
- Unidentified speaker (UNKNOWN): there's
- 3:31
- Unidentified speaker (speaker_2): something called that internet theory from many, many years ago, where people uh hypothesized that. the uh the internet would be dead because everybody will be reading on the same thing. And I think AI accelerates that process quite a whole lot. So now a lot of the generated blog posts or people
- 3:51
- Unidentified speaker (speaker_2): maybe might be using AI to generate a bunch of content for SEO and whatnot, and they all sound the same. And AI is feeding beating off those energy to trend the next generation. of models. And overall, we just uh lose out a lot of the human voice and our creativity and original
- 4:11
- Unidentified speaker (speaker_2): thoughts in inside.
- 4:13
- Unidentified speaker (UNKNOWN): Yeah.
- 4:13
- Unidentified speaker (speaker_1): I mean, it feels a bit like the uh video streaming industry, right? There's now more content than ever, but a lot of it's pretty average.
- 4:21
- Unidentified speaker (speaker_1): Yes. When you keep making the the amount of things produced bigger and bigger, like there's only so much good. stuff, and now it's actually harder to find.
- 4:29
- Unidentified speaker (UNKNOWN): I
- 4:29
- Unidentified speaker (speaker_1): know our chief marketing officer Chris Bailey says it's better to be different than good often in uh trying to get something to cut through. So now it's genuinely when I see someone that's ridden something themselves, or I think they have at least, and it's distinctive. I I'm far more willing to overlook, you know, silly mistakes or whatever, because I'm like, no, at least it's them, at least I can really tell what they thought.
- 4:49
- Unidentified speaker (UNKNOWN): Um
- 4:49
- Unidentified speaker (speaker_1): and there is that idea that to write is to think, right? Until you've actually written it down, it's just kind of noise inside your own head.
- 4:56
- Unidentified speaker (UNKNOWN): Um
- 4:56
- Unidentified speaker (speaker_1): at least for myself personally, I prefer to write in a notepad first. Where AI is good is I then take a picture and then I get it to try transcribe it and that's way faster than me me typing
- 5:06
- Unidentified speaker (UNKNOWN): it
- 5:06
- Unidentified speaker (speaker_1): up. So I was gonna say how do you how do you use it when you're thinking, when you're writing, or do you just not use it at all when you you go to work with a a keyboard key and you're off or
- 5:16
- Unidentified speaker (speaker_2): so so there's there's two two sides of the process where one is in more engineering term how do I ingest more data into my brand or how do I read and process more data or how do I read more content because there's there's just so much more content nowadays long form short form and whatnot papers YouTube videos and and one of the things I do
- 5:36
- Unidentified speaker (speaker_2): at least uh personally is I inject I read all of the content and I put the content I have. I'm a heavy user of Obsidian. So I turn all the content that I read into markdown notes in my obsidian repository. So this helps me recall or remember the things that I read because a lot of times
- 5:56
- Unidentified speaker (speaker_2): it's sometimes I might have a light bulb going on, and and that's when the time I uh I need to either verify.
- 6:03
- Unidentified speaker (speaker_2): if I read it somewhere else or is that true? And and and having a good note-taking system that definitely helps a lot with that, right? In the old days, it was it was me purely with pen and pencil breaking everything down, and now there's there's AI assistant uh to help with that, and that's that helped it tremendously. But for most
- 6:23
- Unidentified speaker (speaker_2): of the writing process, I rely mostly on hand. written notes. And I do use AI to help me correct some grammatical mistakes or do do a deep research. Let's say, for example, if I write an article, a quick article, uh I want to make sure the technical integrity of the claims that I make in the
- 6:43
- Unidentified speaker (speaker_2): article, then I will utilize As AI to help verify my claims. And those are extraordinarily. helpful. But I try not to use AI to write on the mechanical part because for whatever reason, right, it was not able to write in my voice or capture
- 7:03
- Unidentified speaker (speaker_2): my voice. And that
- 7:05
- Unidentified speaker (UNKNOWN): I
- 7:05
- Unidentified speaker (speaker_2): want to be at least me.
- 7:08
- Unidentified speaker (speaker_2): And I believe my audience, they want to hear me talk or to hear me write instead of hearing me prompting. AI to write. In the old days, right? I think Mark Twain or someone has a quote that hey I don't have time to write a short letter, so I wrote you a long one. And I find the same thing with
- 7:28
- Unidentified speaker (speaker_2): AI if hey I if I don't have time to write you a good content, I will throw AI slop at people. Which is from time to time
- 7:39
- Unidentified speaker (speaker_2): does feel a bit insulting that uh that hey, you know, uh you you are not willing to spend enough time crafting your voice, crafting your own content to communicate to your audiences.
- 7:53
- Unidentified speaker (speaker_1): Yeah. And
- 7:54
- Unidentified speaker (UNKNOWN): I think
- 7:55
- Unidentified speaker (speaker_1): you have a great post on this AI slot paradox, I think. Was that your term, or did you get that from somebody else?
- 8:01
- Unidentified speaker (speaker_2): AI slot paradox, I can came up with that, but I'm not the the origin of the word AI slob
- 8:09
- Unidentified speaker (UNKNOWN): that is from everywhere. That's
- 8:11
- Unidentified speaker (speaker_2): that's everywhere. So
- 8:12
- Unidentified speaker (UNKNOWN): I came up with that,
- 8:13
- Unidentified speaker (speaker_2): right? Yeah.
- 8:14
- Unidentified speaker (UNKNOWN): So,
- 8:14
- Unidentified speaker (speaker_1): like, what do you think? Again, you have the advantage you're an AI and machine learning engineer. You understand what's actually under the hood, so to speak. For you know, a marketer, a journalist, um, somebody who's a you
- 8:27
- Unidentified speaker (UNKNOWN): know
- 8:27
- Unidentified speaker (speaker_1): content producer, like what are they getting wrong when they
- 8:32
- Unidentified speaker (UNKNOWN): use
- 8:32
- Unidentified speaker (speaker_1): these tools and then put out, as we say, sort of the things that would fall into the slop category as opposed to things that are you know actually helping their audience and engaging them?
- 8:42
- Unidentified speaker (speaker_2): I think in general, if I'm going to explain this using an a AI or machine learning term, this is what we call a mode collapse. So in any model, AI, machine learning, and whatnot, they tend to output the most common word used,
- 9:02
- Unidentified speaker (speaker_2): which means that if you do not prompt AI in a very specific way, or if you do not steer AI every single sentence, it will always reverse. revert to the mean, to the mode, the most common way of it to write. And that's where
- 9:22
- Unidentified speaker (speaker_2): uh things happen, where you can see a lot of people writing in the same way. Uh, some of the linguistic patterns you will see is AI tend to favor a lot of what I believe it is called staccato sentences, very short, punctual. sentences, one after
- 9:40
- Unidentified speaker (UNKNOWN): one,
- 9:40
- Unidentified speaker (speaker_2): one after another. And or there's the classic, uh it is not, but it is, right? The those sort of uh uh
- 9:51
- Unidentified speaker (UNKNOWN): sort of setups,
- 9:51
- Unidentified speaker (speaker_1): right? Like that's what
- 9:52
- Unidentified speaker (UNKNOWN): I pick up,
- 9:53
- Unidentified speaker (speaker_1): is often how how it sets up its key point is kind of somewhat melodramatic. And then also the uh the classic M-dashes, which I think everyone's picking up now. The M-dash is kind of like, oh no. So anytime I see those, it almost makes my brain switch off when I go and not reading this.
- 10:11
- Unidentified speaker (speaker_2): Yeah. Yeah.
- 10:13
- Unidentified speaker (speaker_2): It's also interesting how how either
- 10:17
- Unidentified speaker (UNKNOWN): an
- 10:17
- Unidentified speaker (speaker_2): adoption of AI tools or a version of AI rating is also changing the way we write. So I'm gonna be a 100% hundred percent honest, right? Before ChatGPT, I wasn't aware of the proper way of using M-dashes. And I picked up, I learned how to write uh using some
- 10:37
- Unidentified speaker (speaker_2): M-dashes from time to time from AI, because it just looks great initially. And nowadays it could be that I will want to limit the use of M-dashes M dashes going forward. just so that my writing doesn't read too much like AI. Same thing with emojis, right? Uh AI
- 10:57
- Unidentified speaker (speaker_2): tend to use a lot of emojis, or the AI last year tend to use a lot of emojis in their writing, and that doesn't look good. And that's something, some little things that you spot. So so yeah, we are adapting to our tools and learning from our tools as well.
- 11:13
- Unidentified speaker (speaker_1): Okay. So essentially anyone that's gonna put something in their feed or certainly in mine, please make sure you've edited at the very least. Please make sure you're not putting M dashes. Please make sure that you are not just going with whatever the LLM has put out. And make sure you're different, right? Like that's you know, tip to anyone. Make sure you speci specifically
- 11:33
- Unidentified speaker (speaker_1): sound like you, regardless of who you're communicating to. Because otherwise it's very hard to trust, I would say.
- 11:39
- Unidentified speaker (speaker_1): And if someone can't trust your communic what you're communicating, then it it loses its its impact. Um let's talk kind of more about your your day day job, if you like, your your area of expertise about engineering.
- 11:54
- Unidentified speaker (speaker_1): How exciting versus how frustrating is it at the moment being a
- 11:59
- Unidentified speaker (UNKNOWN): an
- 11:59
- Unidentified speaker (speaker_1): expert on AI and machine learning and and on engineering specific specifically? Because
- 12:04
- Unidentified speaker (UNKNOWN): I
- 12:04
- Unidentified speaker (speaker_1): can't remember if it was end of last year or something this year, there was the big hype about vibe coding. And there's this idea that everyone can build anything if they've just got enough prompts. How do you sort of take that as an engineer and someone who like thinks through the whole process? And yeah, how do you weigh up your, as I say, excitement versus potential frustration with this
- 12:24
- Unidentified speaker (speaker_1): um narrative?
- 12:25
- Unidentified speaker (UNKNOWN): Yeah,
- 12:26
- Unidentified speaker (speaker_2): I think just this is super exciting in the sense that it is changing the way we work, changing the way we think, even changing the way we plan things. So, as an example,
- 12:39
- Unidentified speaker (speaker_2): previously, the coding portion of software engineering is limited by the number of hours and the number of humans or human developers we have. on board. And so we invented a whole lot of processes that say using Gira tickets, story
- 12:59
- Unidentified speaker (speaker_2): pointing, and all these things, because human writing code was the bottleneck. And AI removed and mitigated a lot of that bottleneck. So now it's not human writing code being the bottleneck.
- 13:15
- Unidentified speaker (speaker_2): but then it is going to be what we should be doing, what we should be planning, becoming the bottleneck. And perhaps there's always the co-review portion, which is which I believe is getting automated pretty quickly as well. So it becomes what do we want to choose to do? Right. And
- 13:36
- Unidentified speaker (speaker_2): also the size of engineering team is. also getting impacted, right? Because now coding and the number of code output is no longer the bottleneck, and everyone can wield five, ten, three, five, ten agents at once. Then do we still need a two-pizza team, one pizza
- 13:56
- Unidentified speaker (speaker_2): team? Uh, and and one of the risks is that in the old days we do we we do planning projects. process so that we ensure everybody is working on things independently so we don't we don't step onto each other's. And and now if engineers are using, let's say,
- 14:16
- Unidentified speaker (speaker_2): five coding agents, right? There's only so much work you can parallelize without things uh without agents stepping over each other and creating a whole big mass of spaghetti. So the way we organize is also changing over time and very quickly as well. And it's
- 14:36
- Unidentified speaker (speaker_2): also interesting that you mentioned like uh the the the what we went through like token maxing, uh as people call it. Use as much token as possible uh from earlier this year to now everybody is a lot more conscious.
- 14:51
- Unidentified speaker (speaker_2): about the cost because Pika because once you see a $5,000, $10,000 token bill coming from your token services, that puts a scare to people, right? So there's
- 15:04
- Unidentified speaker (UNKNOWN): a lot of that's a
- 15:05
- Unidentified speaker (speaker_1): low number, Han. I saw I saw hundreds of millions of my the news I was reading for certain companies.
- 15:10
- Unidentified speaker (UNKNOWN): And
- 15:11
- Unidentified speaker (speaker_1): you know, I I've got to say actually the all of the models, right, that I use personally. personal life and then uh in in my work life, they all seem to have adopted a little bit of a social media tone. They try to keep me asking more questions now. Whereas when they first came out, they'd kind of answer the question and I'd ask follow ups
- 15:31
- Unidentified speaker (speaker_1): if I had them or if they hadn't missed something. Now it's almost like everything I ask, it gives me like three, four, five options or other things. to try and get me to prompt again, is what it feels like. I don't know if that is deliberate because the companies obviously want me to use tokens, or if that's um just because the the models are getting more curious and and more conversational. But uh it
- 15:51
- Unidentified speaker (speaker_1): feels like there's an incentive
- 15:53
- Unidentified speaker (speaker_2): for the
- 15:53
- Unidentified speaker (speaker_1): companies providing it to keep me on there and keep my attention
- 15:56
- Unidentified speaker (UNKNOWN): there.
- 15:57
- Unidentified speaker (speaker_2): Yeah, I think I think it's it's a natural natural progression, right? So if we think in terms of a a very junior engineer, when they just start started to code, they will take a question and will solve it, right? So that's you know, your high school, high school computer science or your entry-level like
- 16:17
- Unidentified speaker (speaker_2): college computer science. And as the agent gets smarter and smarter, and now they're definitely extraordinarily smart and intelligent in doing some of the tasks. Then they will be,
- 16:29
- Unidentified speaker (UNKNOWN): they
- 16:29
- Unidentified speaker (speaker_2): will take on a much more senior staff level of role, questioning and asking for clarifications. And sometimes they will be over, they will be very, very eager to do more. So from time to time on the coking agent side, you will see that you provide a little scope,
- 16:49
- Unidentified speaker (speaker_2): but the agent creates. a whole big blast surface in terms of the number of changes because it is eager, uh eager to work more. So it sometimes goes out of bound. And that's something that still requires a human expert out there to make sure that,
- 17:09
- Unidentified speaker (speaker_2): hey, please don't change the things you're you are not supposed to change. Hey, this architecture design. decision shouldn't be, shouldn't belong to this change. It should be done in a separate task. So it does become very overly eager from time to time. It's not too dissimilar to human
- 17:29
- Unidentified speaker (speaker_2): growing from a junior type of capability to more senior. And put potentially going forward, they will be able to define their own goals. They will learn from their own learnings. and whatnot, and will eventually become uh recursive self-in self-improvement.
- 17:45
- Unidentified speaker (speaker_1): Yeah. So maybe not quite as nefarious as I perhaps thought, but uh just eagerness, as you said, to uh to impress and improve.
- 17:53
- Unidentified speaker (UNKNOWN): In
- 17:53
- Unidentified speaker (speaker_1): when I'm thinking about the risk practitioners hand that listen to this this show, right? And they're thinking, okay, well, talked about marketing and that they all receive marketing and they have to sift through what's real and what's not. And then they all use engineering, right? Every single risk practitioner in the world has to use software to do their job, right? Every single one,
- 18:11
- Unidentified speaker (UNKNOWN): um,
- 18:11
- Unidentified speaker (speaker_1): or to do it well at least. Like, if
- 18:15
- Unidentified speaker (UNKNOWN): they're
- 18:15
- Unidentified speaker (speaker_1): so they're a non-engineer, they're a risk practitioner, like what do they need to understand about tools that are built with AI versus not? Because you mentioned obviously smaller teams, the developer numbers or the developing hours are no longer the bottleneck, but actually there's a premium now on planning and on figuring out like what's going to get in each
- 18:35
- Unidentified speaker (speaker_1): other's way. Like what are the good and bad downstream impacts for somebody that might might be on the receiving end or the end user of the the tools that are being built with AI enabled engineering teams?
- 18:48
- Unidentified speaker (speaker_2): Sure. I think that's that's a very good question.
- 18:52
- Unidentified speaker (UNKNOWN): I
- 18:52
- Unidentified speaker (speaker_2): think there's there's one effect called Gellman Amnesia, where when you read a newspaper, you open it up, read the newspaper on the subject that you do not understand, the newspaper seems to be writing really, really well, c covering all the all the
- 19:12
- Unidentified speaker (speaker_2): different cases, providing some documents. providing you information. But on the subject that you know and understand well, then you will find that the newspaper they don't know what they are writing about, or they don't know what they're talking about.
- 19:29
- Unidentified speaker (speaker_2): And this this is true for a lot of the AI marketing materials. Everything looks so perfect, aging Agent can do everything uh so well. that you don't need anybody, blah, blah, blah. But the the the reality is once you once we go down and start to
- 19:48
- Unidentified speaker (speaker_2): investigate a lot of this AI tooling, we'll find there could be a lot of deficiencies because there's not a whole lot of domain expertise involved in creating some of these tools.
- 20:03
- Unidentified speaker (UNKNOWN): So
- 20:03
- Unidentified speaker (speaker_2): there could be a agent skill, right? So so the so so there is a skill is a is a file that describes, you know, that tells the agent what to do to put to complete a task, right? So there could be agent skills created by third party. There's probably any agent skill for everything nowadays created by third party that
- 20:23
- Unidentified speaker (speaker_2): doesn't have the expertise in in risk and in complex compliance use cases. And those skills look good. And for AI models, if you input anything, it will always output and return you something that's so convincing, but there's no guarantee or there's no easy way to validate if that is
- 20:43
- Unidentified speaker (speaker_2): correct or incorrect, unless you have domain experts, unless you have trusted data. And that's where we come in as a team, where the way we build a lot of to build all of our agent services from our from our screening agent for our
- 21:03
- Unidentified speaker (speaker_2): uh IDD report and all of the all of this uh agency Agent take services that we have, we put a whole lot of focus on the evaluation point. part of. So does the agent actually do the work? How are we going to define the agent is doing the work? And also,
- 21:24
- Unidentified speaker (speaker_2): once the agent is live and running, how are we going to monitor if the agent's output is correct? It is still legitimate. Maybe not that that's not the right best word, but it's is is is the the output of the agent. still good? How do we audit? When do we audit? And all this sort
- 21:44
- Unidentified speaker (speaker_2): of scenarios should be taken into consideration as a uh in a holistic way, instead of okay, there's an agent, agent output is good, uh and there's a big assumption, and let's just go with that.
- 21:56
- Unidentified speaker (UNKNOWN): Um,
- 21:56
- Unidentified speaker (speaker_1): these seem to be the same points uh that were around when it was just machine learning back, you know. I say back in the day, six, seven years ago, we were talking about machine learning models to reduce false positives across a number of uh risk uh domains and risk tasks. And people saying, oh, there's
- 22:12
- Unidentified speaker (UNKNOWN): a
- 22:12
- Unidentified speaker (speaker_1): lot of things that get flagged incorrectly. We need to reduce that. And we said, okay, we'll use probabilities and supervise uh learning models. And at the time it was all about explainability and model governance documents. Um Right. is this the same answer as essentially, but now it's an agent as the technology as opposed to just the machine learning model?
- 22:33
- Unidentified speaker (speaker_1): Again, as a non-technical person, but who's been around this space for you
- 22:37
- Unidentified speaker (UNKNOWN): know
- 22:37
- Unidentified speaker (speaker_1): 10 years now, although I don't like to say it, um it sounds very uh sounds very similar.
- 22:44
- Unidentified speaker (speaker_2): Correct. It it should be it it it is it is at the end, an AI model is based on machine learning, right? It's trend using machine learning methods. It's it should and it is evaluated using machine learning methods. So it is part of machine learning. Um it may be a superset of machine learning, but it's it should be evaluated
- 23:04
- Unidentified speaker (speaker_2): the same way, right? It is really at the end of the day,
- 23:09
- Unidentified speaker (UNKNOWN): the
- 23:09
- Unidentified speaker (speaker_2): way we evaluate machine learning models or AI agents, the end goal is the same. How can we make sure we bring a trusted product to serve to our customers? And yes, there could be uh especially for some of this task that's more sensitive and involve
- 23:28
- Unidentified speaker (speaker_2): compliance use case and risk. Right. And so it's it's the rigor that we should be looking at.
- 23:35
- Unidentified speaker (speaker_1): Yeah. When you've talked about the domain expertise being critical. and there'll be vendors like Moody's and some of our competitors that have been around a long time, doing this a long time, that have that domain expertise. Many customers, particularly the large ones, might also say that they have expertise.
- 23:51
- Unidentified speaker (UNKNOWN): And
- 23:51
- Unidentified speaker (speaker_1): they could say, well, with AI, we could now, you know, build tools. Like, does the build versus buy A consideration for compliance and risk use cases change because of vibe coding coding/slash slash AI? engineering, in your opinion? Or
- 24:06
- Unidentified speaker (UNKNOWN): would
- 24:06
- Unidentified speaker (speaker_1): it would it still be the case of like it's cheaper, easier to maintain when you have one service being maintained for many, many customers versus every customer building their own solution agentically? But as you said, you know, building with agents is not risk-free. Free. So yeah, I'm trying to figure out for practitioners listening, how do they now assess build versus
- 24:26
- Unidentified speaker (speaker_1): buy? given the all of the noise about AI in their newsfeeds?
- 24:31
- Unidentified speaker (speaker_2): That's another good question, right? So if we circle back, the cost of generating something is extraordinarily low nowadays.
- 24:40
- Unidentified speaker (speaker_2): But the cost of maintenance is still sky high, right? The cost of maintenance, up maintaining the software, updating the software. maintaining the data, and changing the data, updating the data, all those still cost tremendously. And I think
- 25:00
- Unidentified speaker (speaker_2): one of the big factors we do need to think about is not the cost.
- 25:06
- Unidentified speaker (speaker_2): We need to think in terms of total cost of ownership. How much does it cost for us to operate as a system? Because this is not a this is not something irregular software. It's not something that we should be just create a prompt and sell it to customers. And at the end, hands off,
- 25:26
- Unidentified speaker (speaker_2): we don't need to do anything more, right? For a lot of our services, especially in the uh compliance side, it is not just the software, but the support and most importantly, the operational. data and operational competencies we bring to the table. And even for myself,
- 25:46
- Unidentified speaker (speaker_2): nowadays, for yes, for my some of my personal app, right? I have some like little personal app for to-do list or to track some of my progresses here and there. Those are completely fine. Those are not customer-facing. customer facing. It's really just for me as a person. But if it's if it's something I need to operationalize as a company,
- 26:06
- Unidentified speaker (speaker_2): then it's definitely a lot more
- 26:10
- Unidentified speaker (speaker_2): easier, a lot lower total cost of ownership if you buy instead of build in-house.
- 26:18
- Unidentified speaker (UNKNOWN): Yeah.
- 26:18
- Unidentified speaker (speaker_3): So
- 26:19
- Unidentified speaker (speaker_1): it's again like bringing it back to my regular person, by that, I mean non-engineering example of like me and my wife will talk about, oh, let's go make this. you know, particular dinner or this particular dessert. And by the time you've bought all the ingredients, you're like, oh wow, like we we could have bought that for a lot cheaper just because the economy is of scale. Um, it feels very similar. Like you're
- 26:39
- Unidentified speaker (speaker_1): saying it's not that hard to produce, right? You can throw stuff together, you can average something new or ever, but if um you're doing it for many people, you might as well
- 26:47
- Unidentified speaker (UNKNOWN): give get a
- 26:47
- Unidentified speaker (speaker_1): service.
- 26:48
- Unidentified speaker (speaker_2): I I I I have a lot of bottles. bottle of spices in my kitchen cabinet that's half used and probably use them once uh once a year, once every couple of years. And yeah, it does it make sense to uh to build in-house.
- 27:03
- Unidentified speaker (speaker_2): Sure, you can, but it doesn't mean that you should,
- 27:07
- Unidentified speaker (speaker_1): right? This is I mean, the spice example is a great one. It definitely better than uh what I just used in terms of the overall cooking, but it's
- 27:15
- Unidentified speaker (UNKNOWN): I
- 27:15
- Unidentified speaker (speaker_1): went to an event a few weeks ago uh specifically about implementing AI across the enterprise. It was hosted by one of the major model providers and one of their systems integrator partners. And
- 27:25
- Unidentified speaker (UNKNOWN): it was
- 27:26
- Unidentified speaker (speaker_1): fascinating because they were very honest, right? And about a hundred, hundred and fifty-person event said, look, ninety-five percent of AI uh projects or you know, agentic solutions that are being tried. piloted, et cetera, are just kind of dying after the pilot. Like they're not get making to production. The ones that do make it to
- 27:46
- Unidentified speaker (speaker_1): production are not making it enterprise-wide.
- 27:48
- Unidentified speaker (UNKNOWN): So
- 27:48
- Unidentified speaker (speaker_1): there's no lack of effort, innovation, creativity, et cetera. But that ability to scale and govern and maintain at the moment seems it to be lacking in in 19 out of 20 cases.
- 28:02
- Unidentified speaker (UNKNOWN): Is
- 28:02
- Unidentified speaker (speaker_1): that Does that surprise you, or is that par for the course and what you'd expect in terms of where we're at as a business world right now?
- 28:10
- Unidentified speaker (UNKNOWN): I
- 28:10
- Unidentified speaker (speaker_2): think that's where we're well within expectation, right? So uh
- 28:15
- Unidentified speaker (UNKNOWN): as
- 28:15
- Unidentified speaker (speaker_2): I said previously, right now, all these coding agents remove of writing code as a bottleneck. So that explains where we have the explosion. an explosion of very nice looking demos that sound so promising. But once those demos need to land,
- 28:35
- Unidentified speaker (speaker_2): then that's where we will need to spend an extraordinary amount of time doing evaluations. Is the agent outputting the right answer? Is the agent citing the right sources or the actual data sources? Right. And it trusted data sources. Is the agent using the right tools to do the
- 28:55
- Unidentified speaker (speaker_2): job? All of this takes a lot of effort and time to make sure that they work properly. And it does also require enough statistical machine learning or mathematical maturity to understand that and sometimes translate that different.
- 29:14
- Unidentified speaker (speaker_2): requirements from stakeholders into code. So all of those are still in effect. So people may think, okay, software engineering is that. No, coding is probably solved, but software engineering is not just coding, it's also the way we maintain, 10, the way we build, the way we
- 29:34
- Unidentified speaker (speaker_2): architect to make sure it's scalable, and all of this these surroundings. surrounding services are still harder than ever. Right. Now, in the old days, it takes, you know, maybe 80% of the effort to create a demo, and then the rest for perfectionization is 20%. Now it's reversed. It's 20%, maybe 10 or 5%
- 29:54
- Unidentified speaker (speaker_2): of the effort to create a demo. But for that demo to go into production, it requires 80% of the work. or more. So that's the effect of AI.
- 30:05
- Unidentified speaker (speaker_1): Yeah. So it's just as difficult. The effort has just shifted to a slightly different plane for all the professionals involved. I mean, again, you you've listed out a few things there of like if you're a practitioner, particularly around risk, you're going to ask, like, okay, like what model is this using? How is has it being been built? What are the I hear a lot
- 30:25
- Unidentified speaker (speaker_1): about weights versus unweights. unweighted models, et cetera? You're going to ask about the evaluation. How have you made sure that this is correct and is going to stay correct? And then there's the piece that we typically talk a lot about in terms of connected intelligence, decision grade intelligence, or the beta foundations is another term I've heard. Is that
- 30:44
- Unidentified speaker (UNKNOWN): are
- 30:44
- Unidentified speaker (speaker_1): those all the things you need to make sure that an AI solution is delivered well? Like are those things that have to be true? Like you need to go through that checklist? Or that there are other things that you'd also include if somebody was evaluating a solution? Like what are the things you would say tick off before you progress the discussion?
- 31:02
- Unidentified speaker (speaker_2): Sure. I think the the most important thing is data. That's that's that right. So so AI at the end learns from data and utilize data to do its innovation. analytics and to provide the intelligence it has. So at the end, the most important part is the data. That what
- 31:22
- Unidentified speaker (speaker_2): data are we making certain decisions upon? There's always Google search. You can make any decision off Google search, and that search page might land on someone's, someone's blog post or some online rumors, or, God forbid, one of the internet forums. right? So there's there's data everywhere, but where is
- 31:42
- Unidentified speaker (speaker_2): the trusted data that we can use to base our information and decisions on? So that would be the first thing I'll be looking at. Where what's the source? Right. So uh in the internet meme, right? Uh source, right? People always ask for source or ask it, ask for like, hey, a where where is this from? Is this an original, like first?
- 32:02
- Unidentified speaker (speaker_2): party source or is it a secondary research, right? And those sources have different different different meanings, uh, especially if you work on the research side of things. Um, so that's the first thing that we should definitely look at and evaluate. What is the source? Are the data legitimate? Are they maintained? Can I trust this
- 32:22
- Unidentified speaker (speaker_2): data? And then the second step is rather than looking at the AI.
- 32:27
- Unidentified speaker (speaker_2): as like which agent you use, which model you use, we the the the second most important one is how do we evaluate the result?
- 32:38
- Unidentified speaker (UNKNOWN): So
- 32:38
- Unidentified speaker (speaker_2): a lot of times in the in the old days before Chat GPT, some people are selling a software, but not a decision, not a solution, not a not a result, not a service.
- 32:51
- Unidentified speaker (speaker_2): And nowadays with AI, a lot of times we are selling AI agents or prior pricing or selling or getting AI agents to do a specific job, to automate a specific task. Then what we should be looking at is can the agent or
- 33:11
- Unidentified speaker (speaker_2): does the agent actually automate the task in
- 33:14
- Unidentified speaker (UNKNOWN): the
- 33:14
- Unidentified speaker (speaker_2): a reasonable time? So there could be various statistical metrics or the various specimens, even business metrics we can look at, right? Let's say, for example, uh using a very simple one, is this agent accurate? What percentage accurate is the agent versus, let's say, human
- 33:34
- Unidentified speaker (speaker_2): workers? And how much time does it take for the agent to do? to to complete a task? How many tokens does it does it cost to to for the agent to complete a task? And how similar is the results between humans, experts, ex uh human
- 33:54
- Unidentified speaker (speaker_2): experts and the agents. So all of this are a lot more important than what dashboard do we have or what what which model we're using? Because at the end, what counts is the results. So we should evaluate the result and work backward from there. So results and
- 34:13
- Unidentified speaker (speaker_2): data. Like how do we evaluate? How do we evaluate and how and can the data be trusted? Those are the two most important things from it in in my perspective.
- 34:23
- Unidentified speaker (speaker_1): I've noted that down as check the data, check the E. eval or the evaluation methodology, and then finally check the total cost of ownership back to your earlier point, right? Like if this is going to do it really well with the right data, but it's going to cost you twice as much as your junior employee, then maybe the junior employee is still good.
- 34:42
- Unidentified speaker (speaker_1): Or senior employee, it depends on where this goes. Uh But at least if you check those three, you can figure out like, are you using AI for the right thing that's really going to move your business forward rather than just using it because it's AI, which um I think we might have been stuck in the last couple of years as a business world of load of mandates, just to use it because we wanted to figure out uh or we
- 35:02
- Unidentified speaker (speaker_1): trusted we'd figure out how to get the best of it later. Um
- 35:07
- Unidentified speaker (UNKNOWN): I
- 35:07
- Unidentified speaker (speaker_1): mean
- 35:08
- Unidentified speaker (speaker_1): there's one more question that actually I was gonna ask earlier and I didn't because I felt like we were on a good track, but just in the spirit of again helping our listeners use this in their day-to-day uh work, particularly if they're non-technology experts, there's a lot of terms, right? We've used model, we've talked about agent today, you talked about skill earlier.
- 35:28
- Unidentified speaker (speaker_1): Others that I've noted down here are tasks, context, harnesses, MCPs. Is it important for practitioners across their domains? or are they a supply chain practitioner or a compliance, say operations person, et cetera? Do they need to know these and be using them day to day? Because
- 35:48
- Unidentified speaker (speaker_1): I again, like to the old world of just normal software, nobody was really needing to understand the code. They just needed to understand the outcome. But as we get into this, this world, do Well, they, do they need this language, in your opinion, as somebody who who is going to be closer to it?
- 36:03
- Unidentified speaker (speaker_2): That's that's a very good question. So
- 36:07
- Unidentified speaker (UNKNOWN): I
- 36:08
- Unidentified speaker (speaker_2): don't think so.
- 36:09
- Unidentified speaker (UNKNOWN): I
- 36:09
- Unidentified speaker (speaker_2): know this might be uh going against the grain.
- 36:13
- Unidentified speaker (speaker_1): No, against the grain is what I like, Han.
- 36:15
- Unidentified speaker (speaker_2): Uh on the popular popular uh influencer pose, like yeah, there's the skills, there's MCP, there's the context, you know, you're you know, don't don't try to overload the context and whatnot. But the reality is, um on my personal use,
- 36:31
- Unidentified speaker (speaker_2): I don't care about any of those. Right. Do I need to understand how MCP works? What is it harnessed? You know, what is the context, which model it is? Not really. At the end of the day, I open up one of the apps that I use, right? That's a call claw desktop or OpenAI check.
- 36:50
- Unidentified speaker (speaker_2): ChatGPT codework, uh ChatGPT uh work or codecs. I open it up, I type in a command, you know, it I type in a command and did the AI did do the work for me properly. That's the most important thing, right? All of the surrounding, uh,
- 37:10
- Unidentified speaker (speaker_2): the underlying model, the agents, though of all of those can be. hidden underneath one abstraction below. So all of this can be abstracted away with a good product.
- 37:23
- Unidentified speaker (speaker_1): Hmm.
- 37:24
- Unidentified speaker (UNKNOWN): So
- 37:24
- Unidentified speaker (speaker_1): it's just another tool. Yeah.
- 37:26
- Unidentified speaker (UNKNOWN): It's
- 37:26
- Unidentified speaker (speaker_2): just another tool, right?
- 37:28
- Unidentified speaker (UNKNOWN): Yeah.
- 37:28
- Unidentified speaker (speaker_2): And and sometimes you if you feel it true, right? Like is say uh the way you talk to Siri, the way you talk to that say uh Alexa, versus nowadays, the way you talk to ChGB. ChatGPT, you you don't need to understand which model, which aging, which harness, and how to manage context. You know, talking
- 37:48
- Unidentified speaker (speaker_2): to Chat GPT is a lot less, a lot, a lot easier, or ChatGPT sounds, feels a lot more intelligent than Alexa from several years ago.
- 37:59
- Unidentified speaker (speaker_1): Yeah. I'm pleased you said it. I mean few. a months ago, I was starting to go, right? I
- 38:05
- Unidentified speaker (UNKNOWN): I
- 38:05
- Unidentified speaker (speaker_1): often someone who likes to read the source material. So you
- 38:08
- Unidentified speaker (UNKNOWN): know
- 38:08
- Unidentified speaker (speaker_1): you talked about something I like to go to the original. So I was like, I'm going to start reading like the help documentation of the different model providers. Because I need to like get all this stuff into my brain.
- 38:18
- Unidentified speaker (UNKNOWN): And
- 38:18
- Unidentified speaker (speaker_1): then increasingly, as like, you know, work picks up, you've got too many things to do, like too many other things in your brain. I was like, no, I'm just going to start using it and using it when I need it. And when I don't need it, trust that I have other ways of doing things. So what you've just said kind of gives me a bit of confidence. It does feel like as the productization matures, it's just you know, it's just
- 38:38
- Unidentified speaker (speaker_1): another set of capabilities. You will need people, right? That there needs to be, as ever, the people that know the technology in and out and can ask the right questions and evaluate. But they'll they are not likely to have the same expertise as the domain expertise. experts who just need to use the tool and know that you know which ones are they allowed to use. Um, just with uh an eye on the time,
- 38:58
- Unidentified speaker (speaker_1): um
- 38:59
- Unidentified speaker (UNKNOWN): I
- 38:59
- Unidentified speaker (speaker_1): would love for you just to point people to some of the writing that you do. As I said, I normally ask for recommended resources, but I'm gonna tell you which one to recommend this time, uh, which is your own writing.
- 39:08
- Unidentified speaker (UNKNOWN): Uh
- 39:09
- Unidentified speaker (speaker_1): would you maybe just uh let people know where they can see that and collect that and the types of things you're writing about at the moment?
- 39:15
- Unidentified speaker (speaker_2): Sure. So I am working on a couple of things right now. I have a book that's called AI Evaluation and Alignment, the seminal papers, which is a book about how do we evaluate AI agents and how do we align AI agents based on a few seminal papers in the field,
- 39:35
- Unidentified speaker (speaker_2): and that is up and coming with men's Minutes publishing. And I also keep my personal blog, which I write a lot. Uh I wish I could write more, but uh, but uh I I keep a a record on what I have, uh what's on my mind and what's more timely discussions
- 39:55
- Unidentified speaker (speaker_2): of some of the things that's that's happening on the AI technology side. And also I I regularly post things on LinkedIn. And I think, you know, going back to your point, I think one of the core reasons why we don't need to obsess too much with the underlying abstraction
- 40:15
- Unidentified speaker (speaker_2): is really the world of AI is shifting extraordinarily fast. And there are tons of different jargons and new words people in people invented. But I underneath the hood at the end, at the end of the day, it's how do we use AI to help us? The
- 40:35
- Unidentified speaker (speaker_2): AIs are built for humans and and we are. And so the best way to to learn is to use the AI and get a good feel of the boundary of its capabilities. And perhaps it will be better going forward It might go it it might be worse going forward as well. So,
- 40:55
- Unidentified speaker (speaker_2): you know, that developing that taste to see, all right, for this task, we can use AI. For that task, we shouldn't be using AI. I think that's a lot more important
- 41:05
- Unidentified speaker (speaker_2): than understanding what's MCP, what's the latest MCP key with release these from last week, um going Coin live. Right. And so at the end of the day, we don't want to uh spend five dollars of uh tokens to change a file name using cloud code.
- 41:22
- Unidentified speaker (speaker_1): Well said. Han, thank you so much. We'll get links for all of those things in the show notes. So if you're listening, I really encourage you to go click and either follow Han or buy the book or do all of those things. Um, I said it's been a really helpful reality check for me, Han. Just being excited about AI but not being carried away with the marketing
- 41:41
- Unidentified speaker (speaker_1): hype. So appreciate your time and your work and uh yeah, look forward to speaking to you again another time.
- 41:46
- Unidentified speaker (speaker_2): Thank you so
- 41:47
- Unidentified speaker (UNKNOWN): much.
- 41:55
- Unidentified speaker (speaker_0): Thanks for listening to this Moody's Talks podcast.
- 41:59
- Unidentified speaker (UNKNOWN): To
- 42:00
- Unidentified speaker (speaker_0): find out more about the topics discussed, please follow the links in the show notes. You can check out other Moody's Talks podcasts by visiting Moody's.com/slash podcasts.