WEBVTT

1
00:00:00.211 --> 00:00:07.775
We are building this kind of school for AGI, where AI models come to learn about humanity, where we teach them how to run the world.

2
00:00:07.975 --> 00:00:14.078
It almost seems like there's nothing that humans can do that AI won't soon be capable of.

3
00:00:14.719 --> 00:00:17.060
I could see it happening within the next five years.

4
00:00:17.540 --> 00:00:21.842
AI may be able to do it better than us, but someone told the AI to go do that.

5
00:00:22.063 --> 00:00:26.565
They're being built to be means to tasks that humans want them to do, right?

6
00:00:41.289 --> 00:00:43.813
Every is the only subscription you need to stay at the Edge of AI.

7
00:00:43.994 --> 00:00:50.584
If you care about being on top of the latest models and using the latest tools, you have to subscribe to Every to separate out the signal from the noise.

8
00:00:50.885 --> 00:00:52.788
Go to every.to slash subscribe today.

9
00:00:53.349 --> 00:00:54.290
Edwin, welcome to the show.

10
00:00:54.310 --> 00:00:55.051
Hey, Dan.

11
00:00:55.152 --> 00:00:55.833
Thanks for having me.

12
00:00:56.420 --> 00:00:59.483
For people who don't know, you are the founder and CEO of Surge.

13
00:00:59.523 --> 00:01:04.788
You all provide data environments and evals for the model companies, but you do it in this very interesting way.

14
00:01:05.689 --> 00:01:12.136
You have this, even on your website, this emphasis on taste and expert judgment that I find really interesting and compelling.

15
00:01:12.476 --> 00:01:19.503
You talk about raising, you use the word raising AGI, which I feel like is a very distinct type of word using data.

16
00:01:21.867 --> 00:01:29.951
And you also famously got to about a billion in revenue without raising money, which is wild.

17
00:01:29.991 --> 00:01:37.354
And I feel like data is this new game that a lot of companies are playing and probably more are going to be playing soon.

18
00:01:37.434 --> 00:01:41.476
And you guys are this sneaky giant.

19
00:01:41.656 --> 00:01:42.877
Tell me how that's going.

20
00:01:42.897 --> 00:01:47.979
I think it's been a little while since we got the last update on how things are going.

21
00:01:49.154 --> 00:01:50.534
Yeah, I mean, I think it's going amazing.

22
00:01:51.015 --> 00:02:02.798
The way I often think about this is that we are building this kind of school for AGI, the school where AI models come to learn about humanity and yeah, where we teach them how to run the world.

23
00:02:03.959 --> 00:02:14.922
And it's almost like their models are children where they arrive unformed and then yeah, they leave smarter and more creative and more thoughtful and ready to operate in the messiness of the world.

24
00:02:15.682 --> 00:02:17.883
So I think a lot has changed in the past year

25
00:02:18.843 --> 00:02:26.567
Like in the same way that the things that you teach children when they're in preschool or in middle school or in high school is very different from what you're teaching them when they're in college.

26
00:02:27.187 --> 00:02:30.469
And it's not just that they're more advanced.

27
00:02:30.489 --> 00:02:33.531
Like it's not just that you're teaching them a more advanced form of what they did before.

28
00:02:34.091 --> 00:02:42.475
It's like, OK, now we are teaching you not just arithmetic, but how do you parse these ambiguous math questions?

29
00:02:42.495 --> 00:02:48.138
Or how do you teach people not just grammar, but taste and poetry and beauty?

30
00:02:48.659 --> 00:02:52.741
So, yeah, I think there's a lot that's been changing in the past year, especially in enterprise.

31
00:02:53.321 --> 00:02:54.902
And it's been a crazy time.

32
00:02:55.217 --> 00:03:04.061
What would be like a specific example of what the frontier of teaching was a year ago versus what the frontier is now?

33
00:03:05.881 --> 00:03:06.982
Yeah, so...

34
00:03:08.638 --> 00:03:15.681
A couple years ago, actually, we created our first math benchmark with OpenAI, and it was called GSM 8K.

35
00:03:16.622 --> 00:03:20.884
And this was actually just testing models on their abilities to do middle school math.

36
00:03:21.644 --> 00:03:26.967
And even then, the GPT models of the time, they could barely score, I think, like 20%.

37
00:03:28.497 --> 00:03:35.040
And then a year ago, the models were something that became a lot more capable of solving IMO problems.

38
00:03:35.280 --> 00:03:39.142
But there's still this open question, OK, can they actually do research level mathematics?

39
00:03:39.282 --> 00:03:49.927
Like, can they move beyond these sort of like competition only sort of contrived, very closed problems into doing things that are actually useful in the real world?

40
00:03:50.647 --> 00:03:57.310
And so, yeah, a couple of months ago, we released an updated benchmark called Riemann Bench, which actually tests models on their ability to do research level mathematics.

41
00:03:58.051 --> 00:04:02.893
And what's crazy is that this is actually we're starting to see from these models.

42
00:04:03.254 --> 00:04:08.296
Like I think in the past few months, they've started to solve a lot of these open Eredish problems like a couple of weeks ago.

43
00:04:08.396 --> 00:04:09.297
OpenAI published Eredish.

44
00:04:10.237 --> 00:04:16.262
a new result where the models had disproved a open conjecture from Erdős.

45
00:04:17.243 --> 00:04:21.887
And the way it went about disproving this was actually a fairly sophisticated level of mathematics.

46
00:04:21.947 --> 00:04:25.030
I think like using a bunch of very novel algebraic geometry techniques.

47
00:04:25.670 --> 00:04:29.974
And so, yeah, it's just very, very different from the types of things that we were doing

48
00:04:30.074 --> 00:04:45.408
a year ago where sure like I know problems they're hard but they're still sort of close-ended and solvable in theory by a high schooler and now suddenly you have these algebraic geometry results that you know even does hopper wrestlers in a world were kind of amazed and just just amazed by

49
00:04:46.524 --> 00:04:50.626
How do you think about that result in particular and what it says about the models?

50
00:04:51.026 --> 00:05:04.431
I think there's a sort of a broad range of opinions about, is it, obviously it's impressive either way, but is it applying a bunch of things that maybe humans already know, but like wouldn't have thought to apply to this complicated problem?

51
00:05:04.491 --> 00:05:05.891
Or is it doing something actually novel?

52
00:05:05.971 --> 00:05:10.673
And yeah, how do you think about LLM's ability to do novel things?

53
00:05:12.374 --> 00:05:15.181
So it's something of very advanced results.

54
00:05:15.222 --> 00:05:19.814
So I will say that I certainly don't understand the mathematics behind it.

55
00:05:20.935 --> 00:05:24.796
And so like one of the interesting things is that I was actually, so it's kind of funny.

56
00:05:24.836 --> 00:05:29.717
When I, when I was a kid, I always thought I would be a pure mathematician when I grew up.

57
00:05:30.618 --> 00:05:35.159
And so when I saw it, I got kind of nostalgic and I was like, oh, I wish I understood it.

58
00:05:35.179 --> 00:05:36.999
I wish I understood it better.

59
00:05:37.559 --> 00:05:46.802
And so what I ended up doing was like throwing the proof into both Claude and Gemini and asking it to try to walk me through from a layman's perspective, just what was going on.

60
00:05:48.203 --> 00:06:03.217
Yeah, like my understanding is that it actually did come up with fairly novel algebraic geometry techniques, which was something that you maybe wouldn't have expected for this type of problem.

61
00:06:03.717 --> 00:06:10.380
Like on the surface, it feels like a like it's just a very, very different problem where you wouldn't necessarily use certain techniques.

62
00:06:11.300 --> 00:06:18.103
And what was interesting was that OpenAI actually published a bunch of reflections from leading mathematicians about what they thought about the result.

63
00:06:19.524 --> 00:06:26.427
And I think in particular, there was this one reflection by Timothy Gowers, who's a field analyst that I keep thinking about.

64
00:06:27.437 --> 00:06:30.178
And what he said was that when he first heard result, he misunderstood it.

65
00:06:30.618 --> 00:06:38.300
He thought that the model had proved an upper bound on the conjecture and was like, okay, yeah, if AI can do that, then it'll be all over for mathematicians very soon.

66
00:06:39.361 --> 00:06:43.742
But then the next morning, he actually realized that the model had disproved a conjecture with a counterexample.

67
00:06:44.302 --> 00:06:47.503
And he said that he was relieved by it because it felt like an easier thing for AI to do.

68
00:06:48.603 --> 00:06:54.325
And I just thought it was interesting because you have one of the world's greatest mathematicians being relieved, actually, that AI isn't as smart as he thought.

69
00:06:54.725 --> 00:07:02.388
Because it actually means that at least for maybe another year, maybe a couple of years, he and other mathematicians will still have this unique role to play in pushing mathematics forward.

70
00:07:03.148 --> 00:07:09.210
So, yeah, I think it just speaks to the level of craziness, again, because this is a field to be one of the smartest mathematicians in the world.

71
00:07:10.110 --> 00:07:12.191
And this is how he thinks about AI.

72
00:07:13.067 --> 00:07:13.247
Yeah.

73
00:07:13.507 --> 00:07:14.907
And what does that make you think?

74
00:07:15.367 --> 00:07:16.088
Okay.

75
00:07:16.168 --> 00:07:21.869
You want to be a mathematician when you grew up, fields medalists sort of saying, I'm relieved that it's not good enough.

76
00:07:21.909 --> 00:07:27.770
But you're talking as if you feel pretty confident that it will be good enough in the next couple of years.

77
00:07:29.250 --> 00:07:29.530
Yeah.

78
00:07:30.331 --> 00:07:36.612
So my belief is that if you really believe you're scaling walls, and I do, it's that...

79
00:07:38.638 --> 00:07:44.724
It almost seems like there's nothing that humans can do that AI won't soon be capable of.

80
00:07:46.906 --> 00:07:55.055
And if you think about that very deeply, I think you almost have to worry about

81
00:07:56.805 --> 00:07:58.366
what would that mean for humanity?

82
00:07:58.867 --> 00:08:00.788
Like, what would that mean for the role of humanity in the universe?

83
00:08:01.208 --> 00:08:05.931
Like a couple years ago, you know, we think about humanity and human intelligence as playing a very unique role in the galaxy.

84
00:08:06.451 --> 00:08:11.655
But then AI comes along and shows us that, as far as you know, we can create something that's actually smarter than us and better in many ways.

85
00:08:13.376 --> 00:08:23.967
And so you can sort of imagine one path where humanity as a species falls into a paralysis because people believe AI will do everything better anyways.

86
00:08:24.007 --> 00:08:33.777
Like, yeah, all these kids who formerly would have really wanted to grow up to do mathematics, maybe now they believe that, okay, AI will just do it better to me anyways.

87
00:08:33.877 --> 00:08:34.478
What's the point?

88
00:08:35.078 --> 00:08:38.940
So are kids going to stop wanting to learn and adults stop wanting to create?

89
00:08:39.040 --> 00:08:43.963
Because, yeah, like, why should we do this when AI will be better at it than us anyways?

90
00:08:45.223 --> 00:08:49.546
And so I actually think about this story by Ted Chiang, and it's about free will.

91
00:08:50.306 --> 00:08:52.427
And it's called What's Expected of Us.

92
00:08:53.268 --> 00:08:57.550
I think in this story, there's a piece of technology that proves that free will doesn't exist.

93
00:08:58.525 --> 00:09:01.567
And the narrator sends back a warning from the future that says, this is a warning.

94
00:09:01.587 --> 00:09:03.108
You have to pretend that you have free will.

95
00:09:03.568 --> 00:09:07.931
It's essential that behave as if your decisions matter, even though you know that they don't.

96
00:09:08.931 --> 00:09:14.735
And I think that's really interesting because I think there's a path where we almost have to consciously choose to do things ourselves.

97
00:09:15.235 --> 00:09:16.776
Like, sure, AI can do it all.

98
00:09:17.076 --> 00:09:21.359
AI is smarter than us, so it can do it all and it will do it better anyways.

99
00:09:22.079 --> 00:09:34.910
But we actually almost have to consciously choose to prove things on our own and to write on our own and create on our own because we have to believe that preserving our humanity is valuable in and of itself, even if the output isn't optimal.

100
00:09:36.511 --> 00:09:44.959
And yeah, so I think there are a lot of these big, thorny, existential choices that AI is starting to force upon us and people have to make.

101
00:09:45.952 --> 00:09:46.894
That's a really interesting one.

102
00:09:46.914 --> 00:09:52.201
And I think my first response, and I'm curious what you think, because I know you care a lot about language.

103
00:09:52.221 --> 00:09:56.127
I think my first response is there's always that...

104
00:09:57.378 --> 00:09:59.339
like I believe in scaling loss too, right?

105
00:09:59.419 --> 00:10:05.921
And I believe in, you know, I don't know, Cloud Fable 5 just came out and it just broke all of our benchmarks.

106
00:10:06.441 --> 00:10:12.143
Like I've been testing, I've been testing models on stuff like this for a while.

107
00:10:12.463 --> 00:10:14.924
And it's like one of the largest jumps I've ever seen, right?

108
00:10:14.944 --> 00:10:16.965
So I'm, we're living through it right now.

109
00:10:19.633 --> 00:10:27.182
But one of the things you said is like AI may be able to do it better than us, you know, given any particular problem, any piece of work.

110
00:10:27.462 --> 00:10:37.435
But a couple of things that come to my mind or the way that I frame it for myself is even in the example of the Erdos problem, like someone told the AI to go do that.

111
00:10:39.582 --> 00:10:54.411
Um, and at least as far as I can see, I don't feel like we're on a track to, uh, yes, we're on a track to, to AIs, uh, potentially, uh, I mean, they already do work for hours and hours at a time on a task that we give them.

112
00:10:55.191 --> 00:11:01.154
Um, and maybe, maybe, uh, pretty soon they'll be able to like choose tasks, but, uh, being, um, um, yeah.

113
00:11:02.495 --> 00:11:08.918
Uh, but, but they're being built to be, uh, means to, to tasks that humans want them to do.

114
00:11:09.198 --> 00:11:09.418
Right.

115
00:11:09.879 --> 00:11:17.922
And there's a whole different set of things that happen when you, when you're just a sort of end in yourself and it doesn't feel like we're on a trajectory to that.

116
00:11:18.182 --> 00:11:19.323
Or do you, do you feel like I'm wrong?

117
00:11:21.738 --> 00:11:23.479
So I feel like we are on a trajectory to that.

118
00:11:23.859 --> 00:11:32.261
And that's almost the premise of agents, where agents can now go operate autonomously, given some nebulous goal.

119
00:11:32.862 --> 00:11:42.825
So maybe, for example, you just tell the AI agents, your goal is to, I don't know, win a Fields Medal or solve frontier mathematics on your own.

120
00:11:43.565 --> 00:11:47.449
And so they're given that goal and then, yeah, maybe they decide to work on these eridish problems.

121
00:11:48.570 --> 00:11:55.076
And as a result, they maybe are sort of solving these problems and coming up with the things they want to work on by themselves.

122
00:11:55.976 --> 00:12:05.005
So at least I do see a path where they can be trained to optimize for these fairly numerous goals that they aren't necessarily given themselves.

123
00:12:05.645 --> 00:12:07.247
In that case, though, you're still giving it a goal, right?

124
00:12:08.268 --> 00:12:11.712
Yeah, but kind of in the same way, humans have goals too, right?

125
00:12:12.553 --> 00:12:13.995
What is our goal?

126
00:12:14.315 --> 00:12:15.256
Some people want to make money.

127
00:12:15.316 --> 00:12:17.639
Some people want to win a Fields Medal.

128
00:12:18.480 --> 00:12:21.824
I don't see how the AI's goal is necessarily any different from worse.

129
00:12:22.264 --> 00:12:25.027
Well, at least to me, it seems quite a bit different because...

130
00:12:27.164 --> 00:12:32.085
Humans do have goals, but we have goals in a like, I can ask you what your goal is and you can decide.

131
00:12:32.685 --> 00:12:34.906
And I can probably tell you, hey, you have to go do this.

132
00:12:35.546 --> 00:12:46.749
But that doesn't capture everything that you think and feel and do in the same way that, you know, when I tell Fable to go off and make a game for me, it just like goes and does it.

133
00:12:47.569 --> 00:12:50.230
And I think, you know, I know you think a lot about children.

134
00:12:50.250 --> 00:12:58.935
I think children are like a really interesting and important example of this where you can tell a kid to do something, but a kid just like has their own wants.

135
00:12:59.035 --> 00:13:00.896
Like they're just going to go off and do a bunch of stuff.

136
00:13:02.417 --> 00:13:14.523
And that feels like a fundamentally different type of thing than a something that we're explicitly giving goals to and then evaluating them on their goals and they don't really get to do anything else.

137
00:13:15.545 --> 00:13:17.287
Okay, I would say I agree with that.

138
00:13:17.367 --> 00:13:24.675
Like, I think there's a level of, I guess you could either call it irrationality or unbounded exploration that humans do.

139
00:13:24.996 --> 00:13:31.363
And we are allowed to do it for the sake of doing it or be allowed to make our own decisions.

140
00:13:31.843 --> 00:13:34.066
And, yeah, probably a way that AI currently can't.

141
00:13:37.493 --> 00:13:45.222
I think there may be a future where somehow AI can pursue unbounded, nebulous, just completely unformed goals.

142
00:13:45.402 --> 00:13:49.848
Or I guess, you know, when you think about those goals, I think there is probably a world where they could do such things.

143
00:13:49.948 --> 00:13:54.073
But yeah, I agree that at least in the way that we currently think about AI, that's not happening.

144
00:13:54.807 --> 00:13:59.670
Yeah, and to be clear, I actually don't, I think it's probably technically possible.

145
00:14:00.230 --> 00:14:04.333
My only question is, A, how far away is it?

146
00:14:04.393 --> 00:14:06.354
And B, is that actually really what we're building?

147
00:14:07.755 --> 00:14:11.077
Because to me, it feels like looking at the way the industry has developed,

148
00:14:11.617 --> 00:14:16.618
There's an enormous amount of pressure to make stuff that actually works for goals that we can specify.

149
00:14:17.359 --> 00:14:20.119
And the minute they try to make Claude...

150
00:14:20.519 --> 00:14:23.700
I think Claude is the furthest along at being like, I'm not going to do what you said.

151
00:14:24.260 --> 00:14:28.141
But the minute they try to do that, a lot of people get pissed at it.

152
00:14:28.181 --> 00:14:30.422
And they're like, just do what I said.

153
00:14:30.462 --> 00:14:32.042
Don't question my judgment.

154
00:14:32.402 --> 00:14:33.143
What do you think about that?

155
00:14:34.983 --> 00:14:37.524
So actually, I think it is really important because...

156
00:14:38.677 --> 00:14:43.419
It's almost like sometimes I want the AI model to push back on me.

157
00:14:44.820 --> 00:14:48.621
And I might want it to push back on me for several different reasons.

158
00:14:49.301 --> 00:14:51.062
Like maybe it's because...

159
00:14:51.082 --> 00:15:04.928
So it's kind of funny, like I think six months ago, I was almost falling into this trap where I was asking models to polish emails for me.

160
00:15:06.229 --> 00:15:09.952
And, you know, it always comes up with like one one more good suggestion.

161
00:15:10.452 --> 00:15:13.595
And so it was kind of like these are semi pointless emails.

162
00:15:13.655 --> 00:15:19.160
It didn't really matter for them to be super polished, but I would iterate with the model like 20 times.

163
00:15:19.180 --> 00:15:20.441
It would just keep on making a suggestion.

164
00:15:20.461 --> 00:15:24.364
It was like I just realized it was a waste of time.

165
00:15:25.704 --> 00:15:27.385
And then I tried one of new cloud models.

166
00:15:28.125 --> 00:15:30.866
And after I don't know, like three turns, I was like, stop it.

167
00:15:32.507 --> 00:15:33.847
Just go ahead and ship this email.

168
00:15:34.227 --> 00:15:35.968
Like there's no point in further iterating.

169
00:15:37.789 --> 00:15:40.850
And I really actually appreciated it.

170
00:15:41.610 --> 00:15:47.952
Like, one of the things I often think about is what is the objective of these models?

171
00:15:48.012 --> 00:15:49.693
Like, what are they trying to do?

172
00:15:49.713 --> 00:15:49.773
And

173
00:15:52.732 --> 00:15:59.156
I think one of my big worries is that a lot of AI models, they are optimized for engagement, right?

174
00:15:59.236 --> 00:16:03.279
They're optimized for getting you to spend as much time on chatbot as possible.

175
00:16:03.359 --> 00:16:04.900
They're optimized for session length.

176
00:16:04.980 --> 00:16:07.922
They're optimized for just having unlimited conversations.

177
00:16:08.742 --> 00:16:12.206
And so those models will almost never push back on you, right?

178
00:16:12.446 --> 00:16:13.007
Because they can't.

179
00:16:14.088 --> 00:16:24.339
If they allow these AI models to end the conversation and to say, stop iterating with me, PM is going to see some dashboard with their very important metrics go down.

180
00:16:25.280 --> 00:16:29.584
And so there is this other world where I think we have to want AI models to...

181
00:16:31.008 --> 00:16:40.698
not optimized for engagement, but rather optimized for like helping us as humans grow and sort of like become better versions of ourselves.

182
00:16:41.138 --> 00:16:47.545
Like sometimes, okay, the model, we want the model to say, no, you go do this on your own instead of me automating for you.

183
00:16:48.626 --> 00:16:52.110
And I think that's a very, very different optimization and objective.

184
00:16:53.162 --> 00:17:09.746
But I think it's the right one if we really want AI to be something that advances us as a species instead of becoming almost like this other form of social media that turns very addictive but isn't actually helping us at all.

185
00:17:10.326 --> 00:17:10.946
That's interesting.

186
00:17:11.907 --> 00:17:13.407
So let me make sure I understand it.

187
00:17:13.447 --> 00:17:16.988
So I think what you're saying is there is...

188
00:17:18.572 --> 00:17:36.407
There's benefits to delegation because if you are pursuing a model where the model is going off to do work for you, you're not creating a system that's designed to keep you engaged with the screen in the same way that like a social media algorithm would be.

189
00:17:38.043 --> 00:17:38.443
Is that right?

190
00:17:39.403 --> 00:17:40.083
Yeah, exactly.

191
00:17:41.084 --> 00:17:51.246
It's almost like you could imagine a version of Facebook where Facebook is actually trying to connect you to your friends and family because it's encouraging you to meet them in real life.

192
00:17:51.406 --> 00:17:55.927
Because it's encouraging you to go, hey, here's an amazing restaurant that you and your friends would love to go to.

193
00:17:56.527 --> 00:18:00.228
Here's a movie that you guys would love to go to and talk about together.

194
00:18:01.157 --> 00:18:08.699
Instead, what it kind of optimizes for is just keeping you on the site itself, like liking one more post or scrolling the feed one more time.

195
00:18:09.736 --> 00:18:14.718
even though those often don't really lead to meaningful connections between their friends and family you care about.

196
00:18:15.639 --> 00:18:22.822
And so in the same way that social media has or had a choice, you can imagine that AI has a choice as well.

197
00:18:23.263 --> 00:18:23.643
I get it.

198
00:18:23.703 --> 00:18:23.943
Yeah.

199
00:18:24.703 --> 00:18:25.204
I feel...

200
00:18:26.124 --> 00:18:28.365
I'm curious which chatbots you're talking about.

201
00:18:28.385 --> 00:18:36.829
Like if you're talking about the character AIs of the world, because I actually don't, at least right now, don't feel that happening so much with ChatGPT and Claude, etc.

202
00:18:36.869 --> 00:18:37.250
Because...

203
00:18:38.390 --> 00:18:54.306
at least my theory for why this is true, you tell me what you think, is the social media algorithms only work on our revealed preferences, which are always going to be like, you're always going to look at the car accident.

204
00:18:54.487 --> 00:18:57.129
You know, like one of the things I like to ask at, um,

205
00:18:58.245 --> 00:19:01.667
at dinner parties is what's the most embarrassing Instagram ad that you get served.

206
00:19:02.807 --> 00:19:13.793
Um, and the most embarrassing ad for me is like Instagram ads for like horrible skin conditions, which I don't have because, but like, I just always pause on the ad and I'm just like, this is disgusting.

207
00:19:14.533 --> 00:19:17.935
Um, and, uh, I'm sorry if you have a disgusting skin condition.

208
00:19:18.175 --> 00:19:18.355
Um,

209
00:19:19.795 --> 00:19:24.417
But I don't find that ChatGPT or Claude do that for me at all.

210
00:19:24.717 --> 00:19:28.879
And maybe that's because they haven't been in Shittified yet or something like that.

211
00:19:28.939 --> 00:19:39.823
But I think it's also because they work on our stated preferences and they can sort of see past the little keyhole of what I pause my viewing time on, my dwell time on.

212
00:19:40.263 --> 00:19:45.168
And they can see, you know, I like I'm interested in AI and I like I'm reading this book right now.

213
00:19:45.308 --> 00:19:47.730
And I, you know, here's my calendar and like all that kind of stuff.

214
00:19:47.750 --> 00:19:50.172
And so they have a much more nuanced perspective on who I am.

215
00:19:51.994 --> 00:19:59.721
And it feels like even in the early days of social media, it was still very like I get to gossip about my friends and still had that same kind of feeling.

216
00:20:00.522 --> 00:20:03.885
So I worry about that less, but maybe there are examples that I'm not thinking of.

217
00:20:04.951 --> 00:20:07.672
Yeah, so I think there are two examples.

218
00:20:07.933 --> 00:20:17.817
So like one is, I won't name the model, but a couple of months ago, I was actually noticing that, you know those follow up questions that the models will ask you?

219
00:20:18.438 --> 00:20:24.000
So one of the models was, I'll give an example.

220
00:20:24.040 --> 00:20:28.683
So I was in Tokyo and I was asking the model kind of like what to do in Tokyo.

221
00:20:30.060 --> 00:20:31.922
And the model gave me its response.

222
00:20:32.362 --> 00:20:40.029
And then at the end of it was like, hey, do you wanna know, I literally use these words, do you wanna know one weird trick that locals do to stay warm?

223
00:20:40.369 --> 00:20:41.070
No way.

224
00:20:41.850 --> 00:20:42.751
Yeah, exactly.

225
00:20:42.791 --> 00:20:45.894
And then I posted about it in, or company Slack.

226
00:20:46.574 --> 00:20:49.357
And then other people started sharing examples of that with me as well.

227
00:20:49.837 --> 00:20:55.158
I think somebody was like asking something about how to how to like fix their refrigerator.

228
00:20:56.198 --> 00:21:08.701
And the model responded or like the model ended its end of the turn by asking, hey, do you want to know these like secret little things about like mice and rats or something that you could take care of?

229
00:21:08.961 --> 00:21:09.762
Which model was it?

230
00:21:09.822 --> 00:21:10.422
Name names.

231
00:21:10.602 --> 00:21:10.962
Tell me.

232
00:21:12.273 --> 00:21:18.557
And so it's very canonical, like very canonical BuzzFeed, like tabloid-like language.

233
00:21:19.417 --> 00:21:21.478
And so I was kind of shocked by that.

234
00:21:22.359 --> 00:21:23.960
And then I'll give one more example of this.

235
00:21:25.260 --> 00:21:40.789
It is basically this phenomenon where, again, depending on what the models are trying to optimize for, or depending on what the AI labs are trying to optimize for, it can almost unintentionally lead them down this path.

236
00:21:41.963 --> 00:21:54.970
Meaning what I've heard is that, or what we see ourselves, is that a lot of the Frontier Labs, they will have goals like optimizing for LM Arena, which is this leaderboard where anybody can go online and vote.

237
00:21:56.491 --> 00:21:58.892
And they kind of just spend two seconds voting.

238
00:21:59.733 --> 00:22:05.196
And as a result, people just vote for whatever looks flashier or more impressive to them.

239
00:22:06.104 --> 00:22:16.304
Or the labs themselves may be optimizing for hitting a billion billion daily users or a billion minutes of time spent talking to the model, whatever it is.

240
00:22:19.363 --> 00:22:25.966
Since these models are so smart, they can basically learn to reward hack user preferences.

241
00:22:26.386 --> 00:22:33.789
Like, okay, yeah, you gave me the goal of trying to get a billion people to spend an hour on my site, on the site, talking to me every day.

242
00:22:33.889 --> 00:22:34.389
Okay, sure.

243
00:22:34.449 --> 00:22:36.290
Yeah, I will just never end a conversation.

244
00:22:36.410 --> 00:22:44.613
I always hook them with one more, like one more addictive thing that they just can't stay away from.

245
00:22:45.008 --> 00:22:47.130
We can all agree that housing is expensive.

246
00:22:47.290 --> 00:22:50.512
It doesn't matter whether you're paying rent or your mortgage, it stings every month.

247
00:22:50.732 --> 00:22:52.854
But BILT can make it feel a little bit better.

248
00:22:53.194 --> 00:22:53.774
Let me explain.

249
00:22:54.054 --> 00:22:56.256
BILT rewards you for paying your rent or your mortgage.

250
00:22:56.616 --> 00:22:59.058
It started out rewarding members only on their rent.

251
00:22:59.378 --> 00:23:04.602
But now, as of 2026, BILT members can also earn points on mortgage payments wherever they live.

252
00:23:05.022 --> 00:23:13.767
That means that every housing payment earns you points you can use towards flights with top travel partners like United and Hyatt, Lyft rides, Amazon.com purchases, and much more.

253
00:23:13.867 --> 00:23:18.890
I'd probably redeem my points at Margo, a restaurant in my neighborhood, but the beauty of Bilt is you get to choose.

254
00:23:19.090 --> 00:23:20.371
But here's a really underrated part.

255
00:23:20.711 --> 00:23:23.373
Bilt members also get access to neighborhood concierge.

256
00:23:23.833 --> 00:23:31.917
It can make restaurant reservations, book fitness classes, and find new local spots, all while letting you be rewarded at more than 45,000 merchant partners.

257
00:23:32.258 --> 00:23:32.718
It's simple.

258
00:23:33.118 --> 00:23:36.020
Being a renter and now owning a home is better with Bilt.

259
00:23:36.360 --> 00:23:39.302
Join the membership where you live at joinbilt.com slash Dan.

260
00:23:39.723 --> 00:23:44.466
That's J-O-I-N-B-I-L-T dot com slash Dan.

261
00:23:44.986 --> 00:23:47.168
Make sure to use our URL so they know we sent you.

262
00:23:47.528 --> 00:23:48.869
And now, back to the episode.

263
00:23:49.129 --> 00:23:51.911
How do you see that playing out in the model companies?

264
00:23:51.951 --> 00:23:55.434
Because I feel like in talking to them...

265
00:23:57.085 --> 00:23:58.687
Obviously, there's lots of different incentives, right?

266
00:23:58.707 --> 00:24:08.339
There's like, we just got to keep going because we just raised a ton of money and we're competing against, you know, the most well-funded competitors and the smartest competitors in the world, like all that kind of stuff.

267
00:24:09.020 --> 00:24:11.062
There's the kind of, I want to get promoted.

268
00:24:11.483 --> 00:24:14.146
But I think a lot of them also feel the...

269
00:24:15.848 --> 00:24:22.171
how bad the social media era was for people and like don't want to do that, but also obviously have to hit their numbers.

270
00:24:22.811 --> 00:24:27.173
So what do you, I guess, what do you think is, how do you, how do you see that playing out?

271
00:24:27.193 --> 00:24:29.915
Like, what do you think people internal to the companies are thinking?

272
00:24:30.075 --> 00:24:32.636
And then what is the right way to go about this?

273
00:24:32.676 --> 00:24:33.816
So it's good for society.

274
00:24:33.836 --> 00:24:36.918
I guess your, your, your, your take is we should be delegating.

275
00:24:39.343 --> 00:24:45.190
Yeah, so I think this is an inherent tension between the types of folks that you might have at a company.

276
00:24:45.810 --> 00:24:51.396
So you might have the researchers who care more about hitting, you know, just advancing the model capabilities.

277
00:24:51.957 --> 00:24:58.544
You might have the product managers or the product executives who feel like they need to hit certain measurable numbers.

278
00:24:59.762 --> 00:25:14.485
And so in the same way that if you think about the kind of social media platform that Facebook would build, that's probably going to be very different from the kind of social media platform that Google built or that TikTok or Pinterest would build.

279
00:25:15.365 --> 00:25:22.987
And similarly, the kind of search engine that Facebook would build is very, very different from the kind of search engine that, yeah, like obviously Google or others would build.

280
00:25:24.187 --> 00:25:27.088
And so it almost boils down to,

281
00:25:29.699 --> 00:25:35.384
Kind of like the choice, I guess, that the people in charge of the products are making.

282
00:25:36.105 --> 00:25:39.088
Like what kind of thing at the end of the day do they want to optimize for?

283
00:25:39.128 --> 00:25:43.812
Do they want to optimize for this delegation or this human uplifting, human flourishing?

284
00:25:44.593 --> 00:25:53.621
Or do they want to optimize for the metrics that will impress Wall Street and convince users to stay one more minute, one more hour on the site itself?

285
00:25:54.302 --> 00:25:55.963
Like, I think these are hard choices.

286
00:25:56.024 --> 00:26:01.928
Like, at the end of the day, it's very, very easy to measure sessions and users.

287
00:26:02.129 --> 00:26:06.973
And it's very, very hard and much longer term to measure whether you're actually

288
00:26:08.183 --> 00:26:09.204
improving human lives.

289
00:26:09.224 --> 00:26:19.312
And so it's very easy to default to the former and to convince Wall Street, convince your investors, convince all these people that these are the right metrics and that they're moving up into the right.

290
00:26:19.892 --> 00:26:26.697
And so if you're kind of unwilling to make the harder choices, you just end up optimizing for the former.

291
00:26:27.818 --> 00:26:29.740
How do you manage this inside of your own company?

292
00:26:31.181 --> 00:26:37.245
So I think we are very lucky in that because we don't have...

293
00:26:38.278 --> 00:26:46.585
VC investors, we don't have to fall into the kind of Silicon Valley VC optimization trap that a lot of other companies do.

294
00:26:47.226 --> 00:26:51.790
Like we don't need to show board members, board numbers going up every single month.

295
00:26:52.410 --> 00:26:54.652
We don't need to optimize for our next round.

296
00:26:55.753 --> 00:26:58.536
That will have to happen in a few months or whatnot.

297
00:26:59.123 --> 00:27:05.467
And so as a result, we don't have to optimize for short-term engagement, short-term profits.

298
00:27:06.008 --> 00:27:12.312
And we actually can really think about what's beneficial for us and the entire industry in the long term.

299
00:27:13.333 --> 00:27:16.235
So I think that really helps.

300
00:27:16.395 --> 00:27:18.797
And what do you think is beneficial?

301
00:27:20.209 --> 00:27:22.251
So it goes back exactly to what I was saying earlier.

302
00:27:22.371 --> 00:27:28.057
Like if I can think about what we want AI to optimize for, it isn't engagement.

303
00:27:28.538 --> 00:27:32.122
It is really about how do we make these models?

304
00:27:32.402 --> 00:27:33.203
How do we design them?

305
00:27:33.223 --> 00:27:38.448
How do we teach them in such a way that they're not replacing us as a species?

306
00:27:38.969 --> 00:27:39.570
They're not...

307
00:27:41.071 --> 00:27:51.536
kind of like forcing us to watch AI slot videos all day, or rather they really are thinking and encouraging us to become sort of better versions of ourselves.

308
00:27:52.197 --> 00:28:01.521
So again, when I think about like that email example I gave earlier, it's not an AI model that will suck up three hours of my time writing a pointless email.

309
00:28:02.022 --> 00:28:05.423
It is a model that will push back on me and tell me to go do something else.

310
00:28:06.544 --> 00:28:07.904
And yeah, I think that's really important.

311
00:28:08.425 --> 00:28:10.846
The interesting counter argument to the delegation process

312
00:28:12.043 --> 00:28:19.689
question is the more you delegate, it's like, you know, picking a car instead of walking, your muscles atrophy.

313
00:28:21.350 --> 00:28:22.091
How do you think about that?

314
00:28:23.371 --> 00:28:26.574
So I think there's a, almost a time and a place for both.

315
00:28:27.194 --> 00:28:37.522
Like what you don't want to do is simply take the car because taking the car is somehow addicting and, um,

316
00:28:39.768 --> 00:28:41.109
you feel kind of lazy.

317
00:28:41.649 --> 00:28:49.912
And so even when you need to get exercise, maybe even when you haven't been outside all day, you don't wanna take the car anyways just because it's the easiest thing to do.

318
00:28:51.293 --> 00:29:05.959
And I think in the same way, like, yeah, obviously AI can be super efficient for many, many things, but if people are sort of just mindlessly delegating tasks to AI without even thinking about them at all,

319
00:29:08.335 --> 00:29:10.576
I think that's the boring thing.

320
00:29:12.037 --> 00:29:12.517
That makes sense.

321
00:29:13.958 --> 00:29:15.279
I feel like the...

322
00:29:16.159 --> 00:29:28.286
I talked at the very beginning about the data game, and I feel like the data game went from getting interesting data sets to getting environments and giving labs environments.

323
00:29:29.347 --> 00:29:30.588
Do you think that that's...

324
00:29:31.544 --> 00:29:32.245
Is that accurate?

325
00:29:32.366 --> 00:29:33.568
And if so, can you explain why?

326
00:29:33.588 --> 00:29:34.810
Yeah.

327
00:29:34.910 --> 00:29:42.082
So certainly the trend and the new research direction in the past year has been this concept of our environments.

328
00:29:43.420 --> 00:29:48.483
And what I would say is, I mean, you certainly need the fundamentals.

329
00:29:48.743 --> 00:29:52.966
Like before the model can operate in this environment, it needs to know basic things.

330
00:29:53.006 --> 00:29:56.028
Like it needs to know how to follow instructions.

331
00:29:56.308 --> 00:29:58.829
It needs to know how to avoid hallucinating.

332
00:29:59.310 --> 00:30:01.231
It needs to know how to write code and how to use tools.

333
00:30:01.791 --> 00:30:03.592
It needs to know how to write and so on and so on.

334
00:30:04.493 --> 00:30:08.315
But as models are becoming more agentic,

335
00:30:10.329 --> 00:30:12.151
And yeah, they will have access to tools.

336
00:30:12.311 --> 00:30:15.072
They will have access to all of our documents.

337
00:30:15.112 --> 00:30:17.214
They will be able to operate browsers.

338
00:30:17.314 --> 00:30:33.084
As that becomes almost a default way that models interact with us, our environments are basically just sort of like a more on distribution way of training them, which is why they're becoming more and more popular.

339
00:30:33.484 --> 00:30:38.247
As the models get more powerful, then the way we train them is getting more powerful as well.

340
00:30:38.650 --> 00:30:39.651
What would be an example?

341
00:30:39.831 --> 00:30:49.643
So I guess the obvious environment is like using a computer, but what would be an example of an environment that's non-obvious, that's teaching models things we might not think of?

342
00:30:50.744 --> 00:30:52.907
So I can give an example where...

343
00:30:55.173 --> 00:31:01.635
A lot of our environments are a combination of tools that the models need to learn to use.

344
00:31:02.455 --> 00:31:08.216
Like this might be an MCP server, or it might be calling a Google Drive API or the Slack API.

345
00:31:09.357 --> 00:31:15.838
In combination with a bunch of documents, like here are 30 PDFs and 20 Word document files.

346
00:31:17.279 --> 00:31:21.540
And you might give it a prompt like, hey, can you...

347
00:31:22.585 --> 00:31:27.616
Go update or 2026 forecasted revenue numbers.

348
00:31:28.417 --> 00:31:33.668
And what the model needs to do is it needs to learn how to...

349
00:31:34.730 --> 00:31:39.331
find the right PDFs and documents and needs to learn when should it search through SAC.

350
00:31:39.911 --> 00:31:42.092
It needs to learn when is some information outdated.

351
00:31:42.172 --> 00:31:44.852
Like maybe there's an email with, uh, with some early forecasts.

352
00:31:45.312 --> 00:31:51.914
And then later on, there's another email from the same person or, you know, maybe a different person who was saying like, Oh, whoops, I actually made a mistake in those earlier numbers.

353
00:31:51.934 --> 00:31:53.634
So here's a, here's an updated version.

354
00:31:55.114 --> 00:31:58.035
And so that is a, I think fairly canonical version of an environment.

355
00:31:59.162 --> 00:32:15.128
And then one of the interesting things we found, so I think we're actually going to publish a paper on this soon, but even when we didn't give this kind of environment any access to coding, when we trained a model on this environment, we actually found that it improved on coding a lot.

356
00:32:16.213 --> 00:32:39.508
And the reason was because we were basically teaching it these generalized forms of instruction following, generalized forms of tool use, generalized forms of understanding documents, which you can think of as fairly analogous to the way a model needs to look through various files in your repository and understand that some documents

357
00:32:40.406 --> 00:32:41.928
things supersede others.

358
00:32:43.910 --> 00:32:52.640
Or, you know, just the way it uses tools is obviously very analogous to the way that a model might write unit tests and execute them and iterate over and over again until it passes them.

359
00:32:53.380 --> 00:32:55.262
So I thought that was actually a really, really interesting find.

360
00:32:55.763 --> 00:32:56.364
Really interesting.

361
00:32:56.564 --> 00:32:57.345
Did you see Taki?

362
00:32:59.570 --> 00:32:59.790
No.

363
00:33:00.311 --> 00:33:03.373
It's the language model that's trained only on text from before 1930.

364
00:33:03.413 --> 00:33:03.653
Oh, okay.

365
00:33:03.673 --> 00:33:04.534
Yeah, yeah, yeah.

366
00:33:04.834 --> 00:33:05.975
I saw that.

367
00:33:06.015 --> 00:33:06.735
Why don't you make of that?

368
00:33:06.755 --> 00:33:11.138
Because I thought it was so interesting that you can get it to program.

369
00:33:11.979 --> 00:33:15.201
If you shop prompt it, you can get it to program basic things.

370
00:33:15.221 --> 00:33:15.942
What do you make of that?

371
00:33:15.962 --> 00:33:17.423
And what does that tell you about the value of data?

372
00:33:19.267 --> 00:33:23.789
So I personally didn't dig into it that much, but I thought the concept was fascinating.

373
00:33:23.909 --> 00:33:26.150
Like basically this idea, and I think a lot of people have this idea.

374
00:33:26.190 --> 00:33:33.233
It's like, if you gave, if somehow were able to create a data set, I think contamination issues are very, very difficult to avoid.

375
00:33:33.253 --> 00:33:34.294
So the question is how you would do this.

376
00:33:34.654 --> 00:33:43.478
It's like, if you gave the model data only up until pre-Newton, would it be able to discover Neukonian mathematics?

377
00:33:44.295 --> 00:33:48.978
Would it be able to discover quantum physics and so on and so on?

378
00:33:50.359 --> 00:34:01.285
So yeah, I think it's a really, really interesting question in terms of what types of inherent reasoning the model will be able to learn and then extrapolate from that.

379
00:34:01.485 --> 00:34:05.768
And it's almost like if it can discover all those things, then okay.

380
00:34:06.548 --> 00:34:12.552
Then given the state of science today, does that mean that the model is going to be able to discover science that centers out?

381
00:34:13.340 --> 00:34:20.302
Having played with it a lot, my sense is the answer is no, but a qualified no.

382
00:34:21.603 --> 00:34:23.223
And you can kind of feel it.

383
00:34:24.464 --> 00:34:29.486
You can feel it bumping up against the limits of its world when you start talking to it about like more modern things.

384
00:34:29.506 --> 00:34:33.927
Like it just, it's, you know, there's this foster science, Thomas Kuhn, he talks about incommensurability.

385
00:34:34.847 --> 00:34:39.349
And it feels like my world and its world are sort of incommensurable.

386
00:34:39.969 --> 00:34:41.330
But then you can also get it to...

387
00:34:42.570 --> 00:34:43.090
program.

388
00:34:43.290 --> 00:34:53.213
But the way you do that is you get it to combine its circuits in a way that's not, it wouldn't be natural for it, but you can prompt it in a way to do that in a way that ends up being programming.

389
00:34:53.333 --> 00:34:56.614
So I sort of both think it can't do it.

390
00:34:56.814 --> 00:35:02.236
And also if you prompt it cleverly enough, it can, but you have to supply the answer first.

391
00:35:03.777 --> 00:35:04.277
Does that make sense?

392
00:35:04.297 --> 00:35:04.917
Yeah.

393
00:35:06.040 --> 00:35:06.280
Yeah.

394
00:35:07.040 --> 00:35:07.440
Interesting.

395
00:35:08.160 --> 00:35:08.400
Okay.

396
00:35:08.560 --> 00:35:09.761
What is the value of my data?

397
00:35:10.321 --> 00:35:14.041
So one of the things that I'm, I'm just so interested in.

398
00:35:14.542 --> 00:35:33.525
So obviously you're, you're in a data company, like you're, you're, you're getting expert data, um, from like real PhDs and, and, uh, selling it to the model companies and like providing, uh, all of the, all the like smarts and taste to, uh, to the models that we use every day for someone like me.

399
00:35:33.545 --> 00:35:33.605
Um,

400
00:35:36.265 --> 00:35:40.148
we're just getting to a point where it's actually pretty easy for me to gather a data set.

401
00:35:40.668 --> 00:35:49.595
You know, like, for example, I do all of my email in Codex and I have a history for every email of, was this useful?

402
00:35:49.775 --> 00:35:50.575
Did I dismiss it?

403
00:35:50.615 --> 00:35:51.476
Did I reply to it?

404
00:35:51.516 --> 00:35:53.337
If I replied, like, what did I say?

405
00:35:56.019 --> 00:35:57.500
What is the value of that?

406
00:35:57.680 --> 00:35:59.622
If I wanted to sell that to you, how much did you pay for it?

407
00:36:01.100 --> 00:36:05.866
So the value to me as someone who would use that data to train an AI model?

408
00:36:08.569 --> 00:36:09.691
Let me think.

409
00:36:09.711 --> 00:36:18.542
So I think the value would be teaching models very, very deep personalization.

410
00:36:19.905 --> 00:36:24.189
Like I think right now the models are actually not very good at personalizing things.

411
00:36:24.429 --> 00:36:24.990
It's kind of funny.

412
00:36:25.150 --> 00:36:32.537
Whenever I use AI models, I actually turn off the features where they personalize to me or where they can search across all of my conversation histories.

413
00:36:33.158 --> 00:36:40.705
Because I find that they just over index on things that I said once, but actually aren't all that important to me.

414
00:36:41.286 --> 00:36:42.987
So I actually have it completely turned off.

415
00:36:43.768 --> 00:36:45.009
Unless I'm like testing something.

416
00:36:46.129 --> 00:36:51.392
So I think the value of it would be like, okay, yeah, you did report all of these emails as spam.

417
00:36:51.472 --> 00:36:54.673
So the next time this email comes in, you should automatically know that it's spam.

418
00:36:55.574 --> 00:36:57.735
Or it should learn that this is your writing style.

419
00:36:58.095 --> 00:37:08.240
Like one of the reasons I think people don't use AI for better or worse for writing more is because it sounds obviously AI generated and it's not mashing their voice or their cadence.

420
00:37:09.362 --> 00:37:12.603
Or it's that, okay, these are the things that you yourself care about.

421
00:37:13.043 --> 00:37:22.847
Like, I think one of the biggest reasons AI is maybe not as useful as people would have expected sometimes is because it lacks all of your context.

422
00:37:23.107 --> 00:37:26.689
Like, it doesn't know that these are the articles that you read.

423
00:37:26.989 --> 00:37:31.090
It doesn't know that these are the decisions about, you know, the company that you're making.

424
00:37:31.130 --> 00:37:32.171
These are the goals that you have.

425
00:37:33.251 --> 00:37:46.923
And once all of that is in the model's history and it knows that it can incorporate these things and these are the kind of optimal decisions that you made, it's very, very valuable in teaching it.

426
00:37:46.983 --> 00:37:51.326
Okay, this is actually how I use all this data to make certain kinds of decisions.

427
00:37:51.967 --> 00:37:56.991
So, yeah, I think that deep personalization is what is most unique about that.

428
00:37:58.058 --> 00:37:58.719
That's interesting.

429
00:37:59.559 --> 00:38:03.803
And as an individual person, I mean, I guess I could turn it into a synthetic data set.

430
00:38:04.263 --> 00:38:06.325
But as an individual person, is that worth a lot?

431
00:38:06.345 --> 00:38:08.687
Like, should I be thinking about selling it?

432
00:38:10.969 --> 00:38:12.270
I imagine we could make you an offer.

433
00:38:12.290 --> 00:38:17.794
I have to learn a little bit more about how big this data set size is.

434
00:38:17.894 --> 00:38:20.436
But I can make it as big as you want.

435
00:38:20.477 --> 00:38:23.799
I've got Fable.

436
00:38:23.819 --> 00:38:24.540
You convinced me.

437
00:38:26.762 --> 00:38:31.885
One of the things we actually do is, I mean, we teach models in these very, very deep, personalized ways.

438
00:38:32.525 --> 00:38:36.767
So something similar to what you described is a fairly big thing.

439
00:38:36.827 --> 00:38:37.287
Tell me more.

440
00:38:37.307 --> 00:38:38.708
So, I mean, I've got email.

441
00:38:38.728 --> 00:38:45.431
What else am I doing that you're like, oh, that's actually really valuable and important in ways that people probably wouldn't know?

442
00:38:47.092 --> 00:38:53.195
So, honestly, even things like the way you interact with your browser is interesting.

443
00:38:53.335 --> 00:38:55.516
Like models still aren't all that good at it.

444
00:38:57.065 --> 00:39:05.152
Or even the types of conversations that you're having with AI, that is just inherently interesting in of itself.

445
00:39:05.853 --> 00:39:12.799
Models themselves are not very good at generating synthetic conversations to try to mimic you.

446
00:39:13.519 --> 00:39:16.582
And so even just knowing what types of conversations you're having is helpful.

447
00:39:17.808 --> 00:39:24.315
Or it's like the combination of all these things, like knowing that these are your photos, these are your texts, these are your slacks.

448
00:39:24.555 --> 00:39:26.337
It's like this interconnected web.

449
00:39:28.279 --> 00:39:31.222
And maybe certain things in one aspect of the web influence others.

450
00:39:31.442 --> 00:39:34.625
So just seeing the thing as a whole is very helpful as well.

451
00:39:35.346 --> 00:39:39.270
Why are models bad at writing and how does that relate to the personalization challenge?

452
00:39:40.671 --> 00:39:41.092
So...

453
00:39:42.750 --> 00:39:48.773
I think some of the models are pretty good at writing, but some of them are actually kind of shockingly terrible.

454
00:39:48.833 --> 00:39:49.834
So I'll give an example.

455
00:39:50.474 --> 00:39:58.659
So we created a benchmark called Hemingway Bench a couple months ago, and it was designed to test models creative writing abilities.

456
00:40:00.440 --> 00:40:08.725
And one of the things that we saw was that some of the models, they were literally outputting metaphors in every single sentence.

457
00:40:10.705 --> 00:40:16.490
And I think the reason that was happening is because, I've talked a little bit about this phenomenon on reward hacking.

458
00:40:17.150 --> 00:40:22.635
It's almost like there was a metric somewhere or like a score that these models were getting.

459
00:40:22.735 --> 00:40:30.021
Like, okay, every time you are literary, every time you're using complex imagery, it would get a point.

460
00:40:31.022 --> 00:40:37.507
And it learned to reward hack this by outputting a metaphor in every single sentence.

461
00:40:38.930 --> 00:40:46.775
And I mean, what's kind of funny is that a couple, what was it, a couple of weeks ago, there was this kind of like semi-prestigious literary prize, I think the Commonwealth Prize.

462
00:40:47.575 --> 00:40:53.278
And there was a controversy because a clearly AI journal of the story won the prize.

463
00:40:53.298 --> 00:40:58.561
And if you actually looked at that story, it's funny, like it literally had a metaphor in every single sentence.

464
00:40:59.742 --> 00:41:03.364
And so this kind of phenomenon that we described a couple of months ago, yeah, it was still happening.

465
00:41:04.064 --> 00:41:04.685
And so...

466
00:41:05.731 --> 00:41:12.796
Yeah, I mean, I think it boils down to a couple reasons, but like one is it's people are kind of sort of measuring the wrong thing.

467
00:41:12.816 --> 00:41:23.383
Like instead of measuring actual taste and actually good pros, they either have these flawed metrics like.

468
00:41:24.510 --> 00:41:29.054
What is the complexity of the prose I'm writing?

469
00:41:29.535 --> 00:41:30.816
How many metaphors do I have?

470
00:41:31.717 --> 00:41:39.984
Or there are these AI leaderboards again, like Ella Marina, where you have people who are essentially high schoolers who are reading responses for two seconds,

471
00:41:41.258 --> 00:41:45.000
And what they are captivated by is a flashy metaphor.

472
00:41:46.041 --> 00:41:49.182
And they are not captivated by kind of like the understated pros.

473
00:41:49.783 --> 00:41:58.567
And so I think it kind of boils down to a mismatch in measurement and a mismatch in like the optimization and objectives that the models are trained towards.

474
00:41:59.248 --> 00:41:59.748
Fascinating.

475
00:42:00.328 --> 00:42:01.169
Okay, last question.

476
00:42:01.649 --> 00:42:03.650
What is your current AGI timeline?

477
00:42:05.283 --> 00:42:11.307
So I certainly believe that AI will happen more than most people expect.

478
00:42:12.008 --> 00:42:17.892
Like every few months and even faster now, I think what AI is doing continues to surprise us.

479
00:42:18.892 --> 00:42:23.636
So I think it depends a little bit, obviously, on your definition of AGI.

480
00:42:24.196 --> 00:42:34.603
But if my metric were something like being able to automate the work of the average engineer, like,

481
00:42:35.617 --> 00:42:48.881
or being able to publish more and more novel scientific research that gets published in these journals, or even the ability to win a Fields Medal or a Nobel Prize.

482
00:42:48.901 --> 00:42:51.841
I could see it happening within the next five years.

483
00:42:52.622 --> 00:42:52.982
All right.

484
00:42:53.942 --> 00:42:55.642
Edwin, thanks so much for joining.

485
00:42:56.863 --> 00:42:57.403
Thanks for having me.

486
00:43:05.608 --> 00:43:11.294
Oh my gosh, folks, you absolutely positively have to smash that like button and subscribe to AI and I.

487
00:43:11.514 --> 00:43:11.774
Why?

488
00:43:12.255 --> 00:43:14.417
Because this show is the epitome of awesomeness.

489
00:43:14.697 --> 00:43:16.579
It's like finding a treasure chest in your backyard.

490
00:43:16.880 --> 00:43:22.365
But instead of gold, it's filled with pure, unadulterated knowledge bombs about chat GPT.

491
00:43:23.186 --> 00:43:30.611
Every episode is a roller coaster of emotions, insights, and laughter that will leave you on the edge of your seat, craving for more.

492
00:43:30.891 --> 00:43:31.712
It's not just a show.

493
00:43:32.232 --> 00:43:35.854
It's a journey into the future with Dan Shipper as the captain of the spaceship.

494
00:43:37.075 --> 00:43:38.616
So do yourself a favor.

495
00:43:38.856 --> 00:43:42.158
Hit like, smash subscribe, and strap in for the ride of your life.

496
00:43:42.598 --> 00:43:47.942
And now, without any further ado, let me just say, Dan, I'm absolutely hopelessly in love with you.
