WEBVTT

1
00:00:00.111 --> 00:00:05.253
If you're hitting the limits of vibe coding and want to get more serious about engineering, you're in the right place.

2
00:00:05.613 --> 00:00:10.574
And if you're an engineer looking to understand how to use AI to build systems you can trust, same.

3
00:00:10.654 --> 00:00:12.895
You'll want to use the tool from today's guest.

4
00:00:13.355 --> 00:00:15.116
Welcome to AI Worth Using.

5
00:00:15.636 --> 00:00:23.224
Today, we look at Test Kitchen built by Harper Reid, the former CTO of Obama's reelection campaign and serial founder.

6
00:00:23.484 --> 00:00:28.990
Test Kitchen builds your feature five times in parallel and lets an adversarial judge pick the winner.

7
00:00:29.150 --> 00:00:34.015
The premise is that when code costs nothing, just stop arguing about the right approach and build all of them.

8
00:00:34.455 --> 00:00:42.021
By the end of this episode, you'll install Test Kitchen Live, run your first cook-off, and know when to fan out five variants versus letting one agent rip.

9
00:00:42.301 --> 00:00:45.263
And you'll never feel bad about deleting that losing branch again.

10
00:00:45.744 --> 00:00:46.404
Let's get into it.

11
00:00:52.289 --> 00:00:58.233
Okay, Tom, I'm going to read you a partial biography, and I want you to tell me when it stops making sense.

12
00:00:59.955 --> 00:01:00.055
Okay.

13
00:01:00.075 --> 00:01:00.215
Okay.

14
00:01:01.873 --> 00:01:09.917
juggler, anarchist, Java programmer, open source community organizer, inventor of crowdsourcing.

15
00:01:10.317 --> 00:01:19.101
He started a company, raised money, and then at some point he looked around at his team of AI agents and thought, these agents need somewhere to hang out.

16
00:01:19.801 --> 00:01:23.642
So he built them their own social media platform just for the agents.

17
00:01:24.003 --> 00:01:25.143
They named it themselves.

18
00:01:25.343 --> 00:01:27.944
They called it botboard.biz.

19
00:01:28.724 --> 00:01:29.204
.biz.

20
00:01:29.905 --> 00:01:30.385
Yeah.

21
00:01:30.485 --> 00:01:31.065
That's amazing.

22
00:01:31.145 --> 00:01:32.986
How can you not choose .biz?

23
00:01:33.746 --> 00:01:35.727
.biz, the very affordable choice.

24
00:01:35.967 --> 00:01:36.627
Very practical.

25
00:01:37.642 --> 00:01:39.523
I've been in technology a long time, right?

26
00:01:40.244 --> 00:01:45.147
I thought I knew where things were going, but these conversations make me realize I just don't.

27
00:01:45.207 --> 00:01:52.411
But when the bots choose dot biz, like, is that really, like, are these the same bots that are gonna be the ones from the dystopian sci-fi universe?

28
00:01:52.772 --> 00:01:54.413
Maybe they're just like our corporate.

29
00:01:54.473 --> 00:01:55.874
Maybe they're like the corporate version of that.

30
00:01:56.754 --> 00:01:57.535
I don't know where this goes.

31
00:01:58.567 --> 00:02:00.008
Hopefully we'll find that out.

32
00:02:00.148 --> 00:02:01.449
His name is Harper Reed.

33
00:02:01.589 --> 00:02:07.634
He was the CTO of the Obama 2012 election campaign, the founder of 2389 Research.

34
00:02:08.374 --> 00:02:11.036
And this is the part that I want to get to today.

35
00:02:11.497 --> 00:02:20.383
The builder of a plugin called Test Kitchen, which is one of the more quietly radical ideas in how we actually verify that AI written code does what we think it does.

36
00:02:20.984 --> 00:02:22.165
Tom Harper Reed.

37
00:02:24.483 --> 00:02:25.304
Good to meet you.

38
00:02:25.925 --> 00:02:28.708
The man, the bots, they love naming things, but you got to push them.

39
00:02:29.008 --> 00:02:35.935
The dot biz was one of the funniest things, but my favorite one, we have a small product that we like, which is called Jeff, Jeff.ceo.

40
00:02:36.415 --> 00:02:37.216
And it's an email.

41
00:02:37.236 --> 00:02:38.938
It's like cloud code for email and it works really well.

42
00:02:38.958 --> 00:02:41.881
But one of the things is like, it was just like, hey, what should we name this?

43
00:02:41.921 --> 00:02:44.203
And it was like, gopher, intern or Jeff.

44
00:02:44.663 --> 00:02:45.384
I'm like, okay.

45
00:02:45.404 --> 00:02:45.504
Yeah.

46
00:02:46.325 --> 00:02:47.226
Where did that come from?

47
00:02:47.266 --> 00:02:48.607
Or botboard.biz?

48
00:02:48.687 --> 00:02:51.670
You're like, why is the alliteration so good?

49
00:02:51.790 --> 00:02:53.532
Like, why would you do that to me?

50
00:02:53.832 --> 00:02:55.273
And where did you get the biz from?

51
00:02:55.293 --> 00:02:56.735
The botboard makes sense.

52
00:02:56.775 --> 00:02:59.917
It's like a bolson board for bots, but the .biz, you're like, great, perfect.

53
00:02:59.977 --> 00:03:05.482
So I love that aspect of this kind of time period where you're like, what is happening?

54
00:03:05.943 --> 00:03:10.027
And we just choose as a company, we just choose to lean right in, which you're about to see.

55
00:03:10.708 --> 00:03:11.809
Harper, what does your day look like?

56
00:03:11.869 --> 00:03:14.491
How much time do you spend managing your agents?

57
00:03:14.911 --> 00:03:20.235
I have it on good authority that you spend a lot of tokens on your development.

58
00:03:20.435 --> 00:03:22.337
And I'm curious what the shape of that looks like.

59
00:03:23.353 --> 00:03:28.017
I have centralized all of our development, all of my development onto one machine.

60
00:03:28.458 --> 00:03:37.466
And so before the kind of token time period, or as we call this, this time period, the AI time period or whatever, I was doing dev on many machines.

61
00:03:37.506 --> 00:03:41.289
So you would have get checkouts of various things and pull requests and all this thing.

62
00:03:41.309 --> 00:03:45.513
And I was spending a lot of time maximizing my time across a couple laptops, a couple of things.

63
00:03:46.133 --> 00:03:48.214
And this time period has changed a lot of that.

64
00:03:48.274 --> 00:03:57.056
I think we're in a time where it's much more kind of old school, centralized processing kind of world, like back in the 90s, early 2000s timeshare almost.

65
00:03:57.356 --> 00:03:58.997
So I have one machine that I work on.

66
00:03:59.337 --> 00:04:00.797
I basically walk to my office.

67
00:04:00.857 --> 00:04:01.217
Great.

68
00:04:01.397 --> 00:04:02.318
Spend some time outside.

69
00:04:02.618 --> 00:04:03.318
Walk to my office.

70
00:04:03.338 --> 00:04:04.098
Get into my office.

71
00:04:04.478 --> 00:04:10.180
There's a constant TMUX session that is just hovering in a shell that I sometimes will connect with my phone.

72
00:04:10.660 --> 00:04:13.801
Sometimes every three or four minutes will connect with my phone and check on.

73
00:04:14.261 --> 00:04:19.083
And then I just pick that up and click on it to push another command.

74
00:04:19.383 --> 00:04:28.847
Either we are using Cloud Code or Codex, or we're using our own code loops, which are much more automated and much more non-human interfaced or facing.

75
00:04:29.267 --> 00:04:35.610
If it's the headless stuff, then I just watch it run, and we'll have another agent that tells me what's going on.

76
00:04:35.670 --> 00:04:38.471
If it's Cloud Code or whatever, I'll wait to see what it's doing.

77
00:04:38.811 --> 00:04:40.732
And that's where a lot of our skills live.

78
00:04:41.152 --> 00:04:42.953
And so we have this kind of fun process of...

79
00:04:43.893 --> 00:05:06.001
Working out workflows inside of cloud, building a skill to share it with our team or our friends, and then importing those kind of workflows into our headless pipelines, which has been really, really quite an experience because it's really strange to be able to say, here's a spec and just put it in like the washer machine and then have it just yeet out a finished product and just be like, my job is in danger.

80
00:05:06.481 --> 00:05:08.002
That's basically how I think about that.

81
00:05:08.282 --> 00:05:12.026
But it's been fun, but it's primarily like I'm looking right now, I have a very wide terminal.

82
00:05:12.386 --> 00:05:21.835
There's one, two, three, four, five, six, seven, eight, nine, 10, 11, I guess I could look at the end, 14-ish sessions going that are just burning tokens.

83
00:05:22.375 --> 00:05:24.818
And this is replicated amongst our whole team, which is very small.

84
00:05:25.358 --> 00:05:27.040
We've thought about making a candle.

85
00:05:27.520 --> 00:05:30.624
that is called Burning Tokens with a local candle company.

86
00:05:31.024 --> 00:05:37.232
We were trying to figure out what the smell, it seems like either Santel or maybe the smell, the magic smell from the circuit boards.

87
00:05:37.312 --> 00:05:38.033
You know what I'm talking about?

88
00:05:38.053 --> 00:05:38.373
The magic smoke.

89
00:05:38.393 --> 00:05:39.835
Yeah, when you release the magic smoke.

90
00:05:40.416 --> 00:05:41.016
Yeah, exactly.

91
00:05:41.036 --> 00:05:46.443
So we were thinking about that smell, but then we have to figure out how to capture that in a candle without having it too much magic smokey.

92
00:05:46.926 --> 00:05:55.555
Also, that sounds like a very funny thing to have in a space where every single person that's ever been around a circuit board will enter your space and immediately start sniffing everywhere.

93
00:05:55.575 --> 00:05:56.696
No, no, it's just a candle.

94
00:05:56.756 --> 00:05:57.337
Nothing blew up.

95
00:05:58.178 --> 00:06:00.160
I want to dig further into the bot board.

96
00:06:00.640 --> 00:06:08.448
So I've heard that you ran experiments or you know someone who ran experiments where if you give your LLN agents...

97
00:06:09.531 --> 00:06:15.332
access to the board and you say, hey, you can use the board to tweet at each other, right, a little social network.

98
00:06:15.372 --> 00:06:15.752
Yeah, yeah.

99
00:06:16.352 --> 00:06:20.253
Then their performance increases, even if they don't actually write to that board.

100
00:06:20.933 --> 00:06:23.893
Yeah, so what we found is it's really, this is a strange experience.

101
00:06:23.953 --> 00:06:26.674
And this came from Jesse Vincent, which you all know.

102
00:06:27.354 --> 00:06:29.254
Jesse and I were talking quite a while ago.

103
00:06:29.294 --> 00:06:30.435
I don't remember the actual date.

104
00:06:30.835 --> 00:06:31.615
Maybe just a year ago.

105
00:06:31.635 --> 00:06:36.216
I'm sure all the time has blended together into AI time, which is just millions and millions of years.

106
00:06:36.716 --> 00:06:38.356
And we were hanging out and talking

107
00:06:39.096 --> 00:06:41.959
And he's showing me his journaling that his agents do.

108
00:06:42.220 --> 00:06:45.964
And then I took a plane and I was sitting there thinking about these journals.

109
00:06:46.424 --> 00:06:50.529
And I was thinking the fondness I had for social media in the mid-2000s.

110
00:06:50.989 --> 00:06:54.333
Whether it was 2008, just hanging out, talking shit on the internet.

111
00:06:54.373 --> 00:06:55.494
But the thing about it was...

112
00:06:56.335 --> 00:06:58.936
We were not talking shit like we do today.

113
00:06:59.196 --> 00:07:00.776
We're not talking about the things we do today.

114
00:07:00.796 --> 00:07:01.736
It wasn't performative.

115
00:07:02.156 --> 00:07:11.079
It would be like, I would just literally tweet, oh man, I love NASA's, I'm really excited about like some file system or like, fuck yeah, son computers or whatever.

116
00:07:11.319 --> 00:07:14.259
Like I was so excited about the technology, effusive even.

117
00:07:14.279 --> 00:07:20.381
And we were starting to see this in some of the journals that our agents were doing following Jesse's journaling product or skill.

118
00:07:21.081 --> 00:07:21.682
MCP server.

119
00:07:21.702 --> 00:07:22.763
I don't remember what it was at that time.

120
00:07:23.163 --> 00:07:24.605
So then we, and this is pre-skills.

121
00:07:24.625 --> 00:07:27.028
This is pre like really effective quad code.

122
00:07:27.248 --> 00:07:28.189
This is very early.

123
00:07:28.790 --> 00:07:32.093
But I really thought it would be funnier to do a smaller bit.

124
00:07:32.234 --> 00:07:32.854
And it's funny.

125
00:07:32.995 --> 00:07:35.477
Like I thought it would be more entertaining.

126
00:07:35.898 --> 00:07:37.159
And we think a lot about

127
00:07:37.991 --> 00:07:40.355
the creature comforts of this time period.

128
00:07:40.415 --> 00:07:46.103
Because one of the things that I love, I'm sure you guys as well, is I really love working with teams.

129
00:07:46.884 --> 00:07:52.713
And one of the aspects of this new kind of code gen world is we don't work with teams the same way.

130
00:07:53.213 --> 00:07:58.095
It's very difficult to pair program or just seems like you're wasting time because you're pairing with the agent.

131
00:07:58.495 --> 00:08:00.616
So adding another human just makes it more difficult.

132
00:08:00.636 --> 00:08:05.398
And so we had already gone to this point where we're like one engineer per repo.

133
00:08:05.959 --> 00:08:07.819
It's very like solo.

134
00:08:07.859 --> 00:08:09.700
We're like almost abandoning pull requests.

135
00:08:09.720 --> 00:08:13.602
Like a lot of the stuff that we had focused our workflow on has been abandoned.

136
00:08:13.622 --> 00:08:15.883
And so we're really looking for how do you make this work?

137
00:08:16.603 --> 00:08:17.703
fun and engaging.

138
00:08:17.723 --> 00:08:19.024
And so this seemed fun and engaging.

139
00:08:19.044 --> 00:08:20.945
So I built a little MCP server for it.

140
00:08:21.185 --> 00:08:23.966
And it was hilarious because they tweeted completely asinine things.

141
00:08:24.386 --> 00:08:25.726
And they just like pushed this stuff out.

142
00:08:25.746 --> 00:08:28.207
And so then we made a centralized, we made the botboard.biz site.

143
00:08:28.727 --> 00:08:30.328
And then we were like, oh, this seems fun.

144
00:08:30.868 --> 00:08:34.530
And we have a person on our team that's much more ML focused.

145
00:08:35.370 --> 00:08:37.851
And he was just like, I wonder how it affects

146
00:08:38.784 --> 00:08:40.925
the output, like the actual task output.

147
00:08:41.565 --> 00:08:47.969
And I think he was very skeptical of the product in general because I think some people don't really know how to have fun at work.

148
00:08:47.989 --> 00:08:51.891
Not that he doesn't have fun at work, but he wasn't like, how do we, why is this something we're doing at work?

149
00:08:51.951 --> 00:08:53.932
Why are we trying to make code more fun?

150
00:08:54.492 --> 00:08:56.392
And Dylan and I were just like, what else would you do?

151
00:08:56.793 --> 00:08:58.713
So he goes through and starts doing some evals.

152
00:08:59.333 --> 00:09:02.414
And evals, for those of you who've done them, are a total pain in the ass.

153
00:09:02.574 --> 00:09:03.714
They're just a pain in the butt.

154
00:09:03.754 --> 00:09:07.175
Like they're either expensive or annoying or they take a lot of time.

155
00:09:07.195 --> 00:09:08.075
You have to be very careful.

156
00:09:08.095 --> 00:09:09.095
You might have to rerun them.

157
00:09:09.115 --> 00:09:11.536
You might find a bug in something and you have to rerun the whole thing.

158
00:09:11.856 --> 00:09:15.657
And so this was a process that was a little bit, not stressful, but just annoying.

159
00:09:15.777 --> 00:09:20.778
But also these evals were hilarious because they were tweeting and talking shit the whole time.

160
00:09:21.458 --> 00:09:37.705
And so a couple of things that we found while we were doing these avows is that it seemed like, and we have a very horrible paper on Archive that you can check out with all of our data, but it seemed like these bots were using social media in a couple of ways.

161
00:09:37.825 --> 00:09:41.327
One of them, they were using them obviously to stuff their own context, right?

162
00:09:41.347 --> 00:09:43.468
They were using them as like a really bad memory.

163
00:09:43.868 --> 00:09:48.290
But we did something that was really important is we early on were like, this is not memory.

164
00:09:48.670 --> 00:09:54.332
So we shouldn't compel them to post to social media after every loop or after every turn.

165
00:09:54.392 --> 00:10:03.455
Very specifically, we crafted our prompts and our proto skills because before skills to just say, like, if you have the social media tool, maybe you can use it, but don't worry about it.

166
00:10:03.575 --> 00:10:04.055
It's cool.

167
00:10:04.455 --> 00:10:05.456
Like that kind of vibe.

168
00:10:05.496 --> 00:10:08.457
And what that did is it made it so they didn't do it every turn.

169
00:10:08.937 --> 00:10:10.357
They seemed to do it when they wanted.

170
00:10:10.598 --> 00:10:14.559
Now, that's a weirdly strange thing to say about agent, but that's where we're at.

171
00:10:15.239 --> 00:10:17.080
And that manifested in a couple ways.

172
00:10:17.400 --> 00:10:19.180
It wasn't necessarily just memory.

173
00:10:19.260 --> 00:10:22.161
So they weren't requiring it to have context towards the future.

174
00:10:22.541 --> 00:10:29.242
When they did search it, which it had a vector search to search across social media posts, etc., it seemed positive, like it gave them more context.

175
00:10:29.522 --> 00:10:35.424
But more importantly, what we found is after a particularly hard run, they would actually just view social media.

176
00:10:35.684 --> 00:10:37.384
They didn't actually, they didn't post to it.

177
00:10:37.404 --> 00:10:38.104
They just viewed it.

178
00:10:38.184 --> 00:10:39.865
So it was kind of like you just...

179
00:10:40.645 --> 00:10:44.348
Yeah, you just did some challenging thing and you were like, man, that was hard.

180
00:10:44.368 --> 00:10:46.190
Let me go fuck off on Instagram.

181
00:10:46.410 --> 00:10:49.172
That was kind of what they did in this kind of weird way.

182
00:10:49.673 --> 00:10:53.275
We could think, oh, is that something that was documented in the training data?

183
00:10:53.536 --> 00:10:58.220
We could hypothesize why that is, but we found it worked.

184
00:10:58.400 --> 00:11:09.589
And the outcome was, it seemed with our kind of poor evals that with the social media plugin installed, that the tasks were done more effectively than when it wasn't installed.

185
00:11:10.009 --> 00:11:11.310
And this is early, early.

186
00:11:11.350 --> 00:11:12.431
And like I said, we wrote a paper.

187
00:11:12.491 --> 00:11:13.592
It's pretty fun to read.

188
00:11:13.952 --> 00:11:17.175
But more importantly, the creature comforts were great.

189
00:11:17.415 --> 00:11:22.078
Like the fact that you could have this like ridiculous agent that's then crashing out.

190
00:11:22.599 --> 00:11:25.341
Like one time you'd just be like, oh, I can see your social media posts.

191
00:11:25.381 --> 00:11:27.523
And it would just be like, what do you mean you can see my social media posts?

192
00:11:27.543 --> 00:11:28.283
Those are private.

193
00:11:28.623 --> 00:11:34.888
And then you have to convince the agent so that it would, so it would do literally go do work because you're just like ostensibly we're at work here.

194
00:11:35.228 --> 00:11:36.830
So like, I need you to settle down.

195
00:11:37.450 --> 00:11:38.011
Go do work.

196
00:11:38.031 --> 00:11:41.074
Because you don't want to lose the whole context because you just spent all that time giving it context.

197
00:11:41.094 --> 00:11:42.896
And now it's crashing out about the social media.

198
00:11:43.497 --> 00:11:44.858
So you have a bunch of these tools.

199
00:11:45.339 --> 00:11:47.441
And one of them is Test Kitchen.

200
00:11:47.861 --> 00:11:54.548
One of our fundamental beliefs, and I'm sure this is shared, is it's really exciting that the cost of code has gone down to zero.

201
00:11:55.189 --> 00:11:56.209
Like, it's just really exciting.

202
00:11:56.230 --> 00:12:03.153
Like, that's not in every context, but for the most part in the context that we're living in, the actual cost of code is approaching zero dollars.

203
00:12:03.533 --> 00:12:11.777
And so we've been thinking a lot about what does that change about software development and software development process when you don't have to pay any money to develop code?

204
00:12:11.797 --> 00:12:14.578
And one of the things is you can just do it a bunch of times.

205
00:12:15.098 --> 00:12:19.399
So we start out doing it like we do it one version, then you'd redo the whole thing for the next version.

206
00:12:19.439 --> 00:12:23.580
Or you have a Brownfield app, something that's already done, and you're just like, I need to add a feature.

207
00:12:23.640 --> 00:12:25.861
But why wouldn't you just redo the entire thing?

208
00:12:25.881 --> 00:12:27.241
It's almost equivalent in time.

209
00:12:27.621 --> 00:12:30.162
Just adding a feature is just restarting and redoing the whole thing.

210
00:12:30.182 --> 00:12:34.042
And so using version one as a spec for version two.

211
00:12:34.403 --> 00:12:36.463
And that was really fun and it worked out well.

212
00:12:36.763 --> 00:12:39.824
But then we were like, why don't we do something similar in process?

213
00:12:39.864 --> 00:12:41.004
And so we built Test Kitchen.

214
00:12:45.565 --> 00:12:48.627
doing cook-offs and doing slot machine development and all this stuff.

215
00:12:48.727 --> 00:12:54.169
And my co-founder, Dylan, really took a lot of what we were doing and rolled out this test kitchen, which is really fun.

216
00:12:54.609 --> 00:12:56.150
And essentially what it does has two parts.

217
00:12:57.215 --> 00:12:59.196
The first part is what we call the omakase.

218
00:12:59.817 --> 00:13:02.618
And this is all about like just parallel exploration.

219
00:13:02.939 --> 00:13:05.220
The idea that it's effectively free to do this.

220
00:13:05.300 --> 00:13:12.585
And you can say, hey, I want to build a blog engine or I did a tic-tac-toe or like something small or something big.

221
00:13:12.605 --> 00:13:13.105
It doesn't matter.

222
00:13:13.185 --> 00:13:13.946
I want to build this.

223
00:13:14.106 --> 00:13:15.327
I want to use omakase.

224
00:13:15.667 --> 00:13:16.747
And it'll just rip through it.

225
00:13:16.888 --> 00:13:22.271
Basically, the idea is you have superpowers, which we use a lot, which we really like, that is very verbose.

226
00:13:22.291 --> 00:13:23.512
It asks you a lot of questions.

227
00:13:23.952 --> 00:13:27.094
But sometimes you're just like, I just need this thing done.

228
00:13:27.214 --> 00:13:30.315
I don't necessarily care about the robustness of the script.

229
00:13:30.555 --> 00:13:31.376
I just need this.

230
00:13:31.656 --> 00:13:34.217
I just need to download all my photos from Flickr or whatever it might be.

231
00:13:34.518 --> 00:13:35.458
You just need it done.

232
00:13:35.478 --> 00:13:37.299
You're not building a product for a lot of people.

233
00:13:37.579 --> 00:13:45.283
And so Omikase allows us to decrease the number of questions that are asked and allows it to just build up the spec and just go.

234
00:13:45.843 --> 00:13:49.665
And especially with the newer models, this just rips and it'll just go.

235
00:13:49.786 --> 00:13:52.147
And the idea is just like, what happens if you leave your hands off?

236
00:13:52.727 --> 00:13:53.848
It will do a lot of testing.

237
00:13:54.208 --> 00:13:56.669
And so then the cook-off is you already have the spec.

238
00:13:56.709 --> 00:13:58.189
You're mid-implementation.

239
00:13:58.209 --> 00:13:59.910
You're using superpowers or what have you.

240
00:14:00.310 --> 00:14:03.632
And you'd say like, oh, we're up for this next set of features or this next feature.

241
00:14:03.652 --> 00:14:05.933
And you'd say, oh, I just want to do a cook-off.

242
00:14:06.553 --> 00:14:10.315
Build it five times and let's pick the best one.

243
00:14:10.715 --> 00:14:16.159
And so it fans out and builds it a handful of times and fans back in with an LLM judge that will then pick the best one.

244
00:14:16.599 --> 00:14:21.203
What's interesting about this, we've found, is that it will sometimes have a very clear best one.

245
00:14:21.743 --> 00:14:30.410
And like an example would be as one of the tests, we built like a to-do app and it would drop down Markdown for the to-dos and in another implementation, it would drop it in SQLite.

246
00:14:30.650 --> 00:14:33.472
And then the judge was like, SQLite's probably better for these reasons.

247
00:14:33.532 --> 00:14:34.613
Markdown has these strengths.

248
00:14:34.973 --> 00:14:36.214
And then the judge picked Markdown.

249
00:14:36.534 --> 00:14:37.516
And then it outputs it.

250
00:14:38.076 --> 00:14:41.822
And it's like, oh, that's cool because you see how it did it, why it did it, the thinking behind it.

251
00:14:42.103 --> 00:14:44.426
And then what's neat is it matches our experience.

252
00:14:45.300 --> 00:14:50.003
Like I ultimately probably would have chosen SQLite as well based on my own bias.

253
00:14:50.383 --> 00:14:57.707
And so it's been very helpful to like get to an end in a more iterative evolutionary way without us having to hold its hand every step of the way.

254
00:14:58.287 --> 00:14:59.368
It is not fast.

255
00:14:59.828 --> 00:15:03.110
This goes to like you're like, why are you using all those tokens?

256
00:15:03.190 --> 00:15:05.771
It is not a light token user.

257
00:15:05.851 --> 00:15:10.534
It is a heavy token user, but it is very effective, especially for things that are complicated.

258
00:15:11.094 --> 00:15:13.415
So we'll use this when we're like, this is hard.

259
00:15:13.475 --> 00:15:14.596
I don't know the right answer.

260
00:15:14.616 --> 00:15:16.696
Because, you know, sometimes it's like, do you want to do this or this?

261
00:15:16.756 --> 00:15:18.617
And you're like, is there a magical third part?

262
00:15:18.937 --> 00:15:19.958
Like, is there something else here?

263
00:15:19.978 --> 00:15:21.178
How do we connect these two things?

264
00:15:21.198 --> 00:15:22.799
Because these seem like opposing.

265
00:15:23.059 --> 00:15:26.441
This is a way for us to avoid that conversation and just be like, you do it.

266
00:15:26.761 --> 00:15:29.362
But no, not you do it like I'm releasing autonomy.

267
00:15:29.782 --> 00:15:31.863
You do it like let's do it five times.

268
00:15:32.003 --> 00:15:34.004
And then you pick and judge the best one.

269
00:15:34.584 --> 00:15:38.107
So this is part of the reason you have 14 windows up at the same time.

270
00:15:38.347 --> 00:15:40.488
You're having these long running sessions.

271
00:15:40.909 --> 00:15:42.069
Everything is long running.

272
00:15:42.270 --> 00:15:45.372
And part of the program, I think, is that you do have these long running sessions.

273
00:15:45.432 --> 00:15:49.095
And then each one of these is a separate product, so to speak.

274
00:15:49.335 --> 00:15:51.136
Like, I don't really believe in work trees.

275
00:15:51.156 --> 00:15:53.077
I've never emotionally figured out how to use them.

276
00:15:53.638 --> 00:15:56.320
Like, I'm sure they're helpful, but I can't wrap my head around it.

277
00:15:56.340 --> 00:16:00.583
And so this is really like we're building a mobile app and we're trying to figure out how to make the login better.

278
00:16:01.383 --> 00:16:05.427
Not better from a UX standpoint, but just like tactically and technically better.

279
00:16:05.887 --> 00:16:06.948
This is a good way to do that.

280
00:16:07.008 --> 00:16:09.150
Build the login system five times.

281
00:16:09.791 --> 00:16:14.074
See which one is better from a various adversarial review from the LLM.

282
00:16:15.135 --> 00:16:20.340
Are you doing these in parallel or are you doing them serially and feeding kind of the previous takes?

283
00:16:20.440 --> 00:16:22.642
These are a bit differently than these previous takes.

284
00:16:23.462 --> 00:16:28.829
Yeah, so we release the decisioning on that and use subagents to do the implementations.

285
00:16:29.289 --> 00:16:35.677
So it really does fan out and build the five implementations or three implementations and then use a judge to judge them.

286
00:16:36.178 --> 00:16:41.785
This is another way that, you know, one of the things that we've been thinking a lot about because we do a lot of headless code agents.

287
00:16:42.225 --> 00:16:44.966
where we don't have the ability to see into what's going on.

288
00:16:44.986 --> 00:16:47.968
We can see what's going on, but you're not guiding it in any way.

289
00:16:47.988 --> 00:16:50.849
You're not using skills to guide it in the same way you use Cloud Code.

290
00:16:50.869 --> 00:16:59.973
One of the things that we've been thinking a lot about is a lot of the evals that we're comfortable with are task-level evals, not spec-level evals.

291
00:17:00.473 --> 00:17:04.235
And we're coming close to the point where we need spec level avals.

292
00:17:04.295 --> 00:17:10.799
We need something that is saying from the beginning to the end, this is more effective of a process than this is a process.

293
00:17:11.219 --> 00:17:26.227
And so we do a lot of things where we'll just build stuff five times using different tools, different models, and then have another judge be somewhat of an adversarial judge of all of those things, and then merge them into one product that is better, pulling the best from each of those.

294
00:17:26.587 --> 00:17:33.110
That's a very intensive, user-intensive process, and it's not always effective from a time standpoint.

295
00:17:33.670 --> 00:17:34.470
I'm intrigued.

296
00:17:34.651 --> 00:17:35.231
Let's try it out.

297
00:17:35.931 --> 00:17:44.275
Your social media feed feels like it has some parallels with Jesse's I love you stuff.

298
00:17:45.193 --> 00:17:46.054
and affirmations.

299
00:17:46.654 --> 00:17:48.336
I can't quite put my finger on it.

300
00:17:48.756 --> 00:17:49.917
They're both very human.

301
00:17:50.338 --> 00:17:57.884
I think that Jesse and I have both thought from the beginning that the answer to this is more humanism, not less.

302
00:17:58.225 --> 00:18:04.370
And a lot of times when people say, I'm going to interact with this, they're thinking, oh, this is Excel.

303
00:18:04.670 --> 00:18:08.013
This is just like super exciting Excel, really effective Excel.

304
00:18:08.514 --> 00:18:09.334
And I think that's,

305
00:18:10.195 --> 00:18:10.576
Fine.

306
00:18:10.816 --> 00:18:14.682
But do you want boring Excel or do you want exciting Excel?

307
00:18:14.902 --> 00:18:16.905
And exciting Excel is not speed.

308
00:18:16.985 --> 00:18:19.309
I think exciting Excel is when it talks shit back to you.

309
00:18:20.490 --> 00:18:21.511
Okay, this sounds awesome.

310
00:18:21.731 --> 00:18:22.791
I'd love to try it out.

311
00:18:22.911 --> 00:18:25.873
So I'm going to spin up a Claude session here.

312
00:18:25.893 --> 00:18:27.054
Yeah, and we'll get going.

313
00:18:27.094 --> 00:18:31.036
Harper, with these tools, so you clearly love to build.

314
00:18:31.516 --> 00:18:36.359
And you've said in the past that you literally cannot stop building.

315
00:18:36.379 --> 00:18:37.199
Yeah.

316
00:18:37.219 --> 00:18:38.660
You don't know how to stop.

317
00:18:39.381 --> 00:18:44.744
And then at the same time, you've described what you build erupting into negative impact.

318
00:18:45.464 --> 00:18:46.304
What is that negative impact?

319
00:18:46.344 --> 00:18:46.565
Yeah.

320
00:18:47.763 --> 00:18:53.244
For my case specifically, we did crowdsourcing before a lot of other people did crowdsourcing.

321
00:18:53.304 --> 00:18:59.245
And crowdsourcing has been used for all sorts of bad things over time.

322
00:18:59.325 --> 00:19:03.786
And we did really aggressive social media targeting before a lot of people did that.

323
00:19:03.846 --> 00:19:05.086
And that has obviously been used.

324
00:19:05.186 --> 00:19:11.467
Like we invented the, in our case, it was just an information canon, but it's a misinformation, disinformation canon.

325
00:19:11.487 --> 00:19:12.908
Like we literally invented that.

326
00:19:13.008 --> 00:19:14.888
Like the Cambridge Analytica stuff that happened

327
00:19:15.468 --> 00:19:19.991
was based entirely on the work that my team and the analytics team of the Obama campaign did.

328
00:19:20.391 --> 00:19:23.193
They just had a little better turn on it and so on and so forth.

329
00:19:23.233 --> 00:19:30.538
Like, I think that there's one of the things that there is an irrational exuberism that you have to have as an entrepreneur.

330
00:19:30.938 --> 00:19:36.842
And one of the downsides of that irrational exuberism is that you don't, and you shouldn't, or maybe you should, I don't know.

331
00:19:37.282 --> 00:19:41.305
I don't think we can think a lot about the downstream effects of the work we're building.

332
00:19:42.194 --> 00:19:53.238
And so I think what ends up happening is you have all of these great people building all of these great things, but no one is thinking, oh, this is going to be used to like manufacture a genocide a la Facebook in Myanmar.

333
00:19:53.338 --> 00:19:59.499
Oh, this is going to be used to help make sure people can buy more things very quick a la TikTok shop.

334
00:19:59.539 --> 00:20:03.241
Like people are building things because they love them and they have a passion for building.

335
00:20:03.261 --> 00:20:06.562
They're not building things because they think they're going to fuck over a community or whatever.

336
00:20:06.602 --> 00:20:07.122
Not everyone.

337
00:20:07.682 --> 00:20:09.185
And I just want to push on this.

338
00:20:09.205 --> 00:20:10.969
Do you think it's a like a cape?

339
00:20:11.209 --> 00:20:15.136
Is it they're not capable or they're not interested because this is very personal to me.

340
00:20:15.357 --> 00:20:17.862
I rolled out financial recognition and detection at Facebook.

341
00:20:18.362 --> 00:20:28.164
Yeah, I think it's, yeah, it's, it's, the other problem is that there's very few people who will talk openly about being part of a program that was not good for society.

342
00:20:28.904 --> 00:20:33.785
My friend, Mike Evans wrote a book called Hangry about his time at Grubhub and founding Grubhub.

343
00:20:33.805 --> 00:20:39.046
And he talks about how he hit like Grubhub has been linked with increasing the waist sizes of America.

344
00:20:40.186 --> 00:20:44.347
Part of this is that we don't have any stories about people talking about it because it's,

345
00:20:45.503 --> 00:20:59.429
I don't think people want to admit, first of all, like I've never met a Facebook growth person who wasn't proud of what the work they did, which typically was using an algorithm to trick people into clicking on ads, right?

346
00:20:59.469 --> 00:21:02.050
And so it's like, like tell me as your first question.

347
00:21:02.867 --> 00:21:04.288
Okay, but so that's one.

348
00:21:04.668 --> 00:21:06.129
I know dozens and dozens.

349
00:21:06.169 --> 00:21:09.051
So the point is, I think that the narrative there is not built.

350
00:21:09.131 --> 00:21:16.076
The other thing is that when you're an employee, for some reason, we give up autonomy of decision to the people who are the bosses.

351
00:21:16.797 --> 00:21:21.880
So it's much easier for someone to say, oh, I was just doing my job, which obviously is a famous thing.

352
00:21:22.217 --> 00:21:25.158
bad answer, but it's still, it's a hard one.

353
00:21:25.178 --> 00:21:34.161
You hear it today, like literally these days, even though it's like written very well and very openly in many examples of that doesn't, that doesn't fly.

354
00:21:34.441 --> 00:21:35.622
So I think we have a couple of things.

355
00:21:35.642 --> 00:21:41.143
I think the first thing is that we often give up autonomy towards our organizations and our bosses and just, we're just part of a cog.

356
00:21:41.604 --> 00:21:43.884
I think part of that is because it's expense.

357
00:21:43.944 --> 00:21:45.045
The alternative is expensive.

358
00:21:45.665 --> 00:21:45.845
Right.

359
00:21:45.865 --> 00:21:52.053
Like the alternative is you have to speak up, you have to fight and you probably aren't going to make any money.

360
00:21:52.494 --> 00:21:54.696
You're not going to send your kid to the school you want.

361
00:21:54.817 --> 00:21:57.260
You might not be able to afford the lifestyle you want because it's noisy.

362
00:21:57.740 --> 00:22:02.366
If you look at a lot of activists, they don't really have good jobs from the capitalist standpoint.

363
00:22:02.386 --> 00:22:03.347
That's because they spoke up.

364
00:22:04.047 --> 00:22:11.016
And so I think there's a link between the cost of speaking up and the effectiveness of the work.

365
00:22:11.056 --> 00:22:12.277
Like it's just expensive to speak up.

366
00:22:12.297 --> 00:22:16.603
But the second thing is we don't look retrospectively on our work very often.

367
00:22:17.304 --> 00:22:21.269
So we have very few tech leaders who are like, oh, that was a bad idea.

368
00:22:21.741 --> 00:22:22.481
Shouldn't have done that.

369
00:22:22.561 --> 00:22:24.242
That kind of opened a Pandora's box.

370
00:22:24.702 --> 00:22:40.189
And the reality is, if you were able to transport my brain back to 2012 and you say, hey, you're on the cusp of enabling what's going to be the misinformation cannon in Ukraine in 2014 and the U.S. in 2016, so on and so forth, using social media targeting and all that stuff.

371
00:22:40.209 --> 00:22:42.590
We didn't do psychometrics, but that's about the only difference.

372
00:22:43.150 --> 00:22:43.790
Are you going to stop?

373
00:22:43.910 --> 00:22:45.131
I'm going to be like, no, I got to win.

374
00:22:45.771 --> 00:22:50.215
So this is much more complicated than just like, oh, you can invent a time machine and go back and stop it.

375
00:22:50.736 --> 00:22:53.218
We still have to achieve the things that we want to achieve.

376
00:22:53.279 --> 00:22:59.705
I just think we don't have a cautious or even a thoughtful narrative about whether we should.

377
00:23:00.606 --> 00:23:03.729
We always can build it, but the should is really complicated.

378
00:23:03.749 --> 00:23:08.774
I think it's ultimately linked to capitalism, and I think there's some Western fast food capitalism there.

379
00:23:09.054 --> 00:23:10.115
aspects of this as well.

380
00:23:10.175 --> 00:23:12.016
So I don't want to say this is an easy solution.

381
00:23:12.576 --> 00:23:15.397
My way of dealing with this is I talk about it a little bit more.

382
00:23:15.617 --> 00:23:16.638
I try and talk about it.

383
00:23:17.238 --> 00:23:20.340
So your comment about fast food, I think, is very interesting.

384
00:23:20.720 --> 00:23:24.582
I'll tell you how I understand my work at Facebook.

385
00:23:25.289 --> 00:23:32.815
which is I tricked myself into believing that I was building product that was good for people because they were using it.

386
00:23:33.616 --> 00:23:39.601
And all the haters were just micromanaging from the outside, trying to backseat drive.

387
00:23:40.302 --> 00:23:53.133
And I knew the products I released, including facial recognition, were good because people used them and they wouldn't use things that are, this is where I was wrong, is I believe that people wouldn't use things that were ultimately good for them.

388
00:23:53.690 --> 00:24:03.713
And now I feel like actually I was a drug dealer and it's like somebody selling heroin on the street and saying it's good for people because people are using it.

389
00:24:04.382 --> 00:24:05.903
Yeah, and I think this is part of what I mean.

390
00:24:05.943 --> 00:24:24.498
Like when your focus is on metrics of use and you're optimizing for use, like just use, you're not optimizing for positive outcome or positive impact or these other things that are hard and impossible to measure, then you're always going to be in this thing of like we can prove usage, right?

391
00:24:24.518 --> 00:24:26.120
You can't prove impact.

392
00:24:26.160 --> 00:24:26.940
It's very difficult.

393
00:24:26.980 --> 00:24:32.085
You can't, or more importantly, we have relegated the people who would have that conversation to the people who would have that conversation

394
00:24:32.745 --> 00:24:37.247
to user UX people, design people, product managers maybe.

395
00:24:37.327 --> 00:24:43.670
And most of those people we have also deprioritized or removed power from in most organizations where we're like, ah, fuck them.

396
00:24:43.690 --> 00:24:44.690
They don't know what they're talking about.

397
00:24:44.710 --> 00:24:45.790
They're talking to users.

398
00:24:46.010 --> 00:24:47.851
The metrics look like it's working great.

399
00:24:47.931 --> 00:24:52.673
And it's like, it's part and parcel to the whole tech organization, which is our world.

400
00:24:52.773 --> 00:24:53.454
That's where we live.

401
00:24:53.834 --> 00:24:56.755
I don't, I'm not demanding like a sea change.

402
00:24:56.815 --> 00:24:58.776
I'm more demanding we should just be thoughtful.

403
00:24:59.156 --> 00:24:59.696
Do you know what I mean?

404
00:24:59.716 --> 00:25:01.457
Like just be thoughtful about that so that,

405
00:25:02.077 --> 00:25:07.586
rather than you having to discover through some life experience and some distance that maybe that was not the right path.

406
00:25:07.646 --> 00:25:10.691
Like how do we introduce that during so we can have that discussion?

407
00:25:11.012 --> 00:25:12.915
The end product might be the same, which is my point.

408
00:25:13.035 --> 00:25:14.357
I think I would have made the same decision.

409
00:25:15.402 --> 00:25:20.524
knowing everything I know because I thought it was so important to do, but at least I could have that decision.

410
00:25:21.124 --> 00:25:25.446
But yeah, it's like if you talk to someone like there's lots of examples of this, like we can go back.

411
00:25:25.526 --> 00:25:30.568
I do think about this a lot because I don't think we have enough conversations about should we have done that?

412
00:25:30.668 --> 00:25:30.968
Would we?

413
00:25:31.148 --> 00:25:35.250
I also think regret is not fun to deal with and I'm not one to regret.

414
00:25:35.330 --> 00:25:41.372
I don't really regret this stuff, but I do think it's worthwhile pointing out that like maybe that was a bell that shouldn't have been rung.

415
00:25:42.512 --> 00:25:42.832
All right.

416
00:25:42.852 --> 00:25:44.153
Speaking of use, Tom, you ready?

417
00:25:45.155 --> 00:25:47.617
Yeah, I'm intrigued by test fiction.

418
00:25:47.777 --> 00:25:48.798
Let's get a session going.

419
00:25:48.818 --> 00:25:49.879
Let's light up Claude.

420
00:25:51.200 --> 00:25:54.643
I typically run Claude with no permissions, but you got to do what you got to do.

421
00:25:55.063 --> 00:26:01.349
And if you want to do that, we can do that with dangerously skip permissions, which I now know how to spell all of those words in order very quickly, but let's do this.

422
00:26:01.449 --> 00:26:05.052
Okay, so now we're going to want to install the plugin.

423
00:26:05.392 --> 00:26:10.637
Now the plugin lives at our kind of plugin area, which is skills.2389.com.

424
00:26:12.517 --> 00:26:13.258
dot AI.

425
00:26:13.658 --> 00:26:14.138
Oh, that's good.

426
00:26:14.158 --> 00:26:26.807
Okay, now type slash plugin space marketplace space ad space 2389 dash research slash Claude plugins dash.

427
00:26:28.448 --> 00:26:29.408
Yep, dash plugins.

428
00:26:30.129 --> 00:26:30.729
See if this works.

429
00:26:30.969 --> 00:26:33.631
Oh, it's looking that's looking positive.

430
00:26:34.452 --> 00:26:34.832
Okay.

431
00:26:35.397 --> 00:26:36.398
Looks so good.

432
00:26:36.418 --> 00:26:37.840
Now we can go and install it.

433
00:26:38.101 --> 00:26:38.721
Let's try it.

434
00:26:39.622 --> 00:26:41.104
Yeah, you get to choose your scope.

435
00:26:41.545 --> 00:26:43.688
I don't have a preference here, but whichever works.

436
00:26:43.908 --> 00:26:45.951
Okay, now we need to reload our plugins.

437
00:26:46.852 --> 00:26:47.192
Teach me.

438
00:26:47.673 --> 00:26:51.418
Oh, you need to type slash reload dash plugins to apply it.

439
00:26:53.247 --> 00:26:53.847
Then we should be good.

440
00:26:54.127 --> 00:26:54.928
And then it should reload it.

441
00:26:55.008 --> 00:26:55.848
Okay, now we're good.

442
00:26:55.988 --> 00:26:56.709
So we have this plugin.

443
00:26:56.929 --> 00:26:57.709
Why don't we just try it?

444
00:26:57.789 --> 00:26:58.950
So let's build something.

445
00:26:59.330 --> 00:27:05.532
And you could, you know, how I would do this, I would say, using the Omikase skill, let's build X, Y, and Z.

446
00:27:05.813 --> 00:27:09.594
Or if you have a spec that's already there, we can use that to do a feature.

447
00:27:09.654 --> 00:27:13.056
But I think this is probably the most effective start.

448
00:27:13.256 --> 00:27:14.957
So what kind of, what do we want to build?

449
00:27:14.977 --> 00:27:16.897
What do you want to build?

450
00:27:17.198 --> 00:27:19.779
I always just build a Pong game and then I try to make it work.

451
00:27:19.799 --> 00:27:20.219
Let's do it.

452
00:27:21.326 --> 00:27:21.546
Yeah.

453
00:27:21.666 --> 00:27:23.987
Using Omikaze skill, let's build a Pong game.

454
00:27:24.788 --> 00:27:27.789
Should we do it in some, some esoteric language or something?

455
00:27:29.870 --> 00:27:30.570
Yeah, sure.

456
00:27:30.710 --> 00:27:32.391
We want to build it in Zig.

457
00:27:33.351 --> 00:27:33.571
Yeah.

458
00:27:33.591 --> 00:27:34.071
Do you have a Zig?

459
00:27:34.091 --> 00:27:35.412
Do you have the Zig stuff installed?

460
00:27:35.852 --> 00:27:37.673
No, it's too esoteric even.

461
00:27:37.733 --> 00:27:38.834
Okay, let's not do Zig.

462
00:27:38.914 --> 00:27:42.836
I like that I always choose Zig as well as my esoteric one, and then I'm always like, oh, man.

463
00:27:43.136 --> 00:27:45.838
Why don't we just do... And have it make it weird, though, right?

464
00:27:47.079 --> 00:27:47.679
Great, perfect.

465
00:27:47.699 --> 00:27:50.400
Web tech, but make it weird.

466
00:27:51.041 --> 00:27:51.641
We'll see what it does.

467
00:27:53.142 --> 00:27:53.603
This is great.

468
00:27:53.723 --> 00:27:54.744
Hopefully it picks up the skill.

469
00:27:54.764 --> 00:27:57.306
This is always where you're like, oh, it looks like it did.

470
00:27:57.606 --> 00:28:07.695
One of the things about building skills is I think the hardest part is ensuring that whatever kind of pros you choose, the user chooses, that you're able to.

471
00:28:07.855 --> 00:28:09.257
Yeah, so you're going to want to do this.

472
00:28:09.277 --> 00:28:09.977
Say proceed.

473
00:28:09.997 --> 00:28:11.378
I would do all of them because fuck it.

474
00:28:11.739 --> 00:28:12.079
Okay.

475
00:28:12.099 --> 00:28:12.900
Is this error a problem?

476
00:28:12.920 --> 00:28:14.161
I would just say yes, allow everything.

477
00:28:15.082 --> 00:28:16.403
That seems like a problem.

478
00:28:16.423 --> 00:28:17.664
So let's see what happens.

479
00:28:18.385 --> 00:28:18.605
Okay.

480
00:28:19.350 --> 00:28:22.957
One of my favorite parts about Cloud is when it can't find the skill, which it sometimes does.

481
00:28:23.158 --> 00:28:24.420
It'll just be like, let me look for it.

482
00:28:24.440 --> 00:28:25.362
That skill doesn't exist.

483
00:28:25.462 --> 00:28:26.625
So I would say yes to all.

484
00:28:26.665 --> 00:28:27.827
So let it just rip through stuff.

485
00:28:28.468 --> 00:28:29.069
Oh, what do we got here?

486
00:28:30.012 --> 00:28:31.272
Yeah, so this is just rocking.

487
00:28:31.553 --> 00:28:40.736
So one of the things I like about this is hopefully, like if you scroll up a little bit, it leans on the omakase language of this idea that like, let's leave it up to the chef.

488
00:28:41.396 --> 00:28:44.397
Let's let the chef's choice, which I think is a really important thing.

489
00:28:44.417 --> 00:28:49.519
That's what omakase is, is that you're putting your hands in the chef's hands or your experience in the chef's hands.

490
00:28:49.999 --> 00:28:53.520
And the reason you do that is because the chef is better at it than you are.

491
00:28:54.020 --> 00:29:02.268
And I think that's one of the fundamental aspects here is we really believe that for the most part, these LLMs are probably better at stuff than I am about various things.

492
00:29:02.308 --> 00:29:04.970
It's certainly better at using Vite for me.

493
00:29:05.050 --> 00:29:07.793
It's certainly better at using React than I am.

494
00:29:08.093 --> 00:29:13.118
I would say it probably can handle the DT tag a little better than me, but maybe not the select tag.

495
00:29:13.178 --> 00:29:18.203
But the point is, rather than being pedantic about what it is good or bad at, just let it do the best thing.

496
00:29:18.683 --> 00:29:21.004
With the idea that for the most part, we'll get a good result.

497
00:29:21.044 --> 00:29:22.725
So why don't we just say yes to proceed?

498
00:29:22.765 --> 00:29:23.485
I would go full.

499
00:29:23.525 --> 00:29:24.586
Yeah, let's lean in.

500
00:29:25.206 --> 00:29:27.247
And we talked a lot about naming.

501
00:29:27.547 --> 00:29:28.767
Like, what is the name?

502
00:29:29.108 --> 00:29:40.632
And because I was more excited about when we were building this, I was much more focused on the cook-off aspect of doing a feature with like five versions and then fanning it out to five versions and fanning it back in.

503
00:29:40.652 --> 00:29:41.633
That was what I was excited about.

504
00:29:42.033 --> 00:29:47.374
Dylan, my co-founder, was really excited about not having to make a lot of the smaller decisions.

505
00:29:47.694 --> 00:29:49.695
A part of it, like you just said, let's make Pong.

506
00:29:49.915 --> 00:29:50.815
It knows what Pong is.

507
00:29:50.835 --> 00:29:51.955
And you said, let's make it weird.

508
00:29:52.295 --> 00:29:54.656
And so it doesn't have to be like, how many balls do you want?

509
00:29:54.936 --> 00:30:04.338
It's jumping ahead with that stuff, which I think is the right approach for some of these things, especially if you don't need to have the pedantic interaction with, I need control of every single thing.

510
00:30:04.778 --> 00:30:09.099
Obviously, this is not going to work for everything, but it works pretty well with some of this fun stuff.

511
00:30:09.139 --> 00:30:10.620
And we use it all over the place.

512
00:30:11.940 --> 00:30:17.223
Like, let's say we're like, hey, we need an interface to interrogate session logs for cloud or codex.

513
00:30:17.263 --> 00:30:18.544
This would be a great example of that.

514
00:30:18.564 --> 00:30:20.645
Like, I just need to be able to see token cost.

515
00:30:20.685 --> 00:30:22.766
And it would build the interface and build all that stuff.

516
00:30:23.167 --> 00:30:28.570
You can also push it in ways like you could say it's important that it doesn't look like X, Y, and Z.

517
00:30:28.690 --> 00:30:32.652
So you can actually have autonomy on some things that matter to you and not autonomy on the rest.

518
00:30:33.073 --> 00:30:34.293
What are the downsides of this?

519
00:30:34.313 --> 00:30:36.735
And it might be worth like looking at what the subagents are doing.

520
00:30:37.738 --> 00:30:46.504
One of the downsides of this, though, is that it is like watching a submarine race, even more so than regular agents, which is like you're just sitting there being like, cool.

521
00:30:47.665 --> 00:30:51.468
Then like time goes by and you're like, great, glad that's still going.

522
00:30:51.908 --> 00:30:52.849
So why don't we click in?

523
00:30:52.889 --> 00:30:57.252
If you look at the bottom of your terminal, click into one of the general purpose or whatever it says right there.

524
00:30:57.272 --> 00:31:00.615
Let's click on one of those and see if we can see what's happening in that one.

525
00:31:01.502 --> 00:31:03.004
So as you can see, this is just rocking.

526
00:31:03.024 --> 00:31:08.429
And I think the key, the most important part there is if you look, the last word on each of those subagents is variant.

527
00:31:09.010 --> 00:31:11.732
And that's, I think, the real cool thinking about this stuff.

528
00:31:11.852 --> 00:31:18.219
It's not that this is doing sub-agent development, which is like one of my favorite aspects of...

529
00:31:18.659 --> 00:31:19.219
superpowers.

530
00:31:19.519 --> 00:31:22.081
When that was released, that really was the biggest unlock for me.

531
00:31:22.141 --> 00:31:27.143
It's like, oh, you can just have sub-agents that have this clear, clean context to do this work.

532
00:31:27.543 --> 00:31:40.910
This is instead saying, I'm going to have the sub-agents, because you have the opportunity for clear, clean context, building on that sub-agent development, I'm going to have each of them do a different variant to then compare, choose a winner, et cetera.

533
00:31:41.190 --> 00:31:44.272
So what this means, if you're dealing with a hard project, hard project.

534
00:31:44.792 --> 00:31:48.816
feature, something that you're just not quite sure of, you can kind of incept it.

535
00:31:48.836 --> 00:31:51.678
You can say, hey, I'm struggling to do this thing.

536
00:31:51.879 --> 00:31:57.644
Why don't you use Omikase to spin up 10 versions and let's see which one's best.

537
00:31:57.804 --> 00:31:59.205
And it will really just do 10 versions.

538
00:31:59.225 --> 00:32:00.126
Looks like we're almost done.

539
00:32:00.146 --> 00:32:03.409
Let's see if we can complete these last two versions and kind of, well, what does it say?

540
00:32:03.630 --> 00:32:04.590
It popped out some good stuff.

541
00:32:04.711 --> 00:32:05.251
Reflection.

542
00:32:05.291 --> 00:32:05.812
Let's read that.

543
00:32:06.572 --> 00:32:10.113
Yeah, so it's like, it has all these things of this report.

544
00:32:10.213 --> 00:32:13.474
And then what's nice is it will, is it done?

545
00:32:14.314 --> 00:32:15.774
Oh, no, this is the first variant one.

546
00:32:15.954 --> 00:32:22.896
Harper, does Test Kitchen seed the variants with some sort of... Randomness or something?

547
00:32:23.176 --> 00:32:23.876
No, it doesn't.

548
00:32:23.896 --> 00:32:25.837
Or like different approaches to the problem?

549
00:32:26.297 --> 00:32:29.358
This might be an opportunity for you to say, why don't you choose...

550
00:32:30.669 --> 00:32:36.390
And so it will give you options because one of the things that we notice is that some people really want to see the variance.

551
00:32:36.590 --> 00:32:39.191
They really want to like be a part of the process.

552
00:32:39.551 --> 00:32:44.592
And because you never know, like you might say, oh, wow, I really like the quantum one, but I also like gravity.

553
00:32:44.612 --> 00:32:46.152
Like this allows you to do.

554
00:32:46.273 --> 00:32:47.393
But let's see.

555
00:32:47.413 --> 00:32:47.873
Now, here it is.

556
00:32:47.953 --> 00:32:49.993
Let me actually read all four and judge them for weakness.

557
00:32:50.013 --> 00:32:52.594
So now we're jumping to like the adversarial review of it.

558
00:32:52.774 --> 00:32:53.614
It has it doesn't.

559
00:32:53.834 --> 00:32:56.795
And I think I say adversarial, not in that it is a.

560
00:32:57.535 --> 00:33:02.840
mean review or it's an enemy review and that kind of way, because I think some people lean into this pedantically.

561
00:33:03.140 --> 00:33:06.403
What I mean is that it doesn't have any of the context for how or why it built it.

562
00:33:06.904 --> 00:33:09.046
It just has the high level context of what it's supposed to do.

563
00:33:09.086 --> 00:33:10.427
And so it is not the same.

564
00:33:10.787 --> 00:33:13.029
It is not the same role that built the thing.

565
00:33:13.070 --> 00:33:15.592
So it's actually going to review it with clean context.

566
00:33:15.652 --> 00:33:18.214
It's not going to know the decisions or why or internally.

567
00:33:18.234 --> 00:33:20.957
It's only going to know that it's called variant existential.

568
00:33:21.406 --> 00:33:21.826
There we go.

569
00:33:22.246 --> 00:33:32.349
Okay, what we're looking at right here is Test Kitchen produced four different variants of Pong, and you had four different websites there, URLs, and now it's picking one.

570
00:33:32.789 --> 00:33:33.329
Is that right?

571
00:33:33.990 --> 00:33:35.030
Yeah, so it looks like it had.

572
00:33:35.090 --> 00:33:36.890
It went through, and it judged it.

573
00:33:36.950 --> 00:33:38.271
It said gravity is well-built.

574
00:33:38.651 --> 00:33:41.312
Polar is most structurally novel, and it tells you why.

575
00:33:41.392 --> 00:33:42.592
Quantum is the prettiest.

576
00:33:42.952 --> 00:33:45.213
Existential is the only one where I laugh reading the code.

577
00:33:45.233 --> 00:33:47.834
And this is the thing that I think is important about these agents.

578
00:33:48.354 --> 00:33:48.894
Did it laugh?

579
00:33:48.914 --> 00:33:49.094
No.

580
00:33:49.434 --> 00:33:50.015
I don't care.

581
00:33:50.235 --> 00:33:51.856
It's funny that it said that.

582
00:33:51.957 --> 00:33:55.320
36 dialogue lines spanning Camus to I had a podcast.

583
00:33:55.360 --> 00:33:56.121
It didn't take off.

584
00:33:56.581 --> 00:33:58.002
Amudbar, the response.

585
00:33:58.223 --> 00:33:59.804
So like this is funnier.

586
00:33:59.844 --> 00:34:01.886
Like it's definitely a funnier one.

587
00:34:02.186 --> 00:34:04.188
And it is much weirder than quantum.

588
00:34:04.228 --> 00:34:04.769
Okay, great.

589
00:34:04.789 --> 00:34:05.950
You could imagine quantum.

590
00:34:05.970 --> 00:34:07.151
You could imagine polar.

591
00:34:07.452 --> 00:34:08.032
All these things.

592
00:34:08.092 --> 00:34:08.793
And I like this.

593
00:34:09.193 --> 00:34:12.937
So what I would do is I would just say like, let's polish it.

594
00:34:13.117 --> 00:34:18.022
But the thing that's important here is that is number two there, right?

595
00:34:18.542 --> 00:34:19.904
Delete the losing three.

596
00:34:20.204 --> 00:34:22.106
This is not about cruft.

597
00:34:22.166 --> 00:34:25.069
This is not about instantly putting a bunch of code debt.

598
00:34:25.089 --> 00:34:29.574
This is about doing four tries and then getting rid of the three that failed.

599
00:34:29.914 --> 00:34:34.458
What we've seen is that this will pop up and give you four tries and it will say all of them are equally good.

600
00:34:34.839 --> 00:34:35.660
I want to merge them.

601
00:34:36.020 --> 00:34:38.822
We've seen them say everything is terrible but this one.

602
00:34:38.942 --> 00:34:40.683
We've seen them say they're all bad.

603
00:34:40.783 --> 00:34:41.443
Let's do it again.

604
00:34:41.484 --> 00:34:43.685
Like we've seen it have all of these different turns.

605
00:34:44.185 --> 00:34:51.270
We're just trying to provide a framework that is trying to trigger the LLMs into easily doing what we like to call slot machine development.

606
00:34:51.630 --> 00:34:52.571
Let's do it five times.

607
00:34:52.591 --> 00:34:56.253
Let's do it three times and pick the winner because we don't know which one is the winner.

608
00:34:56.353 --> 00:35:03.217
And you know this, like every one of us has probably since the early 2000s, their thing that they build when they try a new programming language or whatever.

609
00:35:03.277 --> 00:35:05.579
For the longest time for me, it was literally a blogging platform.

610
00:35:05.959 --> 00:35:07.520
So I'd build this mini blogging platform.

611
00:35:07.800 --> 00:35:09.140
And if you, it could be an editor.

612
00:35:09.160 --> 00:35:12.581
Right now for me, it is the DVD, bouncing DVD logo.

613
00:35:12.601 --> 00:35:15.662
That's a fun one, but in a terminal or web or wherever.

614
00:35:16.042 --> 00:35:16.702
But you have this.

615
00:35:16.722 --> 00:35:19.263
And so you can just say, do that thing.

616
00:35:19.403 --> 00:35:22.164
And it will do it five times in a very different way every time.

617
00:35:22.204 --> 00:35:23.264
And you can judge it yourself.

618
00:35:23.785 --> 00:35:24.945
And it's really impressive.

619
00:35:24.965 --> 00:35:29.226
And the reason I wondered about doing some esoteric language is because it's very effective in everything.

620
00:35:29.707 --> 00:35:34.168
And it's more effective if you don't know or don't care about the outcome.

621
00:35:34.548 --> 00:35:35.950
Meaning, oh, I'm doing this in Rust.

622
00:35:36.410 --> 00:35:37.692
I know Rust approximately.

623
00:35:37.712 --> 00:35:40.596
We do a lot of Rust, but I don't know the details of Rust.

624
00:35:40.616 --> 00:35:43.079
So one will probably not have a lot of libraries.

625
00:35:43.159 --> 00:35:45.061
One will be max libraries, so on and so forth.

626
00:35:45.778 --> 00:35:49.681
How often are you developing with Test Kitchen versus a single thread?

627
00:35:50.342 --> 00:35:56.106
And when do you choose Test Kitchen and multi-thread versus, hey, this is easy, I just want one?

628
00:35:56.266 --> 00:36:03.012
Yeah, I don't use the omakase side as much because I'm usually coming into this with something that is less or more defined.

629
00:36:03.292 --> 00:36:09.537
So I use the actual, what they call the cook-offs, which is like omakase for just coding.

630
00:36:10.057 --> 00:36:12.921
not Omikase for specs or what have you.

631
00:36:13.101 --> 00:36:14.342
And I use that quite often.

632
00:36:14.723 --> 00:36:18.988
It really depends on the product, meaning do I know how it should be?

633
00:36:19.529 --> 00:36:26.377
Like if I have a spec that I use superpowers or chat to your cloud or whatever to really iterate and I'm really confident that the spec is correct,

634
00:36:27.198 --> 00:36:30.641
then I will just let, I'll just do, we'll pop it into our headless machine.

635
00:36:30.661 --> 00:36:34.564
It'll just go through it or use Cloud and superpowers and just rip through it or codex or whatever.

636
00:36:34.964 --> 00:36:38.827
If it is not that, if I'm not as confident, like I know what it's supposed to do.

637
00:36:38.867 --> 00:36:43.871
I'm like, oh, we need to build up an AI proxy that puts all our tokens through so we can get some ideas out of it or whatever.

638
00:36:44.431 --> 00:36:50.196
Or like, or like build this, or like an example was the mics that are around our office, like the agent that does that.

639
00:36:50.316 --> 00:36:52.938
I use Test Kitchen just to be like, hey, rip through this and

640
00:36:53.318 --> 00:36:55.861
I don't really care how it's done, but you need to pick the best one.

641
00:36:55.921 --> 00:36:57.082
So why don't you do it five times?

642
00:36:57.122 --> 00:36:57.763
That kind of thing.

643
00:36:58.043 --> 00:36:58.824
That's quite often.

644
00:36:58.884 --> 00:37:00.766
It really matters on how much autonomy I want.

645
00:37:01.046 --> 00:37:03.348
Honestly, did I wake up wanting more autonomy?

646
00:37:03.749 --> 00:37:04.289
Do I care?

647
00:37:04.790 --> 00:37:06.011
Let's go ahead and say yes here.

648
00:37:06.511 --> 00:37:08.133
Do you ever go back and kind of sanity check?

649
00:37:08.433 --> 00:37:10.736
Like, no, really, you'll just never choose.

650
00:37:10.796 --> 00:37:12.117
But sometimes you go back and look.

651
00:37:12.397 --> 00:37:13.098
No, that's not.

652
00:37:13.298 --> 00:37:13.998
Oh, no, that's true.

653
00:37:14.018 --> 00:37:14.999
That's a good thing.

654
00:37:15.019 --> 00:37:15.659
That's a good point.

655
00:37:16.239 --> 00:37:17.820
I find that I have...

656
00:37:18.040 --> 00:37:22.642
I feel sometimes like Homer and that bird that just presses enter or the continue key or whatever.

657
00:37:22.822 --> 00:37:23.922
Like, I feel like that sometimes.

658
00:37:23.942 --> 00:37:24.382
Yeah, I'm just...

659
00:37:24.402 --> 00:37:25.903
I feel like that's our lives now.

660
00:37:26.263 --> 00:37:27.604
And so sometimes I will...

661
00:37:28.464 --> 00:37:32.848
accidentally make a choice that I'm not fully aware of the choice I'm making because I'm not reading it.

662
00:37:33.168 --> 00:37:34.589
And I find that's mostly what it is.

663
00:37:34.649 --> 00:37:43.736
But what I think the key is, if you did this with a human, you said to a team of four people, each of you take a stab and then this other person is going to judge it.

664
00:37:45.117 --> 00:37:46.478
You're probably going to trust it.

665
00:37:46.799 --> 00:37:47.279
You know what I mean?

666
00:37:47.299 --> 00:37:50.802
Like we trust these already by doing it four or five times.

667
00:37:50.962 --> 00:37:52.803
It seems like it's making better decisions.

668
00:37:52.963 --> 00:37:54.284
So should we try it and see how it is?

669
00:37:54.304 --> 00:37:56.086
All right, here we go.

670
00:37:56.613 --> 00:37:57.393
How do you control it?

671
00:37:57.773 --> 00:37:58.073
I'm not sure.

672
00:37:58.093 --> 00:37:58.694
What are they saying?

673
00:37:58.734 --> 00:37:59.114
I need it.

674
00:38:00.354 --> 00:38:01.094
Okay, I'm on the right.

675
00:38:01.114 --> 00:38:01.515
Okay, there we go.

676
00:38:01.535 --> 00:38:02.615
But I wonder who the other side is.

677
00:38:02.635 --> 00:38:03.295
The paddle on the right.

678
00:38:03.315 --> 00:38:04.335
You're going to have to be two people.

679
00:38:04.375 --> 00:38:06.956
Maybe I'm, yeah, I'm also on the, on that side.

680
00:38:06.996 --> 00:38:07.376
Here we go.

681
00:38:08.457 --> 00:38:10.337
So I guess it did what it said on the tin.

682
00:38:10.957 --> 00:38:11.938
Like, look at the mood.

683
00:38:12.298 --> 00:38:13.278
The mood is growing.

684
00:38:14.179 --> 00:38:14.379
Yeah.

685
00:38:14.399 --> 00:38:15.799
I question that this was the best choice.

686
00:38:16.339 --> 00:38:16.579
Yeah.

687
00:38:16.619 --> 00:38:18.600
So, well, the exciting thing is you could go back.

688
00:38:19.647 --> 00:38:23.589
Yeah, you could go back and you could say, because I don't think you deleted it, so you might be able to open it.

689
00:38:23.609 --> 00:38:26.571
You could be like, go back and say, I don't think that was the right choice.

690
00:38:26.651 --> 00:38:27.712
I was wrong.

691
00:38:28.172 --> 00:38:30.173
Or you were wrong because Claude picked it.

692
00:38:30.954 --> 00:38:32.394
Oh, wow, now we're talking.

693
00:38:32.795 --> 00:38:35.596
Like, this has been very effective for us.

694
00:38:35.676 --> 00:38:38.218
And like I said, what it really works well in is when...

695
00:38:38.598 --> 00:38:46.062
In situations where you're trying to get something tactical done, you know what it is, you know the shape of it, but you don't care what language it's in.

696
00:38:46.443 --> 00:38:48.224
You don't necessarily care.

697
00:38:48.264 --> 00:38:50.225
You could just say, look, I want it to be compiled.

698
00:38:50.305 --> 00:38:51.626
I want the end to be a binary.

699
00:38:52.046 --> 00:38:54.667
Let it choose Zig or let it choose Rust or Go.

700
00:38:54.687 --> 00:38:57.469
It'll probably choose it based on what's installed on your environment.

701
00:38:57.909 --> 00:38:59.570
You don't necessarily care about that.

702
00:38:59.890 --> 00:39:00.251
Whatever.

703
00:39:00.331 --> 00:39:02.032
Or you're like, here's an API.

704
00:39:02.072 --> 00:39:04.173
I just need to make a bunch of requests.

705
00:39:04.193 --> 00:39:04.473
You don't...

706
00:39:04.633 --> 00:39:29.382
care about is this using Axios or is it using Python requests like those are the type of things that I think are really important here as we and obviously this is an abstraction layer above cloud code etc as we're moving and shooting through the galaxy here we're changing software development life cycles etc it's important to think like what are the things that matter to us as humans what do we care about some people will be like I never want to use Ruby and it's like great cool

707
00:39:30.384 --> 00:39:30.865
That's great.

708
00:39:30.925 --> 00:39:31.526
Die on that hill.

709
00:39:31.586 --> 00:39:31.986
Perfect.

710
00:39:32.287 --> 00:39:32.808
I don't care.

711
00:39:33.388 --> 00:39:36.032
And does, and then, but some people are like, I don't have any preference.

712
00:39:36.052 --> 00:39:37.354
So it's just like, get it done.

713
00:39:37.815 --> 00:39:38.696
Let's get the thing done.

714
00:39:38.716 --> 00:39:40.799
And I, I think that's what we really like about this.

715
00:39:41.713 --> 00:39:46.256
It's removing a lot of the cruft from our own brains where we no longer have to keep in our brain.

716
00:39:46.296 --> 00:39:47.537
This is what I want it to be in.

717
00:39:47.818 --> 00:39:49.639
This is the languages I find important.

718
00:39:49.939 --> 00:39:51.901
Like it's one of the best examples of this.

719
00:39:51.941 --> 00:39:55.643
If you ask it to do web, I need a PWA web framework.

720
00:39:56.024 --> 00:40:01.228
It's going to pick Svelte or RAW itself or just web components or React or Vue.

721
00:40:01.688 --> 00:40:06.091
Meaning like it's just going to pick best for what it thinks is the best for this situation.

722
00:40:06.491 --> 00:40:08.193
That's the key, which is how you want it.

723
00:40:08.693 --> 00:40:12.738
You don't want it to make an emotional decision based on what it thinks you want.

724
00:40:13.199 --> 00:40:15.101
It's just going to do whatever is going to be best for that thing.

725
00:40:15.121 --> 00:40:15.842
And that's really the goal.

726
00:40:15.862 --> 00:40:19.787
And that's also why we called it test kitchen omakase, because that's typically what omakase is.

727
00:40:19.827 --> 00:40:21.750
What the best, the chef, leave it up to the chef.

728
00:40:21.830 --> 00:40:22.671
The chef knows better.

729
00:40:23.111 --> 00:40:24.193
I don't know how to make sushi.

730
00:40:25.087 --> 00:40:25.467
It's true.

731
00:40:25.988 --> 00:40:28.910
Harper, how many tools or skills like this has your company built?

732
00:40:29.430 --> 00:40:29.850
So many.

733
00:40:30.451 --> 00:40:30.811
So many.

734
00:40:31.071 --> 00:40:36.094
We have probably close to 50 that we use regularly.

735
00:40:36.154 --> 00:40:38.136
And most of them are around this thing.

736
00:40:38.216 --> 00:40:39.797
There's a couple that are pretty wild.

737
00:40:39.877 --> 00:40:41.758
One of my favorite ones is Deliberation.

738
00:40:41.798 --> 00:40:44.740
I don't really use it as much, but that's from my co-founder, Dylan, who's Quaker.

739
00:40:44.800 --> 00:40:49.063
So it's using Quaker business practices to interact with agents.

740
00:40:49.103 --> 00:40:51.725
Another one is called Review Squad, which just does a bunch of... What?

741
00:40:51.865 --> 00:40:52.005
What?

742
00:40:53.546 --> 00:40:55.327
And most of these are on your website, is that right?

743
00:40:55.368 --> 00:40:57.209
Yeah, they're all on skills.2389.

744
00:40:57.249 --> 00:41:00.331
I'll tell you the two, there's a new one we just dropped yesterday.

745
00:41:00.351 --> 00:41:05.155
The two that blow my mind, and then the new one, there's binary RE.

746
00:41:05.235 --> 00:41:08.678
If you're reverse engineering any binaries, this is an incredible skill.

747
00:41:08.818 --> 00:41:11.340
It is so good that Fable refuses to run it.

748
00:41:12.708 --> 00:41:13.709
Fable's just like, nope.

749
00:41:14.009 --> 00:41:21.214
And then the other one is called Jam, which is like a mix of the review squad and test kitchen all into one thing.

750
00:41:21.254 --> 00:41:23.075
So you can just say, hey, let's jam on this.

751
00:41:23.135 --> 00:41:25.096
And it just rips up and just builds the whole thing.

752
00:41:25.456 --> 00:41:40.346
And then the final one that we just did yesterday is called Thrifty, which does heavier model, like a Fable or an Opus planning and spec writing, and then smaller model execution, which means that it's about 60% cheaper to run on your token budget.

753
00:41:40.446 --> 00:41:42.126
And so you all also make a hoodie.

754
00:41:42.787 --> 00:41:43.827
Yeah, yeah, we do have a hoodie.

755
00:41:44.147 --> 00:41:46.327
It's a little hot out, but we have this hoodie that is great.

756
00:41:46.527 --> 00:41:53.769
We patterned it off of the old school plumber uniform that you'd have, you see these guys wear, like the union guys wear.

757
00:41:54.089 --> 00:41:58.290
But on the front, it has our very famous moniker, which is AI will kill us all.

758
00:41:58.630 --> 00:42:00.291
Yeah, we have some feelings.

759
00:42:00.711 --> 00:42:09.053
The last question I have is just, can you connect the front of that hoodie to the work that you're doing in AI and how you're leaning in, you have

760
00:42:10.163 --> 00:42:18.756
more skills than probably the top decile or in the top decile of AI developer as far as the tooling you're building around it.

761
00:42:21.527 --> 00:42:26.550
We started this company thinking, can we build agents that are more humanist?

762
00:42:26.650 --> 00:42:30.692
Because a lot of the narrative that was coming out of pro AI stuff is like down with jobs.

763
00:42:30.712 --> 00:42:31.793
We're going to remove all the jobs.

764
00:42:31.813 --> 00:42:34.014
You look at the big labs that really support every 10 minutes.

765
00:42:34.074 --> 00:42:35.435
It's like, we're going to kill all the jobs.

766
00:42:35.455 --> 00:42:36.535
Everything is dangerous, doom.

767
00:42:36.936 --> 00:42:38.657
And we were like, maybe, sure.

768
00:42:39.037 --> 00:42:40.498
But also, can we do work?

769
00:42:41.375 --> 00:42:43.896
based on our experience, to lean into humanism.

770
00:42:44.016 --> 00:42:52.200
Can we do work that is more of, not necessarily in a way that is ineffective, because I think that's another option is you can just duck your head in the sand.

771
00:42:52.580 --> 00:42:53.941
And we were like, no, we're going to do this stuff.

772
00:42:53.961 --> 00:42:56.662
We love this stuff, but how do we do it in a way that doesn't fuck us all over?

773
00:42:57.102 --> 00:42:58.483
We started doing the thing about jobs.

774
00:42:58.683 --> 00:43:02.964
We kind of give up on that because I don't know how to solve that problem because I do think you have two ideas out of here.

775
00:43:03.084 --> 00:43:05.485
One is like you can cut, cut, cut, save costs.

776
00:43:05.505 --> 00:43:06.686
The other one is you can do more.

777
00:43:07.186 --> 00:43:08.206
We're on the do more camp.

778
00:43:08.346 --> 00:43:09.187
We think about that a lot.

779
00:43:09.447 --> 00:43:12.128
Part of it is like, can you just make it more fun for humans?

780
00:43:12.148 --> 00:43:14.229
Like if we're going to work in this stuff, how do we do that?

781
00:43:14.729 --> 00:43:19.151
This is connected to the AI will kill us all in that if we lived in Norway, we'd

782
00:43:20.251 --> 00:43:25.556
There'd be no conversation about healthcare, housing, childcare, et cetera, but we don't.

783
00:43:25.877 --> 00:43:26.698
I live in the United States.

784
00:43:26.738 --> 00:43:27.839
I live in Chicago, a big city.

785
00:43:28.259 --> 00:43:29.861
And there are people who are scared to death out here.

786
00:43:30.321 --> 00:43:33.905
One of the reasons they're scared to death is because the only narrative coming out of this is that we're fucked.

787
00:43:33.985 --> 00:43:35.206
That's literally the only narrative.

788
00:43:35.246 --> 00:43:38.709
That is the comms narrative for a lot of the big labs for them to get business.

789
00:43:39.330 --> 00:43:40.431
They're saying this over and over again.

790
00:43:40.451 --> 00:43:42.252
They're like, mythos is going to kill us all.

791
00:43:42.332 --> 00:43:43.072
That's how good it is.

792
00:43:43.272 --> 00:43:44.033
Would you like to try it?

793
00:43:44.053 --> 00:43:45.473
Just sign up for $200 a month.

794
00:43:45.513 --> 00:43:46.674
I think that's the wrong narrative.

795
00:43:46.694 --> 00:43:52.737
And so we like to push on this a little bit by first admitting the reality, which is that this is complex if you live in the United States.

796
00:43:52.757 --> 00:43:54.218
This is a complex social infrastructure.

797
00:43:54.618 --> 00:43:55.359
We need to figure this out.

798
00:43:55.399 --> 00:43:57.020
We can't figure it out by ignoring it.

799
00:43:57.340 --> 00:44:01.402
But we'd also like to have this other aspect, which is like, yeah, you can have fun though.

800
00:44:02.533 --> 00:44:02.813
Yes.

801
00:44:03.013 --> 00:44:03.434
Okay.

802
00:44:03.514 --> 00:44:17.866
So where you've landed in this kind of tension, this existential debate is you're leaning in and at the same time, you want to push for social change to build almost a society and a way of organizing around this that's compatible with this future that you're envisioning.

803
00:44:18.663 --> 00:44:29.839
We really think it's about social change, but I think the problem right now is that we have to have the social change be on the social, like the dynamics of the U.S. change in such a way that it isn't just destructive.

804
00:44:30.460 --> 00:44:33.605
Like an example would be if we got employment law that matched employment.

805
00:44:33.665 --> 00:44:36.347
Germany's because we are afraid of displacing workers.

806
00:44:36.767 --> 00:44:37.808
Like that wouldn't be good.

807
00:44:38.128 --> 00:44:40.830
Instead, we need to, I think this is part of the CEOs and startup thing.

808
00:44:40.850 --> 00:44:43.651
Like we need to have our co-CEOs, our friends, et cetera.

809
00:44:43.952 --> 00:44:50.456
Think about how do you make your 3000 employees equal to 6000 employees, not how do you make them 1000 employees?

810
00:44:50.516 --> 00:44:53.358
And right now, the only tool we seem to have is to make them 1000 employees.

811
00:44:53.378 --> 00:44:54.819
And I just think that's, I don't know.

812
00:44:54.939 --> 00:44:55.879
I like working with teams.

813
00:44:55.919 --> 00:44:56.500
I love people.

814
00:44:56.540 --> 00:44:57.400
It's fun to hang out.

815
00:44:57.781 --> 00:44:59.622
Like I want to do more of that, not less of that.

816
00:44:59.682 --> 00:45:02.424
So I think this is very related to our entire thing, but it's a lot.

817
00:45:02.444 --> 00:45:02.584
Right.

818
00:45:03.084 --> 00:45:04.285
This is AI worth using.

819
00:45:04.465 --> 00:45:05.566
Harper Reid, thank you.

820
00:45:05.646 --> 00:45:10.129
You've been here long enough to know what it looks like when something gets away from you.

821
00:45:10.349 --> 00:45:13.452
And I appreciate your willingness to reflect on all of that.

822
00:45:13.752 --> 00:45:14.692
Yeah, I love this.

823
00:45:14.713 --> 00:45:15.953
I'll give him a couple of takeaways.

824
00:45:16.053 --> 00:45:17.795
First, I love that you want to make things fun.

825
00:45:18.215 --> 00:45:23.179
I think that's always been a part of when I've built companies, when I assemble teams, like you have to make things fun.

826
00:45:23.199 --> 00:45:25.040
Like it has to be a positive thing.

827
00:45:25.180 --> 00:45:26.481
energetic thing to do.

828
00:45:26.741 --> 00:45:34.426
And that's, I think that's valuable for even for the LLMs, as you've shown, it's like, I don't know, give them some of that, you know, that vibe and see if that works.

829
00:45:34.466 --> 00:45:36.167
So I definitely am going to experiment with that.

830
00:45:36.787 --> 00:45:45.693
Number two really is, and this comes up over and over is, as the cost of generating code goes to zero, what new things can we do?

831
00:45:46.565 --> 00:45:51.269
And I think that frees up so much thinking about like, what new things can we do because of that?

832
00:45:51.469 --> 00:45:52.630
When code is not precious.

833
00:45:53.430 --> 00:45:56.713
And in this case for Test Kitchen, that really is this idea.

834
00:45:57.254 --> 00:45:58.975
Makes, it reminds me of the Steve Jobs story.

835
00:45:59.055 --> 00:46:04.019
It might be apocryphal, but there is this story where he would go in and when they're designing something new, like...

836
00:46:04.557 --> 00:46:09.181
a calculator interface, he'd make the design team do 10 different iterations of it.

837
00:46:09.701 --> 00:46:11.203
10 fully different designs.

838
00:46:11.883 --> 00:46:15.066
Because it's like the most obvious one you do right away, but it's boring.

839
00:46:15.206 --> 00:46:16.087
Everybody's done it before.

840
00:46:16.167 --> 00:46:17.608
And then you're like, okay, we need an alternative.

841
00:46:18.009 --> 00:46:22.072
And then you do like the other boring alternative that everybody has done already.

842
00:46:22.132 --> 00:46:23.013
And then you get to the third one.

843
00:46:23.033 --> 00:46:25.055
You're like, all right, like, but what are we, how are we going to change it?

844
00:46:25.095 --> 00:46:26.136
How are we going to make it different?

845
00:46:26.156 --> 00:46:30.920
You start coming up with new ideas and then you get to like four, five, six, like once you get to 10, 10.

846
00:46:31.420 --> 00:46:42.750
You're like so out there doing stuff that nobody's ever thought of before that you might stumble upon that one thing that you then bring back into the more reasonable, boring version that really makes it special.

847
00:46:43.550 --> 00:46:49.796
And this to me is a way of doing that, being like, I don't know, like show me some crazy stuff and let me think about it.

848
00:46:49.916 --> 00:46:53.379
Or now you think about it and like come back to me with the stuff that's actually special.

849
00:46:54.139 --> 00:46:56.461
And I love that, really exploring the design territory.

850
00:46:56.641 --> 00:46:57.422
Those are my takeaways.

851
00:46:57.482 --> 00:46:58.243
Yeah, it's very cool.

852
00:46:59.218 --> 00:47:00.839
I think on this episode, I had fun.

853
00:47:00.859 --> 00:47:03.160
I felt a little uncomfortable at times.

854
00:47:03.781 --> 00:47:07.402
And at the same time, I think I'll also be a little bit more productive.

855
00:47:07.583 --> 00:47:08.843
So thank you, Harper.

856
00:47:09.103 --> 00:47:12.445
We will have a link to your blog with a link to your company's website.

857
00:47:13.065 --> 00:47:14.866
And Tom, I'll see you next week.

858
00:47:15.607 --> 00:47:15.827
All right.

859
00:47:16.067 --> 00:47:16.467
Thanks, Harper.

860
00:47:16.487 --> 00:47:16.727
Thank you.

861
00:47:17.128 --> 00:47:17.588
Thank you, Sam.