WEBVTT

1
00:00:00.151 --> 00:00:01.032
Hi, I'm Mike Taylor.

2
00:00:01.132 --> 00:00:11.079
I'm the head of tech consulting at Every, and I sat down with Kyle Daigle, the COO of GitHub, and talked to him about what is happening on the front lines of coding agents.

3
00:00:11.440 --> 00:00:15.543
We have 17 million pull requests coming in every month to GitHub now,

4
00:00:15.943 --> 00:00:20.804
It's growing exponentially, and that puts them at the forefront of what's happening in this new economy.

5
00:00:21.044 --> 00:00:29.546
We talked about how that affects their users, as well as how this affects open source maintainers, and we covered a topic which is dear to everyone's hearts.

6
00:00:29.726 --> 00:00:35.888
How do I stop my $200 a month coding agent subscription from ballooning into a $2,000 a month usage limit?

7
00:00:36.048 --> 00:00:44.310
In this interview, we did something a little bit different, which is I told Kyle I had made an AI clone of him to practice the interview, and he revealed something surprising in return.

8
00:00:44.530 --> 00:00:45.530
Here's the conversation.

9
00:00:45.830 --> 00:00:45.863
Thank you.

10
00:01:00.138 --> 00:01:02.639
Every is the only subscription you need to stay at the edge of AI.

11
00:01:02.859 --> 00:01:09.420
If you care about being on top of the latest models and using the latest tools, you have to subscribe to every to separate out the signal from the noise.

12
00:01:09.720 --> 00:01:11.621
Go to every dot to slash subscribe today.

13
00:01:12.121 --> 00:01:14.601
Hey, Kyle, thanks for spending some time with me at the conference.

14
00:01:14.681 --> 00:01:17.182
So, of course, yeah, it was good to meet you as well.

15
00:01:17.222 --> 00:01:26.004
And the day before, and I feel like we already kind of covered a few of these questions, but I think it'd be good to, you know, for the wider audience so they can understand what's going on here.

16
00:01:26.144 --> 00:01:26.564
Absolutely.

17
00:01:26.685 --> 00:01:27.265
Yeah.

18
00:01:27.285 --> 00:01:36.013
So the first thing I think is we were talking about is really interesting is that the demographics of the customer are changing, right?

19
00:01:36.033 --> 00:01:43.060
Like a lot of people who previously maybe never used GitHub or never used developer products before are now using them.

20
00:01:43.120 --> 00:01:46.263
So how has that changed the way that you decide the product roadmap?

21
00:01:46.383 --> 00:01:54.030
Yeah, yeah, I mean, you know, I think for GitHub in particular, we've always really had this really expansive view of what a developer is.

22
00:01:54.051 --> 00:02:02.459
I started as a developer before I would have ever called myself a dev, you know, where I was just like writing code, but it was just for me.

23
00:02:02.539 --> 00:02:04.901
And I went personally like a completely different career path.

24
00:02:04.981 --> 00:02:06.623
I didn't go to school for computer science.

25
00:02:07.083 --> 00:02:13.067
I was going to art school, I wrote code to pay for art school, which was a very silly decision as an adult now, I guess.

26
00:02:13.127 --> 00:02:28.076
But then, you know, that sort of journey of just like, I can create tools with the team and deliver them to people who can have that same experience of like, I just want to build an app that's for me or for my family, maybe as a startup, maybe as a business.

27
00:02:29.867 --> 00:02:33.588
We very much have serious developer tools.

28
00:02:33.668 --> 00:02:37.909
You know, like all the largest businesses are using GitHub.

29
00:02:38.069 --> 00:02:50.592
But when I look at something like the GitHub Copilot app, I see just as many developers that are using AI every day, running multiple projects, all kinds of agent sessions at the same time.

30
00:02:51.172 --> 00:02:54.633
And I see our legal team at GitHub using

31
00:02:55.013 --> 00:03:00.035
the GitHub Copilot app or the finance team, or I was meeting with a customer today and they were saying the same thing.

32
00:03:00.476 --> 00:03:09.460
A lot of the folks that the industry would call knowledge workers or just non by trade developers are using these tools to build little apps or assets for them.

33
00:03:09.480 --> 00:03:18.824
And so while our focus is very much on developers, I think we want to make it easier for people to choose to try to write some code.

34
00:03:19.324 --> 00:03:25.767
and make sure there's always an on-ramp into writing some software now with things like the GitHub Copilot app.

35
00:03:26.207 --> 00:03:30.869
Yeah, and then how do you help developers deal with the burden of all of that extra?

36
00:03:31.309 --> 00:03:35.091
There's a flood of PRs now, like open source maintenance I talked to are drowning.

37
00:03:35.511 --> 00:03:36.651
What needs to happen to help them?

38
00:03:38.412 --> 00:03:44.154
Yeah, I mean, I think for all developers, we're, you know, building tools like the Copilot Code Review.

39
00:03:44.814 --> 00:03:54.316
It's now agentic, so it finds a lot more novel vulnerabilities and you can just like comment and the, you know, the agent will take that on and go implement the change if you want to.

40
00:03:54.637 --> 00:04:03.159
So I think that Code Review step is in some ways like overlooked as a really great way to get PRs to a place that are much more easily reviewed.

41
00:04:04.360 --> 00:04:21.636
I think that the agentic merge in the app is another place where we see a lot of times internally and in the community, you may comment on something that might have a code review, and you might go through and get it almost all the way there, but then there's all those manual steps just to finish processing the PR.

42
00:04:22.056 --> 00:04:31.282
Instead, I can go in and set exactly what I want to allow GitHub Copilot to do and say, okay, now go merge this PR and wait for CI and wait for policies and all of that.

43
00:04:31.302 --> 00:04:32.222
I think that's a big part.

44
00:04:32.483 --> 00:04:40.888
On the open source side, it's a unique set of needs because you don't control who's sending everything in, or you haven't really historically.

45
00:04:41.248 --> 00:04:45.691
That's been really where we've been focusing is giving maintainers more tools to decide, well,

46
00:04:46.391 --> 00:04:48.412
do you want to accept all of these PRs?

47
00:04:48.492 --> 00:04:50.293
Like, who do you want to accept them from?

48
00:04:51.093 --> 00:04:58.857
You know, how much work do you need to do to kind of prove that you're going to contribute something that is going to be meaningful to this project?

49
00:04:59.477 --> 00:05:05.380
And that's something that we want to provide tools to open source maintainers, but really leave them in control.

50
00:05:05.900 --> 00:05:09.902
Every community is choosing a slightly different way to approach the problem, and

51
00:05:10.662 --> 00:05:14.666
For GitHub, we've always wanted to leave that in their hands.

52
00:05:14.706 --> 00:05:22.534
Give them tools and enable them, but if a standard comes out of that, or most are using a certain practice, we'll lock that in.

53
00:05:22.775 --> 00:05:27.980
But we don't really ever want to be the first to create a standard or an approach.

54
00:05:28.020 --> 00:05:31.524
I think Mitchell Hashimoto shared the vouch system that they use,

55
00:05:32.271 --> 00:05:35.054
I was getting questions like, well, why aren't you roll this out to everybody?

56
00:05:35.114 --> 00:05:41.019
But there's just as many communities that don't want to use that system because they have their own ideas of how it should work.

57
00:05:41.559 --> 00:05:46.484
And so for now, we're focusing on the building blocks of controls for maintainers.

58
00:05:46.844 --> 00:05:55.492
And then as we all are kind of learning together and as maintainers send feedback in, we'll, you know, we'll cement an entire system if one emerges.

59
00:05:55.832 --> 00:06:10.195
Yeah, and I feel like you have a front row seat to this new agent economy where, you know, I think you said publicly on Twitter that you've had more pull requests submitted per month than you did all last year.

60
00:06:11.215 --> 00:06:13.036
How are those stats exploding?

61
00:06:13.076 --> 00:06:17.977
Yeah, I mean, we're seeing way more activity on GitHub.

62
00:06:18.897 --> 00:06:24.478
You know, we've always been talking about our users, you know, for many, many years and that growth.

63
00:06:25.578 --> 00:06:31.643
This year we're seeing obviously the growth of developers having agents building with them.

64
00:06:32.003 --> 00:06:37.548
And so last year in October at GitHub Universe, we shared there's a billion commits on GitHub for the full year.

65
00:06:38.208 --> 00:06:45.034
We're on track to be 14 billion if the growth is linear this year, which it will not be.

66
00:06:45.814 --> 00:06:49.978
In March, there were 17 million pull requests that were created by agents.

67
00:06:50.278 --> 00:06:52.440
That's just the agent pull requests.

68
00:06:52.800 --> 00:06:56.261
And so there's so much more code being created.

69
00:06:56.321 --> 00:07:01.043
And I think at times everyone goes like, oh, this is all just like slop.

70
00:07:01.103 --> 00:07:03.024
This is all just code that's getting pushed up.

71
00:07:03.384 --> 00:07:04.225
And no one cares.

72
00:07:04.665 --> 00:07:06.405
It's not really true.

73
00:07:06.846 --> 00:07:12.148
We're all just actually getting to the point where we're no longer in the super early adoption.

74
00:07:12.188 --> 00:07:17.490
We're definitely not at the peak, but we're climbing that hill to see what can we build when

75
00:07:18.010 --> 00:07:30.018
It's not just Kyle building, but it's Kyle and one, two to N agents that are using my skills, using my resources, using my context and so on and so forth.

76
00:07:30.258 --> 00:07:38.944
And so like we're investing heavily in preparing for the next wave of growth because it doesn't seem to be kind of like growing and plateauing.

77
00:07:38.984 --> 00:07:45.329
It's just going to continue to grow because no matter where you're building or what tools you're using to build,

78
00:07:46.524 --> 00:07:53.072
all of that code ends up on GitHub, you know, or that's where you're sharing it with the world, or that's where you're collaborating in a PR.

79
00:07:53.873 --> 00:08:00.861
And so we need to be able to support everyone's, you know, sort of agent moment and not just, you know, GitHub Copilot.

80
00:08:00.941 --> 00:08:01.322
Yeah, yeah.

81
00:08:01.882 --> 00:08:04.003
And how does the business model change?

82
00:08:04.023 --> 00:08:12.128
Because I think freemium makes sense, you know, in a human centered world where we go to bed, but the agents are still working while we're asleep now.

83
00:08:12.248 --> 00:08:14.649
So does that change like to usage based?

84
00:08:15.009 --> 00:08:17.251
Like is that you see that kind of where things are going?

85
00:08:18.011 --> 00:08:22.316
Yeah, I mean, I don't think we know yet, ultimately.

86
00:08:22.336 --> 00:08:37.013
Like, I think we very much right now, like, Kyle's going to have a license, or Kyle's using GitHub.com for free, and we've always had API rate limits, you know, and things like that, and that's usually where folks are seeing the agent back pressure, I think.

87
00:08:38.034 --> 00:08:50.403
I think the goal is that if you want to be able to do way more, if you want to be able to, like Peter Steinberger says, you know, 150 agents are doing everything all at once, you know, that's great.

88
00:08:50.443 --> 00:08:52.264
We want to be able to enable that to you.

89
00:08:52.364 --> 00:08:57.048
But at the same time, I want you to have a great core GitHub experience.

90
00:08:57.168 --> 00:08:57.468
you know?

91
00:08:57.869 --> 00:09:04.394
And at the very least, there's some amount of agent usage as part of that that is necessary, you know?

92
00:09:04.714 --> 00:09:11.880
I need, similar to how we, way, way, way back, right, you'd have free public repos, but you didn't have free private repos, you know?

93
00:09:12.140 --> 00:09:18.465
And then we said, okay, well, actually, like, it's fair for an individual to have some code that they don't want to put out into the world and,

94
00:09:18.965 --> 00:09:21.986
we'll give you free private repos to allow you guys to do that.

95
00:09:22.006 --> 00:09:32.708
So GitHub's always evolving as the industry and community does, but we're always sort of focused on, I need to make sure you, the dev, have what you need to be successful.

96
00:09:34.209 --> 00:09:43.171
And then work with enterprises to make sure they have what they need to do at scale, which is usually a little bit different than what an individual dev's doing.

97
00:09:43.271 --> 00:09:50.075
Yeah, and I guess the business model pricing kind of that all leads back into like the wider Microsoft orbit.

98
00:09:50.635 --> 00:09:56.778
And because you have a dual role now, right, partial responsibility for the wider kind of marketing org.

99
00:09:56.818 --> 00:10:00.000
So do you want to talk me through how that's changed and how you prioritize between those two?

100
00:10:00.220 --> 00:10:09.764
Yeah, I mean, so I've been at GitHub for a very long time, 13 years, and as a developer myself and leading engineering teams for a lot of that time.

101
00:10:10.444 --> 00:10:15.727
And I think what's always been unique about GitHub is we really, really focus on the dev.

102
00:10:15.867 --> 00:10:21.109
Like, we're building tools for the developers, and the fact that people like enterprises are

103
00:10:21.749 --> 00:10:29.956
Buying them is awesome and that's definitely great, but we're not building for the buyers, we're building for the developers in 100%.

104
00:10:30.056 --> 00:10:48.411
And so in this, and that's been my focus as the COO of GitHub, which I continue to do, and then now as the Chief Marketing Officer of Developer for Microsoft, my goal is to look across all of Microsoft's tooling, their developer tools, their technology that they're bringing to developers,

105
00:10:48.811 --> 00:10:57.558
and making sure that we're bringing holistic solutions that you can use that are authentic to developer experiences.

106
00:10:58.179 --> 00:11:05.524
And at events like this, where we've taken a very different approach to build this year, we're in San Francisco, first off.

107
00:11:06.625 --> 00:11:14.208
The vibe is a bit different than you know the conference hall set up, you know really focused on Can I go to a session?

108
00:11:14.228 --> 00:11:15.288
Can I use the thing?

109
00:11:15.328 --> 00:11:31.193
I don't want to be pitched on a thing I have to be able to use it expo hall kind of so on and so forth It's really bringing that expertise and you know love and focus on the developer that github's always had to have even broader impact throughout all of Microsoft

110
00:11:31.533 --> 00:11:36.155
Yeah, and did I hear you say that this is the first build that you've had external contributors?

111
00:11:36.315 --> 00:11:46.239
It's the first build that I think by intention we focused on having speakers from the community like in these primary sessions.

112
00:11:46.980 --> 00:11:56.644
That includes in the keynote we had a bunch of folks like Peter, there's sessions from SWIX and others as well.

113
00:11:57.004 --> 00:11:58.725
I think that it's important,

114
00:11:59.745 --> 00:12:02.306
Software development is a team sport.

115
00:12:02.826 --> 00:12:12.268
It seems silly to think that there's any one company, one group, inclusive of GitHub and Microsoft and everyone that can just answer every single question.

116
00:12:12.288 --> 00:12:14.309
That's not how software gets made.

117
00:12:14.349 --> 00:12:19.330
We're all at least using open source, and we're building on the backs of these giant open source projects.

118
00:12:19.691 --> 00:12:24.452
Let's invite people in that can help tell their part of the story together.

119
00:12:24.472 --> 00:12:28.593
Because I deeply believe that that's what developers want.

120
00:12:28.733 --> 00:12:30.014
I know that's what I want.

121
00:12:30.094 --> 00:12:32.236
I know that's what my friends that are developers want.

122
00:12:32.296 --> 00:12:39.541
And when we look at the events and we hear the feedback, they're excited to see people from Microsoft, from GitHub.

123
00:12:39.601 --> 00:12:43.284
And then, oh, I get to see this outside perspective at this event.

124
00:12:43.704 --> 00:12:44.785
It's really meaningful.

125
00:12:45.065 --> 00:12:45.745
Yeah, that makes sense.

126
00:12:45.906 --> 00:12:48.267
And it's a very competitive market, right?

127
00:12:48.327 --> 00:12:49.268
Sure, yeah.

128
00:12:49.308 --> 00:12:52.050
Like, you know, the most competitive market probably.

129
00:12:52.671 --> 00:12:53.251
Maybe the Alaska.

130
00:12:53.811 --> 00:13:00.176
I'm not sure, but how do you differentiate in all of that, given the pace of change is so quick?

131
00:13:00.416 --> 00:13:09.583
Yeah, I mean, I think we continue to focus on our roots, which is, you know, we care a lot about developer choice.

132
00:13:10.043 --> 00:13:11.164
It's always been true.

133
00:13:11.644 --> 00:13:14.446
We care about building for builders and enabling builders.

134
00:13:14.906 --> 00:13:15.287
And so...

135
00:13:15.987 --> 00:13:30.601
I think we're in a moment that's really interesting because we've went from an era of having a ton of APIs, all this access, to a little bit of an unintentional walled garden setup, where you get a kind of affinity.

136
00:13:30.661 --> 00:13:33.264
Sometimes I'll say it's like a little bit of a mousetrap.

137
00:13:33.644 --> 00:13:37.448
And then you realize, oh, this thing's really interesting over here, and then I have to...

138
00:13:38.048 --> 00:13:41.050
oh, I have to go learn a new thing or a new tool or a new account.

139
00:13:41.070 --> 00:13:51.837
And I think for us, we always want to enable developers that are building with GitHub to go use these other tools, and we'll partner with everyone to make that as simple as is possible.

140
00:13:52.177 --> 00:14:06.527
And while I think that there's other folks that are doing similar things, I think the ability to do that across the entirety of building software, and not just the Cogen side or not just the collaboration review side...

141
00:14:06.807 --> 00:14:12.430
but across everything is a real superpower of ours.

142
00:14:12.750 --> 00:14:15.911
And so I think you'll see us invest in our own tech.

143
00:14:15.971 --> 00:14:21.114
Like we talked about the new Microsoft AI models that we'll continue to bring to developers.

144
00:14:21.514 --> 00:14:29.998
We're also continuing to partner with Anthropic and OpenAI and Google and kind of anyone who's bringing a model to market or a coding agent to market.

145
00:14:30.098 --> 00:14:31.979
We'll partner with you and we'll both

146
00:14:32.019 --> 00:14:36.624
let you bring that to us, or we'll omit it through GitHub and GitHub Copilot.

147
00:14:37.184 --> 00:14:38.505
That choice is core.

148
00:14:38.666 --> 00:14:43.450
And that's something that I only will ever back down on.

149
00:14:43.691 --> 00:14:46.774
Because if we do, developers will still choose.

150
00:14:47.194 --> 00:14:51.518
They'll just be stuck in another kind of mousetrap.

151
00:14:52.419 --> 00:14:54.902
And we don't want the world of software to be like that.

152
00:14:55.342 --> 00:14:57.002
Yeah, yeah.

153
00:14:57.042 --> 00:15:04.684
And when you're making the decisions internally, like there was a news cycle recently about how code code cancels, licenses are being canceled.

154
00:15:04.704 --> 00:15:18.368
And like, how do you make the trade off between dogfooding your own products, like using the new models you made or using the GitHub Copilot app, desktop app versus like, you know, using like letting developers kind of experiment with other tools?

155
00:15:18.728 --> 00:15:30.236
Yeah, I mean, we all are using a variety of tools because otherwise you lose track, you know, or you kind of, you're too interested in your own work.

156
00:15:31.016 --> 00:15:37.081
So like for me, you know, I've been a daily driver of a MacBook for many, many years.

157
00:15:37.121 --> 00:15:39.482
I use Windows PCs on the weekends when I play video games.

158
00:15:39.822 --> 00:15:46.187
And I got this role and I have my Mac, I have my PC, and I have my OMARQ Linux box so I can make sure that

159
00:15:46.767 --> 00:15:48.909
Every weekend, I code most Saturdays.

160
00:15:48.969 --> 00:15:50.950
I do my kids' sports activities in the morning.

161
00:15:51.310 --> 00:15:57.634
And then in the afternoon, I'm coding, and I'm swapping between the boxes because I want to understand, OK, what's that experience?

162
00:15:57.654 --> 00:16:01.777
The GitHub Copilot app I only use on Windows because I want to make sure that.

163
00:16:02.705 --> 00:16:05.627
Developers who are on Windows also deserve great apps.

164
00:16:05.707 --> 00:16:08.969
It's not just the audience that's on a Mac, for example.

165
00:16:09.289 --> 00:16:14.272
And that's true across our teams, especially like when we're looking at, OK, what about the coding agents?

166
00:16:14.292 --> 00:16:15.112
What about the harnesses?

167
00:16:15.132 --> 00:16:16.153
What about the desktop apps?

168
00:16:16.193 --> 00:16:17.754
What about memory management?

169
00:16:17.774 --> 00:16:19.034
What about everything?

170
00:16:19.374 --> 00:16:23.097
We have this really great culture of just experimentation.

171
00:16:23.237 --> 00:16:25.438
Everyone is building and using these tools.

172
00:16:25.838 --> 00:16:40.486
while obviously we're putting most of our energy into our own tools, it's such a blind spot that I think it's happened to GitHub in the past where when you're doing something and you're doing it well, you really laser focus.

173
00:16:40.526 --> 00:16:43.408
And that's what every piece of startup energy says, right?

174
00:16:43.428 --> 00:16:46.029
It's like, look down and just keep moving, keep moving fast.

175
00:16:46.670 --> 00:16:48.951
And I think that's myopic.

176
00:16:49.711 --> 00:16:58.614
You know, I think that while I can't spend every day using every tool, when something comes out, I want to know why this is really great.

177
00:16:58.874 --> 00:17:01.535
Why are people having a great experience with this?

178
00:17:02.495 --> 00:17:10.958
Not only so I can understand, but so I can figure out, okay, well, for our goals, for our goal of developer choice, I don't need this.

179
00:17:11.078 --> 00:17:15.180
Like, I want to focus over here, but I want to know why a dev would pick these tools.

180
00:17:15.740 --> 00:17:18.661
And the same thing goes, you know, the same thing goes for our teams.

181
00:17:18.941 --> 00:17:19.101
Yeah.

182
00:17:19.501 --> 00:17:20.903
And how do you filter?

183
00:17:20.963 --> 00:17:23.606
Because obviously a lot of these ideas are relatively short-lived.

184
00:17:24.187 --> 00:17:26.669
Enterprise product development cycles are longer lived.

185
00:17:27.130 --> 00:17:27.691
How do you decide?

186
00:17:28.011 --> 00:17:32.997
Yeah, I think that like right now we're in a moment where we're really looking at the like...

187
00:17:34.661 --> 00:17:39.547
the short term in capturing the ability to have a multitude of agent sessions.

188
00:17:39.968 --> 00:17:42.692
This idea of just, you know, because that seems quite clear.

189
00:17:42.812 --> 00:17:45.696
You know, everyone's doing it, how can we cement it?

190
00:17:46.096 --> 00:17:50.602
But it seems clear on the longer term path, models are gonna continue to get better.

191
00:17:51.967 --> 00:18:08.568
The prices of tokens like token economics is going to be a bigger bigger factor and like what models everyone is using And I do strongly believe that we're not very far off from having serious the serious ability use You know something above a small language model on a local device, you know To be able to do some of our work.

192
00:18:08.628 --> 00:18:09.729
And so if I assume that

193
00:18:10.630 --> 00:18:14.191
I have all this optionality when it comes to tokens, you know, effectively.

194
00:18:15.711 --> 00:18:30.075
The thing that I think seems to be true from the beginning to open claw to now is this idea of like personalization or mine or context or fine tuning with context or like, you know, memory.

195
00:18:30.235 --> 00:18:36.616
All of these ideas seem to be a truth that's been there since, you know, ChatGPT came out or GitHub Copilot came out.

196
00:18:37.236 --> 00:18:40.497
And there's experiments but not a long,

197
00:18:40.517 --> 00:18:44.378
long-term vision, I think, for this across the industry.

198
00:18:44.738 --> 00:18:55.322
So I think it's a good example of where I need to get you to use agents incredibly well, a lot of them, because if you're into using agents, you're not just gonna be staring at a single agent working.

199
00:18:55.882 --> 00:18:59.723
But that's not gonna give you a long-term great experience.

200
00:19:00.283 --> 00:19:11.440
using an agent that you feel like is completing a thought for you will give you that great experience, especially if you did not have to personally codify that thought to your agent.

201
00:19:11.820 --> 00:19:14.244
Always remember that I, insert thing.

202
00:19:14.264 --> 00:19:15.085
That's a lot of work, yeah.

203
00:19:15.125 --> 00:19:15.845
A hundred percent.

204
00:19:15.885 --> 00:19:19.826
Like it should be able to intuit that.

205
00:19:19.926 --> 00:19:30.548
Or potentially, again, like post-trainer fine-tune or frontier-tune, like a model that deeply understands me and how I'm using the work.

206
00:19:30.848 --> 00:19:32.848
That is kind of how we're looking at it.

207
00:19:32.908 --> 00:19:43.210
It's like sometimes it's short-term, and sometimes we've got to take a bunch of bites of the apple or a bunch of attempts at the long-term to get to something really tangible to help us move forward.

208
00:19:43.690 --> 00:19:43.890
Cool.

209
00:19:44.010 --> 00:19:47.832
And I heard the term hill climbing like 100 times yesterday.

210
00:19:49.013 --> 00:19:54.796
And I'm a big proponent of that because, you know, experimented with DSPi, auto research, a few others.

211
00:19:55.297 --> 00:19:57.978
Can you talk a little bit about how that's become a big focus?

212
00:19:58.178 --> 00:20:05.302
Yeah, I mean, you know, I think, you know, Satya and Mustafa will talk about it a fair bit, and Jacob, too, leading the co-pilot group.

213
00:20:06.883 --> 00:20:20.396
The biggest thing that we've kind of learned is we need to use the use of the tools as a core way to improve the underlying use of the models, our own models, etc.

214
00:20:20.896 --> 00:20:25.220
In just the evals that are necessary to ensure that we're actually improving technology.

215
00:20:25.500 --> 00:20:32.963
from things like using the thumbs up, thumbs down data that comes in to using whether you're accepting it and how much you're accepting.

216
00:20:34.004 --> 00:20:42.488
All of that data is enormous to create that magical type of experience that's not just for you but for everyone.

217
00:20:42.848 --> 00:20:45.209
And so every week we're talking about

218
00:20:45.945 --> 00:20:59.709
the hill climbing results, you know, we're looking at the data, we're looking at the improvement, we're looking at both the hard measures and the soft measures, because sometimes the hard measures in evals and rubrics will show that we've made an improvement, but like user sentiment will crash.

219
00:20:59.909 --> 00:21:00.449
Yeah.

220
00:21:00.529 --> 00:21:02.810
You know, even with the same latency and performance.

221
00:21:02.850 --> 00:21:03.890
Yeah, it's overfitting basically.

222
00:21:03.910 --> 00:21:04.631
A hundred percent.

223
00:21:04.731 --> 00:21:07.771
And so like being able to really do that loop

224
00:21:09.932 --> 00:21:11.053
incredibly quickly.

225
00:21:11.474 --> 00:21:34.474
And then I think the main goal is giving everyone one of these hill climbing machines and not have you have to do it the kind of hard way that we've all been doing it, but particularly if you're in an enterprise and you are using M365, we know so much about that data or we could know so much about that data because of all the assets, all the documents, the chats.

226
00:21:34.834 --> 00:21:36.676
And so being able to, you know,

227
00:21:37.176 --> 00:21:44.605
turn on something like frontier tuning and using an MAI Thinking One as the base model.

228
00:21:45.086 --> 00:21:48.890
It shows real results without having to do all that extra work.

229
00:21:48.930 --> 00:21:52.094
And it's been interesting because when I first heard about this, I'll be honest, I was like...

230
00:21:52.955 --> 00:21:56.218
This is like a magic parlor trick, you know, that is not going to be real.

231
00:21:56.238 --> 00:21:57.179
It can't really work, yeah.

232
00:21:57.199 --> 00:22:06.106
You know, and I think the reality is that sometimes where the alpha is, is like where it feels like this is too simple to work, you know.

233
00:22:07.067 --> 00:22:10.009
You know, we all have all this data, and what are we going to do with it?

234
00:22:10.069 --> 00:22:12.311
We have to do all this effort to make it work.

235
00:22:12.632 --> 00:22:17.395
But I think so much has come down the pipe to allow us to just...

236
00:22:18.296 --> 00:22:19.296
Use the data and improve.

237
00:22:19.456 --> 00:22:20.477
Look at the workflow and improve.

238
00:22:20.497 --> 00:22:22.337
And just keep doing the hill climbing.

239
00:22:22.358 --> 00:22:27.299
That's why I think we say it so much, is that it's not these moonshots or like, oh, hill climb.

240
00:22:27.359 --> 00:22:36.783
It is just climb, climb, improve, new eval, improve, new data, improve, and just keep going to get to the point where, you know, we're able to launch these models, seven models for ourselves.

241
00:22:37.123 --> 00:22:40.904
And then, you know, allow customers to use the same or similar, you know, tooling to do it.

242
00:22:41.204 --> 00:22:46.927
Is that the answer to stopping the $200 subscription becoming a $2,000 subscription?

243
00:22:47.368 --> 00:23:01.636
I mean, like, I think that the, you know, $200 subscription to $2,000 is really going to be not only, you know, making these models or, you know, frontier tuning these models so they know you better.

244
00:23:01.976 --> 00:23:07.259
But I also think it's really, really going to be about how can we, particularly for developers, you know,

245
00:23:07.959 --> 00:23:14.948
help you automatically choose the models and potentially either have a model in that step.

246
00:23:15.088 --> 00:23:16.590
Like the model router and GitHub.

247
00:23:16.610 --> 00:23:17.210
100%.

248
00:23:17.331 --> 00:23:17.771
Exactly.

249
00:23:17.811 --> 00:23:21.335
Like the auto model router with task intent in GitHub.

250
00:23:22.357 --> 00:23:27.282
Microsoft Foundry has a model router as well that can do this at an API level.

251
00:23:28.003 --> 00:23:43.826
the more and more that we can help you tell us a bit of where your bars are, like this is an incredibly hard problem and I'm willing to go all the way to the top, or I just kinda wanna sit here and let us help choose the models.

252
00:23:43.866 --> 00:23:55.029
Because there's a lot of times where a lot of the reasons why tokens are expensive is because we're all going and choosing our model of the day or week or hour, and those models are incredibly expensive.

253
00:23:56.169 --> 00:24:01.031
my train of thought is slipping in and out of a hard problem to a simple problem.

254
00:24:01.091 --> 00:24:09.155
Personally, I feel like I'll get an agent to do an enormous amount of work, and then there's always that last step that is a smallish thing.

255
00:24:09.596 --> 00:24:12.997
Like, oh, I don't actually change all the naming of this to this.

256
00:24:14.418 --> 00:24:16.739
like a find and replace, you know what I mean?

257
00:24:17.099 --> 00:24:24.822
But am I gonna actually go and like, oh, I wanna save tokens right now, so I'm gonna go off a 4.8 or 5.5 and down to Haiku or something, you know?

258
00:24:25.063 --> 00:24:25.743
Probably not.

259
00:24:26.983 --> 00:24:28.144
But the tools could.

260
00:24:28.804 --> 00:24:41.409
And I think that will really help us, particularly in the enterprise, but even for individual developers and folks that are building automations and using their co-pilot SDK to power that.

261
00:24:41.489 --> 00:24:42.390
It'll help them too.

262
00:24:42.962 --> 00:24:44.142
I did something a little bit weird.

263
00:24:44.242 --> 00:24:48.604
I hope you don't find it creepy, but I made an AI version of you to practice this interview.

264
00:24:48.664 --> 00:24:49.104
No way.

265
00:24:49.344 --> 00:24:52.965
Yeah, and it's actually been pretty spot on so far, and hopefully you think the questions have been good.

266
00:24:53.045 --> 00:24:54.505
They've been great.

267
00:24:54.545 --> 00:24:57.286
Now I want to see what AI Kyle said.

268
00:24:57.706 --> 00:24:59.167
And, yeah, it's just in the terminal.

269
00:24:59.187 --> 00:25:02.828
I didn't have, like, I didn't go the full whack and make a video thing, but I'm sorry.

270
00:25:04.049 --> 00:25:05.591
But I found it immensely useful.

271
00:25:05.831 --> 00:25:10.277
I just wanted to ask, what other weird things are you seeing people do, internally or externally?

272
00:25:10.537 --> 00:25:17.507
Oh man, so it's so funny that you say that because I do a very similar thing where I,

273
00:25:19.469 --> 00:25:24.671
I have both via the app and then I have a claw that can't talk to work stuff, you know?

274
00:25:24.691 --> 00:25:32.295
So I have separation of a state, where I spend a lot of time having it read everything I write and say.

275
00:25:32.515 --> 00:25:34.936
Like this interview will get fed into it ultimately.

276
00:25:36.177 --> 00:25:43.260
And every day I get a comms report, that's not like what Kyle said, but like,

277
00:25:43.940 --> 00:25:45.560
Kyle, you keep saying this.

278
00:25:46.701 --> 00:25:47.881
This isn't super clear.

279
00:25:48.501 --> 00:25:56.403
Based on how you speak, because I find that I write and speak in a very particular way, that I want to use a lot of metaphors.

280
00:25:56.443 --> 00:26:00.224
And so it'll just give me examples of metaphors that are clearer.

281
00:26:01.024 --> 00:26:05.865
I find that the self-improvement loop as a human

282
00:26:06.465 --> 00:26:08.686
from these agents to be incredibly powerful.

283
00:26:08.706 --> 00:26:12.508
We used to talk about it way back with Hubot at GitHub, like chat ops.

284
00:26:13.009 --> 00:26:17.551
And we used to say, like, humans are way more willing to take critical feedback from robots than other humans.

285
00:26:17.571 --> 00:26:18.712
Yeah, it's less threatening.

286
00:26:18.792 --> 00:26:19.072
100%.

287
00:26:19.112 --> 00:26:28.637
And so when my open claw that I affectionately named Baxter, you know, tells me how terrible I did in something, like, I feel way better going, tell me why.

288
00:26:28.797 --> 00:26:33.760
And then ensure that when I'm writing emails, when I'm writing a script or I'm reviewing details, that...

289
00:26:34.340 --> 00:26:35.702
you're giving me that feedback.

290
00:26:35.722 --> 00:26:41.128
So a lot of my agent loop is really about me and less about the software side.

291
00:26:41.149 --> 00:26:42.871
I still have all those tools too.

292
00:26:43.712 --> 00:26:45.294
But it's always looking backwards.

293
00:26:45.374 --> 00:26:49.379
It's going, okay, the last seven days, I'm going to read all Kyle's emails, Slack messages, you know,

294
00:26:49.939 --> 00:26:53.620
And then give me feedback and then look back at what the agent told me to do.

295
00:26:54.060 --> 00:26:56.001
Did Kyle do it and go back seven days?

296
00:26:56.061 --> 00:26:59.202
That loop is like super, super, super powerful.

297
00:26:59.282 --> 00:27:09.585
And I think honestly, like the type of personal consumer experience that I want out of AI, you know, to be able to tune these tools that way.

298
00:27:09.625 --> 00:27:11.806
Yeah, we need to recursively self-improve as well.

299
00:27:11.826 --> 00:27:12.446
A hundred percent.

300
00:27:12.526 --> 00:27:12.706
Yeah.

301
00:27:12.726 --> 00:27:13.286
Thanks so much, man.

302
00:27:13.426 --> 00:27:13.886
I appreciate it.

303
00:27:13.926 --> 00:27:14.827
I appreciate it.

304
00:27:14.847 --> 00:27:15.227
Thank you.

305
00:27:15.267 --> 00:27:15.627
Enjoyed that.

306
00:27:23.885 --> 00:27:29.567
Oh my gosh, folks, you absolutely positively have to smash that like button and subscribe to AI and I.

307
00:27:29.787 --> 00:27:30.047
Why?

308
00:27:30.507 --> 00:27:32.688
Because this show is the epitome of awesomeness.

309
00:27:32.948 --> 00:27:34.848
It's like finding a treasure chest in your backyard.

310
00:27:35.148 --> 00:27:40.630
But instead of gold, it's filled with pure unadulterated knowledge bombs about chat GPT.

311
00:27:41.450 --> 00:27:48.874
Every episode is a roller coaster of emotions, insights, and laughter that will leave you on the edge of your seat, craving for more.

312
00:27:49.154 --> 00:27:49.974
It's not just a show.

313
00:27:50.494 --> 00:27:54.116
It's a journey into the future with Dan Shipper as the captain of the spaceship.

314
00:27:55.337 --> 00:27:56.877
So do yourself a favor.

315
00:27:57.138 --> 00:28:00.419
Hit like, smash subscribe, and strap in for the ride of your life.

316
00:28:00.859 --> 00:28:06.202
And now, without any further ado, let me just say, Dan, I'm absolutely hopelessly in love with you.