WEBVTT 1 00:00:00.111 --> 00:00:05.277 If you're hitting the limits of vibe coding and want to get more serious about engineering, you're in the right place. 2 00:00:05.617 --> 00:00:10.623 And if you're an engineer looking to understand how to use AI to build systems you can trust, same. 3 00:00:10.683 --> 00:00:12.945 You'll want to use the tool from today's guest. 4 00:00:13.406 --> 00:00:15.148 Welcome to AI Worth Using. 5 00:00:15.488 --> 00:00:21.475 Today you'll learn why our guest says shipping unreviewed AI code is like driving without a seatbelt. 6 00:00:21.855 --> 00:00:25.657 We'll look at RoboRev built by Wes McKinney, the creator of Pandas. 7 00:00:26.277 --> 00:00:30.999 If you've developed with Python, you'll almost certainly use Pandas and you'll want to use RoboRev as well. 8 00:00:31.319 --> 00:00:35.461 These days, Wes doesn't write his code and he doesn't review most of it either. 9 00:00:35.621 --> 00:00:48.626 But before you judge him, know that he's replaced himself with something far pickier, an adversarial review system where every commit his agents produce gets torn apart by a rival AI with zero stake in the code 90 seconds after it's written. 10 00:00:48.826 --> 00:00:57.228 By the end of the episode, you'll know how to install it, the one line rule that makes the whole thing work, and why small commits catch bugs that 4,000 line pull requests hide. 11 00:00:57.448 --> 00:00:59.669 We install it live, mistakes included. 12 00:01:00.089 --> 00:01:00.649 Stick around. 13 00:01:06.651 --> 00:01:15.653 Now, you and I have both been around in the software world long enough to remember when senior engineering meant knowing where the prod API keys were. 14 00:01:17.345 --> 00:01:20.669 Yeah, knowing where they are and knowing when not to use them. 15 00:01:20.749 --> 00:01:28.858 Today, I want to bring on somebody who's arguably one of the wiser voices about where this AI coding wave actually comes from, not the hype version. 16 00:01:30.004 --> 00:01:30.804 I'm super curious. 17 00:01:31.165 --> 00:01:39.048 You know, I've had my own moments of kind of recalibrating what it means to be a software developer with all of these tools that we have now. 18 00:01:39.649 --> 00:01:41.329 Some days it feels like a superpower. 19 00:01:41.349 --> 00:01:44.971 I'm just like cranking stuff out and it's amazing and I love it. 20 00:01:45.371 --> 00:01:52.655 And other days it feels like I'm sort of babysitting a junior developer who just like broke the build. 21 00:01:53.175 --> 00:01:54.757 Our guest today is Wes McKinney. 22 00:01:54.817 --> 00:02:03.206 Wes created Pandas, the Python data analysis library that is at this point basically load-bearing infrastructure for the entire data science world. 23 00:02:03.686 --> 00:02:06.469 Almost all other engineers have used Pandas. 24 00:02:06.829 --> 00:02:09.512 Probably most of them don't understand it, myself included. 25 00:02:09.932 --> 00:02:11.794 The reason Wes is here today is because... 26 00:02:12.595 --> 00:02:21.338 of what he's been building over the last eight months, a software stack entirely agent built to solve problems he has been thinking about for years. 27 00:02:21.878 --> 00:02:33.221 The centerpiece of the system is called RoboRev, an adversarial code review pipeline he built so that every line of code his agents generate gets reviewed by a different model before it ships. 28 00:02:33.641 --> 00:02:34.942 Tom West McKinney. 29 00:02:35.898 --> 00:02:36.719 Wes, welcome. 30 00:02:37.080 --> 00:02:38.021 I've been looking forward to this. 31 00:02:38.261 --> 00:02:38.842 Thanks for having me. 32 00:02:38.982 --> 00:02:39.283 Me too. 33 00:02:39.603 --> 00:02:45.291 Okay, Wes, you told me you muted LLMs in your Twitter timeline in 2023. 34 00:02:45.931 --> 00:02:50.337 As a man who built one of the most used tools in history, 35 00:02:51.018 --> 00:02:52.919 deliberately turning tuning that out. 36 00:02:53.099 --> 00:02:53.840 What was that about? 37 00:02:54.080 --> 00:03:01.725 I think they're still muted in my Twitter feed and mainly because I haven't figured out how to, you know, dig through the settings in X.com and turn off the mute. 38 00:03:01.805 --> 00:03:06.348 I think I also still have open AI muted like the term open AI because I was a little bit, I was tired of, I don't know, 39 00:03:06.428 --> 00:03:10.610 my logging into Twitter and just seeing like nothing but LLM stuff and open AI stuff. 40 00:03:10.650 --> 00:03:22.054 But I think for a long time it was a little bit like, let me go to sleep until this is relevant for me because it felt very research focused and science and experimentation and AI research. 41 00:03:22.374 --> 00:03:23.794 And I'm a systems person. 42 00:03:23.834 --> 00:03:26.215 I like building software and systems. 43 00:03:26.955 --> 00:03:31.157 And just for the longest time, it didn't feel like AI was very relevant for me. 44 00:03:31.757 --> 00:03:38.141 And even up until I think it wasn't until Cloud Code came around that it started to seem, OK, this is starting to become relevant. 45 00:03:38.161 --> 00:03:49.387 I should tune in now and start figuring out how to apply this in my work, because even in 2024, I tinkered a little bit with Cursor and Windsurf and Copilot and VS Code and things like that. 46 00:03:50.508 --> 00:03:52.791 It felt a little bit like fancy autocomplete. 47 00:03:53.011 --> 00:04:00.719 Okay, this maybe helps me a little bit be more productive, but mostly it's getting in my way and I can still do this better and do this faster on my own. 48 00:04:01.260 --> 00:04:07.487 And I was still largely working in Emacs to do, even without LSPs, without a lot of assistants, just writing code into... 49 00:04:08.027 --> 00:04:09.488 a terminal window like a caveman. 50 00:04:09.928 --> 00:04:17.853 But it was really that sea change about a year ago when the first iterations of coding agents came out where it felt like, okay, something is changing, something is different now. 51 00:04:18.413 --> 00:04:22.876 This is starting to become relevant for me, somebody who builds systems, who builds... 52 00:04:22.896 --> 00:04:26.558 I've built largely data systems, like things that have to be correct all the time. 53 00:04:26.598 --> 00:04:28.639 You can't have correctness issues. 54 00:04:28.719 --> 00:04:30.520 It computes the library like Pandas. 55 00:04:30.600 --> 00:04:31.521 It does math. 56 00:04:31.561 --> 00:04:32.421 It does analytics. 57 00:04:32.481 --> 00:04:33.302 It computes things. 58 00:04:33.882 --> 00:04:34.523 And the whole... 59 00:04:35.163 --> 00:04:42.425 AI is non-deterministic, you need to have some confidence that you're building software that you can rely on and that you can sleep easy at night. 60 00:04:42.985 --> 00:04:47.287 And so certainly the first iteration of Cloud Code was pretty rough around the edges. 61 00:04:47.507 --> 00:04:48.087 It would lie. 62 00:04:48.107 --> 00:04:52.608 It would assert that things were production ready and ready to ship when clearly they were not. 63 00:04:52.628 --> 00:04:54.829 It would claim it finished things when it hadn't. 64 00:04:55.289 --> 00:05:03.192 And so it was evident even in those early iterations of CLI, terminal coding agents, that there was a need for substantial guardrails. 65 00:05:03.452 --> 00:05:03.632 And... 66 00:05:04.272 --> 00:05:18.524 systems around the systems to help make like, how can we leverage these tools to actually be productive rather than just creating a bunch of slop that's brittle and isn't safe for real production work in the real world, like businesses that need software that they can depend on. 67 00:05:18.784 --> 00:05:20.846 Was that thinking the genesis of RoboRov? 68 00:05:21.186 --> 00:05:23.768 It was, it took a little while to come to the realization. 69 00:05:23.968 --> 00:05:28.692 Like I really enjoyed like just the general vibe of using cloud code to borrow a term. 70 00:05:28.752 --> 00:05:33.536 But then I found that Claude was simply not that good at doing code review and 71 00:05:33.816 --> 00:05:37.979 And in the meantime, OpenAI shipped their first coding agent, Codex. 72 00:05:38.440 --> 00:05:43.023 And I didn't really enjoy the experience of using Codex to build software. 73 00:05:43.083 --> 00:05:48.767 Just it felt a little bit slow, like the agent a little bit deliberative, like not very eager to do things, a bit timid. 74 00:05:49.168 --> 00:06:01.057 But what I found just by experimentation was that Codex and GPT-5 in particular was tremendously good at finding bugs in code, like finding edge cases, issues in my test suite. 75 00:06:01.517 --> 00:06:03.259 finding security issues, things like that. 76 00:06:03.399 --> 00:06:10.546 And so essentially what I found myself doing was running Claude code in one terminal pane and then Codex in another terminal pane. 77 00:06:10.586 --> 00:06:14.049 And anytime Claude would do something, I would have it commit immediately. 78 00:06:14.309 --> 00:06:17.692 And then in my Codex session, I would say, okay, Codex, review this commit. 79 00:06:18.313 --> 00:06:22.358 And then after a while, I was doing that so many times a day, I was like, why don't I just automate this? 80 00:06:22.418 --> 00:06:23.559 Like, why do I even have to ask? 81 00:06:23.619 --> 00:06:25.101 It should just happen automatically. 82 00:06:25.701 --> 00:06:28.344 And so that was the first week of January this year. 83 00:06:28.404 --> 00:06:35.852 Like the first iteration of RoboRev was just a very simple daemon that would be triggered by a commit hook, post commit hook. 84 00:06:36.032 --> 00:06:36.553 So whenever... 85 00:06:37.554 --> 00:06:48.581 you tell your Cloud Code sessions in Cloud.md, every time I prompt, like every time you return control to me, never leave the Git repository in a dirty state, like never leave uncommitted changes, always commit. 86 00:06:49.041 --> 00:06:51.023 And so that creates like the commit loop. 87 00:06:51.083 --> 00:06:53.304 So anytime the agent does anything, it commits. 88 00:06:53.784 --> 00:06:56.306 And that triggers a code review that runs in the background. 89 00:06:56.866 --> 00:07:04.491 And then initially I was just copy and pasting the code reviews from the RoboRev system into Cloud Code, which got me pretty far. 90 00:07:04.611 --> 00:07:05.432 But eventually... 91 00:07:05.972 --> 00:07:15.156 It became more hook based where you ask, especially during like marathon implementation sessions, especially like using some of the spectrum and development frameworks. 92 00:07:15.316 --> 00:07:19.638 You want it to automatically pull in the code reviews that have piled up while it's implementing. 93 00:07:19.818 --> 00:07:22.360 So I'm a superpowers user, for example. 94 00:07:22.540 --> 00:07:27.122 And so I might implement like a 40 task superpowers plan and then I'll tell. 95 00:07:27.802 --> 00:07:36.465 The superpowers implementer harness run RoboRev fix every five tasks to address code review feedback that is piled up in the background. 96 00:07:36.485 --> 00:07:40.106 And so the idea is that now it's the system that runs in the background. 97 00:07:40.146 --> 00:07:43.227 So it's not impeding progress in your implementation work. 98 00:07:43.267 --> 00:07:44.367 It's running in the background. 99 00:07:44.407 --> 00:07:49.149 So in a sense, it's like helping you move faster, like reviewing the code while you continue working. 100 00:07:49.769 --> 00:07:54.451 But still, their code reviews end up in a durable ledger where they're not forgotten about. 101 00:07:54.651 --> 00:07:58.752 So if there's a failing code review, you have to be you have to explicitly close it either. 102 00:07:58.932 --> 00:08:05.935 Typically, it's the agents closing it, but you can always refer back to that ledger of code reviews and say, oh, there's these 10 commits that have been created by the agent. 103 00:08:06.375 --> 00:08:10.897 Let's pull them all in, collect all the findings and then address them by order of severity. 104 00:08:11.457 --> 00:08:16.001 And sometimes the agents have fixed issues that were found in previous commits in passing. 105 00:08:16.141 --> 00:08:22.967 And so that's actually helpful because the agent has an opportunity to verify like, oh, this was a problem five commits ago. 106 00:08:23.027 --> 00:08:23.628 Now it's fixed. 107 00:08:23.788 --> 00:08:24.248 That's good. 108 00:08:24.288 --> 00:08:25.309 We confirm that it's fixed. 109 00:08:25.449 --> 00:08:29.473 But if there's things that are lingering remaining, then you get an opportunity to fix them. 110 00:08:30.053 --> 00:08:37.775 But what I found is that no matter the agent and no matter how good the models get, they just leave behind a lot of sloppy, buggy code. 111 00:08:38.035 --> 00:08:43.217 And even if the code is conforming to your specification, there's still going to be bugs in the implementation. 112 00:08:43.257 --> 00:08:48.858 There's going to be edge cases that are missed, test cases that are written incorrectly or somehow incomplete. 113 00:08:49.398 --> 00:08:51.999 And so having that adversarial check like... 114 00:08:52.479 --> 00:08:55.960 you're a code reviewer, you didn't write this code, you have no stake in it, find the bugs. 115 00:08:56.000 --> 00:08:57.361 Like, I believe that there's bugs. 116 00:08:57.681 --> 00:08:58.541 There's always bugs. 117 00:08:58.661 --> 00:09:01.702 And so please help find them, help make this the code quality better. 118 00:09:02.242 --> 00:09:13.166 And so having like multiple perspectives on every line of code that gets generated by agents definitely makes the end product in my experience like a lot higher quality and something you can feel a lot more confident about shipping. 119 00:09:13.226 --> 00:09:19.548 By the time you get to the end, if you have a clean review on the whole branch, you can click the merge button in GitHub and feel a lot better about it. 120 00:09:20.138 --> 00:09:21.539 Now, I'm addicted to RoboRev. 121 00:09:21.559 --> 00:09:25.981 I probably spend half a billion tokens a day on RoboRev. 122 00:09:26.001 --> 00:09:32.924 So you're back to shipping open source software, but in early 2025, you described that as a moment of existential dread. 123 00:09:33.425 --> 00:09:41.309 Can you tell us a little bit about that journey from muting LLMs to that experience and how you've changed your thinking about AI? 124 00:09:42.449 --> 00:09:43.510 Your role in AI. 125 00:09:43.691 --> 00:09:56.966 I think I was, you know, I'd read a lot of blog posts and thought leadership around AI and was a little bit unsure what to make about some of the doomerism around AI and the whole idea of software engineers being... 126 00:09:57.567 --> 00:10:00.590 put out of work or not having a purpose in the future. 127 00:10:01.231 --> 00:10:04.654 And so I was grappling a little bit with, what is my role here? 128 00:10:04.714 --> 00:10:05.615 What is my purpose? 129 00:10:05.696 --> 00:10:08.178 Do I, am I still going to have agency in the future? 130 00:10:08.218 --> 00:10:11.221 Is there gonna be value for me in the loop? 131 00:10:11.542 --> 00:10:15.185 Or is my career as a software engineer essentially coming to a close? 132 00:10:15.926 --> 00:10:29.692 And so I think partly what's motivated me about building these tools and working, especially last six or eight months, has been the feeling of regaining agency and figuring out what is the role of an expert software developer in the loop. 133 00:10:29.872 --> 00:10:35.575 And it turns out that we still have a pretty valuable role, which is deciding what to build and knowing when it's good. 134 00:10:36.275 --> 00:10:44.699 And the agents have gotten a lot better, but they still aren't very good at coming up with the novel idea, the concept of what the software needs to do. 135 00:10:45.459 --> 00:10:48.101 And they seem to not be very good about judging when it's good. 136 00:10:48.181 --> 00:10:53.724 So you still need humans in the loop to be able to judge the quality, especially when it's human facing tools. 137 00:10:53.804 --> 00:10:54.724 Is this a good thing? 138 00:10:54.965 --> 00:10:58.527 Agents can tell you, does it work based on what the specification says it should do? 139 00:10:59.087 --> 00:11:05.691 But they have a hard time telling you with some confidence that it's actually good as humans would recognize it as good. 140 00:11:06.531 --> 00:11:16.716 So, yeah, I definitely passed through a bit of an existential crisis and had a few months in early 2025 where I was feeling disengaged and maybe software engineering is just not what I'm going to be doing in the future. 141 00:11:16.816 --> 00:11:18.577 But I definitely don't feel that way anymore. 142 00:11:18.657 --> 00:11:21.178 And as the months pass by, I feel even more like... 143 00:11:22.118 --> 00:11:34.271 I feel folks that have a lot of software engineering experience building production systems have even more value now and can be afforded even more leverage by these tools if they can wield them effectively. 144 00:11:34.671 --> 00:11:37.173 And probably 90% of the challenge is using them effectively. 145 00:11:37.892 --> 00:11:44.960 So Wes, you mentioned that you get this going in a loop and you have one agent reviewing another agent. 146 00:11:45.881 --> 00:11:53.049 Do you feel like Claude is not good at reviewing Codex code or which agent is the best today? 147 00:11:53.069 --> 00:11:53.170 Yeah. 148 00:11:54.058 --> 00:11:55.058 It depends on the day. 149 00:11:55.239 --> 00:11:58.500 Like Claude, I found that Claude has good days and bad days. 150 00:11:58.600 --> 00:12:03.782 Like the last few days, actually, like Claude has been having bad days really since the 4.8 release. 151 00:12:04.042 --> 00:12:06.123 It's been a little bit of struggle bus. 152 00:12:06.563 --> 00:12:07.724 So they really are like humans. 153 00:12:07.884 --> 00:12:09.044 They have good days and days. 154 00:12:09.284 --> 00:12:15.247 Yeah, so I put Claude on ice and I've been largely using Codex as my driver and reviewer. 155 00:12:15.787 --> 00:12:20.509 But even when you're using the same model to implement and review, there's a lot of benefit. 156 00:12:21.109 --> 00:12:25.652 counterintuitively and having reviews done by an adversarial agent. 157 00:12:25.692 --> 00:12:33.257 So like an agent with an empty context, no skin in the game, just here's a commit that was written by another agent. 158 00:12:33.337 --> 00:12:37.180 Here's a rubric for what you are looking for in the code review. 159 00:12:37.200 --> 00:12:44.825 You might want to have a code review that's more design focused or one that's more general purpose, like just look for bugs and quality, general quality issues. 160 00:12:45.825 --> 00:12:48.068 And a brand new agent session. 161 00:12:48.088 --> 00:12:57.260 So the idea is in RoboRev, every review is a brand new non-interactive agent session run with either Claw-P or Codex Exec. 162 00:12:57.500 --> 00:13:01.345 And all the agent harnesses have some form of like non-interactive session. 163 00:13:01.525 --> 00:13:02.586 non-interactive session. 164 00:13:03.186 --> 00:13:14.811 And just through trial and error and experimentation, like we've been able to engineer the prompts and get just the quality, just the review quality to a level where it's very, it's fairly expensive to run. 165 00:13:14.891 --> 00:13:18.913 Like it's a lot more expensive than like using CodeRabbit or a reptile or one of these like 166 00:13:19.513 --> 00:13:24.437 software as a service code review systems that run on top of GitHub and the other code forges. 167 00:13:24.717 --> 00:13:33.063 Personally, even if I were paying API rates, like for my personal development, I'd be spending, I don't know, around $1,000 a month on reviews, maybe a little more some months. 168 00:13:33.663 --> 00:13:39.547 But to me, that's tremendously valuable to not have to spend my time reading the code. 169 00:13:39.587 --> 00:13:44.571 Is it worth $1,000 a month to me to not have to read the code or have to read every line and look for bugs? 170 00:13:44.991 --> 00:13:45.472 Absolutely. 171 00:13:45.492 --> 00:13:46.012 Absolutely. 172 00:13:46.192 --> 00:13:57.325 And I feel like good code review, especially given how much code we're able to generate with these things, that the code review becomes even more important because it's the quality gate about whether you're shipping slop or not. 173 00:13:57.565 --> 00:14:03.732 The code still might be slop in the sense that like it has a bunch of features which are not needed, like you should have said no to a bunch of stuff. 174 00:14:03.752 --> 00:14:03.912 Yeah. 175 00:14:04.493 --> 00:14:06.495 There's things that need to be cut from the product scope. 176 00:14:07.035 --> 00:14:16.162 But just in terms of like pure bugginess, spend $1,000 a month on tokens to have 90% less bugs in the code to me is worth a lot. 177 00:14:16.883 --> 00:14:21.967 Why is your approach more expensive than some of these other alternatives like CodeRabbit or Greptile? 178 00:14:22.402 --> 00:14:35.736 setting a frontier coding agent loose to review not only the diff of the code that was changed, like review this change, but review it in context with the rest of the code base. 179 00:14:36.437 --> 00:14:43.424 So you're giving the agent free reign to do tool calling to if there's something that it finds that 180 00:14:44.165 --> 00:14:51.130 It may not know for certain that it's a bug without writing some speculative test code to verify that it's a bug. 181 00:14:51.170 --> 00:15:06.180 And so it might, if you look at the traces for these reviews, it might run dozens or for very long reviews, it could run hundreds of tool calls to do the investigation and exploration to determine, to find the issues in the code. 182 00:15:06.740 --> 00:15:17.452 I think some of the first iterations of AI code review tools weren't agentic in the sense that they didn't do tool calls and they largely just did static analysis based on the diff. 183 00:15:17.512 --> 00:15:23.058 Look at the diff and maybe you're given access to the surrounding context, like files that were changed. 184 00:15:23.078 --> 00:15:26.541 And if they fit into the model context, they might be included in the prompt. 185 00:15:27.082 --> 00:15:30.264 But they weren't agentic in the sense that they didn't do tool calling. 186 00:15:30.284 --> 00:15:34.427 They weren't allowed to explore and write test code and verify findings. 187 00:15:34.487 --> 00:15:36.869 And so that led to a very high false positive rate. 188 00:15:37.369 --> 00:15:50.038 Not only false positives, but also just missed findings because some of the bugs that occur in real code are just so subtle and maybe difficult to grok just purely based on static analysis of the diff. 189 00:15:50.916 --> 00:15:56.697 This is interesting, Wes, because you're using a subscription here to run RoboRev, right? 190 00:15:56.737 --> 00:16:03.459 Whereas Greptile has to pay API rates so you can use tokens that are subsidized by these subscriptions. 191 00:16:03.499 --> 00:16:03.839 Yeah. 192 00:16:03.999 --> 00:16:05.180 And I'm entirely running. 193 00:16:05.760 --> 00:16:10.761 I haven't paid API rates on any of my RoboRev use since the beginning. 194 00:16:11.221 --> 00:16:14.082 And who knows how much longer that's going to last. 195 00:16:14.882 --> 00:16:19.026 OpenAI in particular has been really generous with the token budgets. 196 00:16:19.346 --> 00:16:30.376 I understand that they're going to reduce the codex token budgets, but they've also been really permissive with respect to non-interactive and programmatic use with the codex subscription. 197 00:16:30.436 --> 00:16:35.981 So Anthropic is actually taking away Claude-P, which is the non-interactive token. 198 00:16:36.141 --> 00:17:01.082 invocations of cloud code which means that to use continue to use robo rev with cloud code users are going to have to pay extra usage or api rates they won't be able to freeload on their max subs whereas it seems that open ai stance is that they're happy to have developers building workflows around codex exec and non-interactive use and maybe they'll based on the economics of tokens maybe they'll change that in the future but for now they seem very permissive 199 00:17:01.582 --> 00:17:17.773 But as I was saying, even if I had to pay for pay the API rates, it's so tremendously valuable to me in terms of my time savings and the amount of software that I can confidently build in a given amount of time that I would be happy to pay the pay for the tokens a la carte if I needed to. 200 00:17:18.855 --> 00:17:23.756 And have you done an analysis of the quality of the bug finding? 201 00:17:23.916 --> 00:17:36.119 I often find when I'm just in a chat session with one of these guys that I have to I find myself being very careful about how I pose the question, because I feel like if I ask it in a certain way, it's going to give me the answer that I want. 202 00:17:36.199 --> 00:17:42.020 And I have to try to be like as impartial as possible so that I'm not leading it down a specific path. 203 00:17:42.820 --> 00:17:49.365 And I feel like anytime you say, hey, here's some code, find the bugs, it's going to be like, absolutely, I will find you a bunch of changes that I could make. 204 00:17:50.166 --> 00:17:55.069 I wonder sometimes, is it worth the bang for the buck or how do you control that? 205 00:17:55.089 --> 00:17:57.571 Where it's like, sure, I could rewrite this in three different ways. 206 00:17:57.631 --> 00:17:59.393 Let me go spend some tokens to do that for you. 207 00:18:00.353 --> 00:18:04.997 How good is it at actually finding real high quality, impactful bugs? 208 00:18:05.117 --> 00:18:05.317 Yeah. 209 00:18:05.798 --> 00:18:13.785 In my experience, it's very good, especially like the latest iterations of the like the GPT-5 models like GPT-5.5. 210 00:18:14.005 --> 00:18:18.549 I find that, again, this is like a vibes based judgment, like not an evals based one. 211 00:18:19.149 --> 00:18:25.295 But based on like my daily driver, like vibes based experience, like I would say that the like... 212 00:18:25.875 --> 00:18:28.956 the rate of validity is somewhere in the 70 to 80% range. 213 00:18:28.996 --> 00:18:42.282 And so you end up with maybe 20%, like 20% false positives, which on the face of it is like not great to have a lot of false positives, but also the agents are pretty effective about pushing back when you tell it like, like, 214 00:18:42.782 --> 00:18:50.047 The agent thinks it found a bug and it looks and it's no, this is not valid because and it will generally explain to you why, like why it's not a bug. 215 00:18:50.567 --> 00:18:57.792 In the 80% of cases, if you present the findings to Claude, almost 80, at least 80% of the time, it feels like it's like good catches. 216 00:18:57.912 --> 00:18:59.953 It was code that was freshly generated by Claude. 217 00:18:59.973 --> 00:19:06.538 And so you're also asking typically with RoboRev, you're one of the things, the principles is that you don't want to have to push the. 218 00:19:07.138 --> 00:19:11.739 to GitHub and wait for code review and then come back 20 minutes later and then fold in the code review findings. 219 00:19:11.759 --> 00:19:16.360 Typically, you're getting the review findings around 90 seconds after the code was generated. 220 00:19:16.420 --> 00:19:25.883 So you're referring, you're providing feedback on the code essentially in the same breath as like it's still in context, like the agent was just thinking about it, just reasoning about it. 221 00:19:26.443 --> 00:19:31.327 And so typically you're presenting review findings that the agent was just working on. 222 00:19:31.367 --> 00:19:51.543 And so it hasn't had an opportunity to compact or like forget the context about what it was working on, which also means that if you present a false positive, it will push if it's incorrect, it will be able to push back without having to do a bunch of reasoning and having to like, you know, refresh its understanding of the code to be able to judge whether it's like a real bug or not. 223 00:19:51.986 --> 00:19:58.173 I will say one, there is like a trend towards like pedanticness, for lack of a better term. 224 00:19:58.253 --> 00:20:11.589 And I think GPT 5.1 and 5.2 were very pedantic and almost to the point where I would often preface, I would tell Claude, here's findings, avoid pedantic rabbit holes, be pragmatic. 225 00:20:11.769 --> 00:20:12.390 I would give like... 226 00:20:12.650 --> 00:20:19.272 prompts like that to help nudge the agent to not chase after like things that are never going to happen for the rest of our lives kind of things. 227 00:20:19.432 --> 00:20:24.254 But for whatever the Codex team is doing, they've definitely made things less pedantic. 228 00:20:24.694 --> 00:20:29.455 And just, yeah, the overgeneral effectiveness seems the rate of quality finding seems higher to me. 229 00:20:29.855 --> 00:20:31.156 I think another thing that helps is that 230 00:20:32.118 --> 00:20:38.043 At any given time, the amount of code that it's looking at is maybe 100 lines of code at a time. 231 00:20:38.063 --> 00:20:40.946 So you're not asking it like review 10,000 lines of generated code. 232 00:20:40.986 --> 00:20:45.269 And that also helps with putting on the horse blinders and looking at stuff that matters. 233 00:20:45.490 --> 00:20:48.832 Okay, Wes, I want to jump into really seeing how this works. 234 00:20:49.433 --> 00:20:52.916 And you have RoboRev installed in your computer, I assume. 235 00:20:53.136 --> 00:20:54.457 Is there anything you can show us? 236 00:20:55.418 --> 00:20:55.738 Sure. 237 00:20:55.898 --> 00:21:02.722 Yeah, let me jump in and just give a little tour of what it looks like when I'm using the system. 238 00:21:03.783 --> 00:21:05.244 If you all can see my terminals. 239 00:21:06.224 --> 00:21:10.587 And see here, this is an actual branch that we're working on RoboRev. 240 00:21:10.747 --> 00:21:13.528 And I'm actually working on the RoboRev code base itself. 241 00:21:13.548 --> 00:21:15.089 I have a codec session here. 242 00:21:15.570 --> 00:21:18.331 This is the RoboRev terminal interface. 243 00:21:18.571 --> 00:21:21.093 And so you can see that this morning I started... 244 00:21:21.953 --> 00:21:23.835 a branch that's actually a get work tree. 245 00:21:24.015 --> 00:21:30.780 And so then these are all reviews that were run as a result of me prompting Codex to do work on this project. 246 00:21:31.101 --> 00:21:36.805 And so over here we have the closed column, which indicates whether the review has been addressed or not. 247 00:21:37.366 --> 00:21:41.009 The P or F is whether the review has passed or failed. 248 00:21:41.649 --> 00:21:49.474 And so if we look at one of these failed reviews, it shows you here's the token use context and output tokens. 249 00:21:49.494 --> 00:21:52.836 So this review would have cost around $1.80 to run. 250 00:21:52.856 --> 00:21:56.198 And so it found a medium severity finding, a low severity finding. 251 00:21:56.678 --> 00:22:01.101 There's a comment which was run by the RoboRevFixSkill when this comment was addressed. 252 00:22:01.301 --> 00:22:05.784 And in the 2E, you press the A key to close or reopen the issue. 253 00:22:07.525 --> 00:22:16.849 And so sometimes when I started building the project, initially what I did was you press the Y key here to copy the review and you can just paste it into codecs and it can address it that way. 254 00:22:16.929 --> 00:22:20.531 But the even easier way, I'll actually, let's see here. 255 00:22:21.111 --> 00:22:26.314 There's some review findings that were, I ran a branch review with. 256 00:22:26.434 --> 00:22:30.916 So if I do RoboRev review branch, so we could review that. 257 00:22:30.976 --> 00:22:32.817 And then maybe I'll also review the same review. 258 00:22:33.877 --> 00:22:37.901 branch with Claude here, and we could also do with Gemini just for fun. 259 00:22:38.562 --> 00:22:44.327 And so you can see that is cued three reviews on this entire branch that I'm working on. 260 00:22:44.908 --> 00:22:46.670 If you look at them, here's the prompt. 261 00:22:46.850 --> 00:22:48.812 You can look at the prompt that's been sent to the agent. 262 00:22:49.432 --> 00:22:56.619 So it includes rolling context like previous review findings, which helps with reinforcing don't keep finding the same things over and over again, find new things. 263 00:22:57.280 --> 00:23:01.903 And there's also the inline diff content from the change that's being looked at. 264 00:23:02.624 --> 00:23:04.105 And so those will run in the background. 265 00:23:04.885 --> 00:23:12.830 And whenever there's findings, so in Codex, for example, you run the RoboRev fix skill, and we see there's already some findings here. 266 00:23:13.491 --> 00:23:19.755 And so you can see that it will use the RoboRev CLI to find open reviews. 267 00:23:20.015 --> 00:23:21.997 It will pull those in, and it will start fixing them. 268 00:23:22.817 --> 00:23:31.779 And the way that makes this work most effectively is in your agents.md, you instruct the agents to always commit. 269 00:23:32.039 --> 00:23:36.240 So commit every prompt, commit every time you return control to the user. 270 00:23:36.400 --> 00:23:40.841 And so this helps them make sure that they don't leave a bunch of unfinished work in the branch. 271 00:23:41.021 --> 00:23:47.022 And I think that creates a commit, which triggers an automatic review that happens in the background in RoboRev. 272 00:23:47.742 --> 00:23:58.590 And so especially like when you're using a spec driven development framework like superpowers, you might have this marathon session where Codex is working for five hours and is generating like dozens of commits. 273 00:23:58.650 --> 00:24:06.716 And so the idea is that the work gets broken up into incrementally reviewable chunks that get analyzed as the agent is implementing. 274 00:24:07.176 --> 00:24:11.540 And so you get review feedback on a narrow slice of work, like one step at a time. 275 00:24:12.240 --> 00:24:18.507 And that, in my experience, tends to surface like the big stuff as well as the small stuff. 276 00:24:19.348 --> 00:24:28.377 And in your agents.md, having the agent always commit creates that constant background trail of code reviews on the generated code. 277 00:24:28.898 --> 00:24:37.522 And then either you can ask Claude or Codex to automatically invoke the fixed skill and address the reviews periodically, or you can invoke it manually. 278 00:24:37.982 --> 00:24:50.248 I tend to be in the habit of invoking it manually, or if I'm using like a long superpowers development thread, like there might be 30 or 40 tasks, I'll ask it to invoke the fixed skill every five or 10 tasks so that... 279 00:24:50.828 --> 00:24:59.379 the agent is pausing to do some cleanup and fixing the problems that have been found by the reviewer agents in the background as time as things move onward. 280 00:24:59.459 --> 00:25:02.643 And so you can see some of these reviews, you can see they run for quite a bit. 281 00:25:03.223 --> 00:25:09.391 So like this Gemini review found something here and I'm sure Cloud and Codex, they might find the same thing and they might find a different thing. 282 00:25:10.252 --> 00:25:11.792 The agents are pretty good about pushing back. 283 00:25:11.852 --> 00:25:14.433 And if there's duplicate findings, it will combine them. 284 00:25:14.493 --> 00:25:16.673 And agents are good at doing that sort of thing. 285 00:25:16.813 --> 00:25:22.995 Justin McCarthy at StrongDM, I think Simon wrote about his... Oh, look, a finished review. 286 00:25:23.215 --> 00:25:29.476 He says he targets 90% of his token budget against validation and reviews. 287 00:25:30.176 --> 00:25:30.376 Yeah. 288 00:25:30.876 --> 00:25:32.116 What's your reaction to that? 289 00:25:32.376 --> 00:25:36.417 As an open source developer, how do you think about the quality of code? 290 00:25:36.477 --> 00:25:40.018 And does that change for somebody who's vibe coding a weekend project? 291 00:25:41.479 --> 00:25:43.901 I've been drinking my own Kool-Aid for a while. 292 00:25:44.161 --> 00:25:51.467 And so because I've been using this system, it's made me hyper aware of how buggy the code that the agents generate is. 293 00:25:52.088 --> 00:25:53.389 I'm a little bit like Team Armin. 294 00:25:53.409 --> 00:25:55.130 Like I call them clankers and not agents. 295 00:25:56.531 --> 00:25:59.254 And maybe we're not supposed to use that word. 296 00:25:59.694 --> 00:26:09.102 But I think having this constant supervision and having the code reviewed so many times by the adversarial agents has just made me so aware of how... 297 00:26:10.082 --> 00:26:13.205 like mid quality, the generated code is. 298 00:26:13.405 --> 00:26:21.152 And so I'm at a point where it feels very unsafe to ship any code that hasn't been reviewed, at least in an autonomous manner. 299 00:26:21.492 --> 00:26:27.658 Certainly having humans reviewing code helps with finding structural problems and things that are overcomplicated and stuff that shouldn't be there. 300 00:26:28.278 --> 00:26:34.601 But at minimum, like the automated reviews feels to not have it feels like driving a car without a seatbelt in mid 2026. 301 00:26:34.961 --> 00:26:46.427 Even as good as the models have become, they still have cognitive deficits and they'll get things broadly correct, but yet leave a trail of like little problems that can, you know, cause bugs in production. 302 00:26:46.507 --> 00:26:48.608 So you can see the agent here is about to finish. 303 00:26:48.668 --> 00:26:52.290 It's fixing like a lint issue caught by a pre-commit hook. 304 00:26:52.710 --> 00:26:53.470 This is Go code. 305 00:26:54.111 --> 00:26:55.231 And so we'll see it commit. 306 00:26:55.311 --> 00:26:58.753 And then the review will get to running a test. 307 00:26:59.174 --> 00:26:59.894 It's running GoVet. 308 00:27:00.334 --> 00:27:04.156 I think whenever we see it commit, we'll see the code review trick kick off in the background. 309 00:27:05.137 --> 00:27:19.805 How much time would you say you spend in preparation and design in describing the high level goals that you have versus more tactical on the ground implementation types of things to get things like the architecture correct? 310 00:27:19.825 --> 00:27:20.045 Yeah. 311 00:27:20.510 --> 00:27:21.511 So there it just committed. 312 00:27:21.551 --> 00:27:22.992 And so here's the running code review. 313 00:27:23.092 --> 00:27:25.093 So it's pretty much instantaneous in the background. 314 00:27:25.593 --> 00:27:28.175 How much time I spend preparing and designing? 315 00:27:28.695 --> 00:27:35.400 At this point, in terms of my time in the process, I spend most of my time in the preparation and design. 316 00:27:36.120 --> 00:27:39.783 And the actual implementation and review and refinement is 317 00:27:40.523 --> 00:27:43.124 has become, for me, largely mechanical. 318 00:27:43.664 --> 00:27:45.524 As I mentioned, I use superpowers. 319 00:27:45.584 --> 00:27:49.766 It's a spec-driven development framework, which I think is really popular for a good reason. 320 00:27:50.566 --> 00:27:56.547 And it creates an upfront burden of deciding what you want to build and working through the design decisions. 321 00:27:57.108 --> 00:28:05.250 But then you build an implementation plan, and then that implementation plan might take anywhere from an hour to five hours to 10 hours to implement. 322 00:28:06.070 --> 00:28:15.953 And that is largely a combination of generating the code, verifying that the code conforms to the thing that you agreed that you were building, like conforming to the spec. 323 00:28:16.633 --> 00:28:22.315 And then I run, I have this RoboRev system reviewing all of the generated code to make sure to be a quality gate. 324 00:28:22.795 --> 00:28:28.837 That's like an extra layer of security and quality assurance on top of the spec conformance reviews that... 325 00:28:29.797 --> 00:28:30.678 superpowers does. 326 00:28:31.258 --> 00:28:36.282 But I also RoboRev can also be used to review the specs that are created by superpowers. 327 00:28:36.362 --> 00:28:50.633 And I also use the adversarial review process as part of the design, especially if I'm building something where I am not confident about my own understanding or expertise in an area, like wading into some problem where I don't have... 328 00:28:51.213 --> 00:28:53.495 pre-experience building by hand, pre-AI. 329 00:28:53.515 --> 00:29:01.542 I feel like I'm in over my head and like I'll ask an adversarial session to review design sections or things like that. 330 00:29:02.082 --> 00:29:05.205 So see here, we can see this review just finished and it passed. 331 00:29:05.805 --> 00:29:07.406 Here's the trace from the review. 332 00:29:07.446 --> 00:29:15.753 So as I was saying, you can see it makes a ton of tool calls to look at different parts of the code that are outside of the scope of the diff that was passed in the prompt. 333 00:29:16.414 --> 00:29:18.696 And so here we can look and see here's the diff 334 00:29:19.276 --> 00:29:24.118 So it actually wasn't very much code that was changed, but even there wasn't very much code changed. 335 00:29:24.198 --> 00:29:34.262 And the Codex exec agent did looks like on the order of 35 tool calls or so in the course of verifying and looking for bugs related to that change. 336 00:29:34.863 --> 00:29:41.065 So yeah, in terms of building stuff, I feel very bandwidth limited in terms of my ability to make decisions every day. 337 00:29:41.625 --> 00:29:44.347 And I try to spend as much of my time in this process 338 00:29:45.287 --> 00:29:52.937 making decisions, going through like the design stages of the high level, like what's in the spec and like what's not in the spec. 339 00:29:53.498 --> 00:30:01.588 And then at a certain point, like my goal is to get as many implementation, semi-automated implementation sessions underway in a given day as I can. 340 00:30:02.209 --> 00:30:19.785 And so there might be days where I have four or five superpowers implementation sessions that are going and there's so much work happening that basically my work is done and I can walk, go for a walk or I can cook dinner or something and trust that there's still work happening and it's being supervised by this system that I've put together. 341 00:30:19.945 --> 00:30:27.108 It could seem maybe like a little bit of a Rube Goldberg machine, but we're working on some things to make it a little more like agent harness native. 342 00:30:27.628 --> 00:30:31.910 So there's less need for some people don't ever want to have to invoke a skill manually. 343 00:30:31.930 --> 00:30:42.775 So we're working on like a hook, like a hook for the harness so that whenever there's reviews that need to be fixed, that the agent will be reminded like, hey, there's reviews that need to be fixed. 344 00:30:42.835 --> 00:30:48.017 If you're idle, fix them, stuff like that, because some people just don't like they want it to all happen automatically. 345 00:30:49.237 --> 00:30:52.679 As far as Rube Goldberg machines go, this is not the most complex that I've seen. 346 00:30:53.359 --> 00:30:53.899 Yeah. 347 00:30:54.099 --> 00:30:58.862 And I've tried to like every few days I take a step back, ask myself, like, does this make sense? 348 00:30:58.902 --> 00:31:00.162 Do I have AI psychosis? 349 00:31:00.542 --> 00:31:03.564 Because there are a lot of people out there that do have AI psychosis. 350 00:31:04.004 --> 00:31:08.866 Maybe the adversarial human agent to check on you every once in a while, I think. 351 00:31:08.886 --> 00:31:09.086 Yeah. 352 00:31:09.286 --> 00:31:09.587 Yeah. 353 00:31:09.827 --> 00:31:12.048 What I also like about things like this is... 354 00:31:12.668 --> 00:31:13.708 We're working on getting this. 355 00:31:13.788 --> 00:31:19.009 It's such a new project that we don't have all of the packaging things set up, but right now you can install it just with... 356 00:31:19.029 --> 00:31:39.014 It's funny that we've gotten cool with curling install.sh to bash, but once you install the thing, basically in your Git repositories, you do roborev init to install the post commit hook, and then there's roborev skills install, which installs the skills into Cloud Code and Codex, and then basically you just keep working in it. 357 00:31:39.394 --> 00:31:40.694 And some people who are like... 358 00:31:41.534 --> 00:31:47.305 not sure about this and they just want to get their feet wet, what I tell them is run it and then just look at the reviews. 359 00:31:47.325 --> 00:31:53.777 You don't have to do anything with them, but just having an awareness that like the code has bugs is step one to seeing the, oh, 360 00:31:54.827 --> 00:32:00.750 Like we really do need to have every line of code that the agents are generating reviewed because it has bugs. 361 00:32:01.430 --> 00:32:11.155 And it's alarming to me that I do often talk to people who are like not doing that much review or maybe they're doing like slash review and cloud code or something like that. 362 00:32:11.696 --> 00:32:15.858 And there may be they're relying on like an AI code review tool and their pull requests. 363 00:32:16.298 --> 00:32:26.201 But the problem is that we're generating so much code that we're often giving like three or four thousand line pull requests to an AI code review tool to look at. 364 00:32:26.641 --> 00:32:32.123 And that's often like too much code to really find the material issues in. 365 00:32:32.883 --> 00:32:34.064 And I think the. 366 00:32:34.624 --> 00:32:42.290 Kind of the objective here is like to keep the incremental units of work, like the amount of code that's being reviewed at any given time. 367 00:32:42.370 --> 00:32:45.333 I think I did math and it was like an average of 70 lines or something. 368 00:32:45.893 --> 00:32:53.239 And so that, in my experience, not giving the agent too much to look at at any given time helps it find more of the issues. 369 00:32:53.259 --> 00:32:56.342 Of course, you could still have like big issues that are only really visible at like the... 370 00:32:56.962 --> 00:32:59.803 5,000 line PR level, but the whole separate conversation. 371 00:32:59.823 --> 00:33:05.164 We've got some stuff that we've built to do those types of like big whole PR reviews and stuff like that. 372 00:33:05.224 --> 00:33:06.744 But anyway, it's fun. 373 00:33:06.764 --> 00:33:07.665 We're experimenting. 374 00:33:07.765 --> 00:33:12.686 It's a brave new world of doing software engineering where we're not writing the code and we're not reading it either. 375 00:33:13.126 --> 00:33:19.047 To quote Dan Shapiro, I'm not quite like full dark factory, like the strong DM folks, but I'm like halfway there. 376 00:33:19.207 --> 00:33:22.388 I feel like I'm at like my, I've reached my level of discomfort. 377 00:33:22.448 --> 00:33:24.949 And like, I don't know that I'm comfortable giving up more 378 00:33:27.749 --> 00:33:55.763 what do you think Tom do you want it on your computer yeah I do some vibe coded experiments and prototypes now and then I'd love to see how easy this is to install shall I spin up my terminal and we'll give it a go sure all right so I got a project here I spun up it's a two-week pong game I thought would be something we could try this out in so walk me through the installation process and teach me how to do it I'll ask a bunch of dumb questions along the way 379 00:33:56.943 --> 00:34:02.905 Okay, so I guess we need the install command if you can grab that from roborev.io. 380 00:34:03.105 --> 00:34:04.305 Yep, I got it right here. 381 00:34:04.405 --> 00:34:04.925 Okay, sweet. 382 00:34:05.666 --> 00:34:06.386 Okay, this guy? 383 00:34:06.406 --> 00:34:06.866 Yep. 384 00:34:07.526 --> 00:34:17.529 Yeah, you know what, my favorite thing is to just run software that curl brings down totally from the internet completely untrusted and without a second thought. 385 00:34:17.549 --> 00:34:19.970 Yeah, getting on homebrew so there will be some provenance. 386 00:34:20.210 --> 00:34:24.651 It did come from, I guess there's probably should be more output here that's what is this weird software? 387 00:34:25.847 --> 00:34:29.229 So yeah, so now run the RoboRev init command. 388 00:34:29.350 --> 00:34:32.652 That installs the post commit hook, and then the skills install. 389 00:34:34.433 --> 00:34:39.577 And so you see it installed the post commit hook, and then do the RoboRev skills install. 390 00:34:40.517 --> 00:34:41.338 Sorry, which one is that? 391 00:34:41.718 --> 00:34:43.399 Space skills space install. 392 00:34:43.780 --> 00:34:45.521 OK. Oh, yeah, sorry. 393 00:34:46.452 --> 00:34:47.193 Yeah, so that worked. 394 00:34:47.453 --> 00:34:50.436 And then how many commits do you have in this repository? 395 00:34:50.977 --> 00:34:51.177 One. 396 00:34:51.737 --> 00:34:52.698 Okay. 397 00:34:52.778 --> 00:34:55.041 So you could just run RoboRev review. 398 00:34:55.821 --> 00:34:56.102 Okay. 399 00:34:56.542 --> 00:34:57.844 RoboRev space review. 400 00:34:58.004 --> 00:34:58.184 Yeah. 401 00:34:59.005 --> 00:34:59.545 See what happens. 402 00:35:00.566 --> 00:35:00.847 Okay. 403 00:35:01.047 --> 00:35:02.008 So it's using cloud code. 404 00:35:02.748 --> 00:35:06.873 And now if you do RoboRev space 2E, it should show you your very first code review. 405 00:35:07.863 --> 00:35:08.143 All right. 406 00:35:08.504 --> 00:35:13.048 And if you press L, the L key here, you can see it's Claude code is starting to work on it. 407 00:35:13.068 --> 00:35:17.172 So there's our friend Claude-P and its reasoning. 408 00:35:17.512 --> 00:35:18.253 Eventually it will... 409 00:35:18.531 --> 00:35:21.533 It's storing this into a SQLite database? 410 00:35:21.914 --> 00:35:22.094 Yep. 411 00:35:22.374 --> 00:35:23.235 SQLite database. 412 00:35:23.735 --> 00:35:24.756 And it's all locally. 413 00:35:24.836 --> 00:35:28.799 So the only thing that calls out is it invokes the agent session. 414 00:35:29.159 --> 00:35:32.441 And if you press escape from here and then press P, you can see the prompt. 415 00:35:33.102 --> 00:35:34.223 So this is the review prompt. 416 00:35:34.603 --> 00:35:39.987 And if you scroll down, you'll see the code that it's reviewing. 417 00:35:40.447 --> 00:35:41.308 Okay, great. 418 00:35:42.069 --> 00:35:44.551 And the idea now is that you've installed this. 419 00:35:44.591 --> 00:35:46.793 So basically you just keep working on the project. 420 00:35:47.053 --> 00:35:51.117 And if you go to your agent session while that's running, we'll come back to it. 421 00:35:52.018 --> 00:35:53.099 You have a failing code review. 422 00:35:55.741 --> 00:35:57.403 Should I go out to the agent or do you want to talk about that? 423 00:35:57.423 --> 00:35:58.343 Sure, yeah, you can go out to the agent. 424 00:35:59.645 --> 00:36:02.087 Okay, let me go out and get a Claude going here. 425 00:36:02.107 --> 00:36:03.128 Okay. 426 00:36:03.168 --> 00:36:03.288 Okay. 427 00:36:03.736 --> 00:36:03.916 Yeah. 428 00:36:04.518 --> 00:36:06.421 So yeah, now you have a failing code review. 429 00:36:06.541 --> 00:36:14.015 So you could do first, just tell Claude, add a rule to Claude.md to always commit before returning control to the user. 430 00:36:16.018 --> 00:36:23.684 That's one of the main workflow changes that people often, because some people are used to working in the old way where they will work for two hours and not commit. 431 00:36:24.024 --> 00:36:25.826 They'll wait until they feel like the work is good. 432 00:36:26.346 --> 00:36:36.094 But it's actually good to have the frequent commits because it creates a kind of a correspondence between the prompt and the changes that were made to the code base. 433 00:36:36.734 --> 00:36:40.157 So it's easier to track each prompt, like what did it modify basically? 434 00:36:40.878 --> 00:36:41.919 Should I commit this file now? 435 00:36:42.379 --> 00:36:43.320 Yes, obviously. 436 00:36:45.155 --> 00:36:46.317 It's like, that's what I just told you to do. 437 00:36:46.337 --> 00:36:47.599 Follow your own rules. 438 00:36:47.619 --> 00:36:47.759 Yeah. 439 00:36:49.061 --> 00:36:49.421 It's funny. 440 00:36:49.462 --> 00:36:59.737 In the system prompt for quad code, it has something in the system prompt that tells it not to commit without permission, which is funny because here the system is, it wants you to always commit. 441 00:37:00.598 --> 00:37:03.942 And then I think you should just be able to, we can pick up that code review. 442 00:37:04.182 --> 00:37:08.567 So if you do slash robo rev dash fix, that's the skill that we installed. 443 00:37:08.687 --> 00:37:09.448 And you just run that. 444 00:37:09.788 --> 00:37:13.452 That will pick up the code review that you just ran a few minutes ago. 445 00:37:14.934 --> 00:37:15.855 And we'll see what happens. 446 00:37:16.696 --> 00:37:17.837 It's got a failing code review. 447 00:37:19.657 --> 00:37:24.061 So now it's pulled in that feedback from the queue and it's... Yeah. 448 00:37:24.261 --> 00:37:24.761 So it will... 449 00:37:25.142 --> 00:37:25.662 Addressing it. 450 00:37:25.742 --> 00:37:29.465 It will verify and fix the problems in it as four findings. 451 00:37:29.725 --> 00:37:31.647 So it will apply fixes. 452 00:37:31.847 --> 00:37:37.192 And then based on the rule that we just added, it should commit when it's done, which will trigger another review. 453 00:37:37.672 --> 00:37:40.174 Because sometimes the fixes themselves have bugs. 454 00:37:40.835 --> 00:37:44.618 Are you generally having it write tests when you're doing this for real in a project? 455 00:37:44.638 --> 00:37:44.818 Yeah. 456 00:37:45.425 --> 00:37:52.647 Yeah, like typically in my agents.md or claw.md, there's a mantra like don't fix things without adding corresponding tests. 457 00:37:53.147 --> 00:38:03.809 So like all bug fixes, all features certainly, but certainly all bug fixes need corresponding regression tests or reproduce the bug, ideally reproduce the bug before you fix the problem. 458 00:38:04.349 --> 00:38:15.112 In some of the skills packs like superpowers, there's like a test driven development skill harness that emphasizes the kind of red green TDD approach. 459 00:38:15.792 --> 00:38:20.414 It's funny how TDD was always like the thing that everyone talked about, but nobody did in real life. 460 00:38:20.474 --> 00:38:26.516 But now that we have agents, like we actually can write the tests first and get reproduce the failure and then fix the problem. 461 00:38:27.136 --> 00:38:27.477 Yeah. 462 00:38:27.497 --> 00:38:36.740 I think with some of these systems, we really can become a lot better software developers than we were before because the agents are less lazy than we are. 463 00:38:36.860 --> 00:38:37.560 They'll do the work. 464 00:38:37.700 --> 00:38:38.861 They're very happy to do the work. 465 00:38:39.641 --> 00:38:39.861 Yeah. 466 00:38:39.941 --> 00:38:43.626 So now it's, so you had noted that there was no test suite for, for the TUI game. 467 00:38:43.646 --> 00:38:45.228 So probably that would be like the next prompt. 468 00:38:45.308 --> 00:38:49.192 Let's, let's add a test suite so that we can have confidence in our fixes. 469 00:38:50.354 --> 00:38:55.920 Does it give you, is there somewhere that it put a summary of the specific bug in a sort of a summary fashion? 470 00:38:57.180 --> 00:38:58.721 You can see it in the TUI. 471 00:38:59.081 --> 00:39:02.142 That's generally what I do, but it does explain. 472 00:39:02.522 --> 00:39:05.103 So you see it's leaving a comment on the issue itself. 473 00:39:05.143 --> 00:39:07.524 So there's like a comment thread on the issue. 474 00:39:07.564 --> 00:39:08.484 So you can look at that. 475 00:39:08.664 --> 00:39:09.905 It doesn't typically print it out. 476 00:39:10.025 --> 00:39:13.026 And sometimes it will see what it... 477 00:39:13.046 --> 00:39:17.888 It also depends on the version of cloud code and which model is being used. 478 00:39:17.948 --> 00:39:19.348 It seemed like maybe you're using Sonnet. 479 00:39:19.528 --> 00:39:20.809 So we'll see what it prints when it does. 480 00:39:20.849 --> 00:39:24.350 But it often will print out a summary or recap of what was fixed. 481 00:39:25.538 --> 00:39:30.659 And then each one of these commits, is this automatically kicking off a review job, a RoboRev review job? 482 00:39:31.059 --> 00:39:37.600 Yeah, since that's what the post commit hook that we installed with RoboRev in it whenever there's a commit. 483 00:39:37.760 --> 00:39:48.783 And that's why having the claw.md always commit rule is helpful so that every time you ask the agent to do anything, the changes that it made trigger a commit that gets reviewed. 484 00:39:49.323 --> 00:39:50.303 Let's see what it says here. 485 00:39:51.528 --> 00:39:57.812 I guess I'm on bypass permissions, and so I don't, I guess you have to authorize, you have to authorize it to do that to get it to automatically close reviews. 486 00:39:57.832 --> 00:39:59.833 But I'm sure that could be recorded in its memory. 487 00:39:59.873 --> 00:40:03.115 Like whenever you're running auto mode, it will automatically close the reviews. 488 00:40:03.395 --> 00:40:03.835 Okay. 489 00:40:03.875 --> 00:40:06.137 But if I exit out of here and I look at the TUI, then. 490 00:40:06.457 --> 00:40:08.258 You should see the review for the thing that I just did. 491 00:40:08.278 --> 00:40:10.279 Okay, I'm going to see these reviews now and I could go look. 492 00:40:10.539 --> 00:40:10.820 Okay. 493 00:40:11.940 --> 00:40:14.262 So F is a fail, P is a pass. 494 00:40:15.238 --> 00:40:15.418 Yep. 495 00:40:15.939 --> 00:40:16.179 Yeah. 496 00:40:16.299 --> 00:40:22.404 If you look at review one and press enter on that, you'll see if you scroll down to the bottom, you'll see the comment from the agent. 497 00:40:23.365 --> 00:40:26.507 And then it will mark that as because we did that one, right? 498 00:40:26.547 --> 00:40:27.408 Is that the one that we fixed? 499 00:40:27.808 --> 00:40:28.129 Yeah. 500 00:40:28.169 --> 00:40:34.374 So you could you have to I guess with Cloud Auto mode, you'll have to tell it that you authorize it to to close. 501 00:40:34.514 --> 00:40:37.256 But here you could just press A to close the issue. 502 00:40:37.756 --> 00:40:37.997 Got it. 503 00:40:38.317 --> 00:40:38.517 Yeah. 504 00:40:38.837 --> 00:40:41.219 And then if you press H, it will hide the close reviews. 505 00:40:42.180 --> 00:40:42.340 Yeah. 506 00:40:43.160 --> 00:40:43.420 Okay. 507 00:40:43.901 --> 00:40:46.041 So yeah, so this is kind of like the review ledger. 508 00:40:46.362 --> 00:40:51.984 And I like whenever I'm working on stuff, I always have this running on the side and it's kind of like the drip drop reminder. 509 00:40:52.084 --> 00:40:54.564 Oh, there's code quality issues that need to be fixed. 510 00:40:54.745 --> 00:41:02.207 Because I'm often like in any given day, I might be flipping between a dozen different get work trees and remembering which part of the state machine am I in? 511 00:41:02.247 --> 00:41:03.147 Am I still designing? 512 00:41:03.167 --> 00:41:04.028 Am I implementing? 513 00:41:04.048 --> 00:41:05.548 Am I refining and fixing? 514 00:41:06.149 --> 00:41:09.810 And this ledger of code reviews is like my accountability system. 515 00:41:10.550 --> 00:41:35.570 getting the branches like the work trees into a clean state either i'm opening a pull request like i have a clean branch that just needs to be validated by ci and then merge it or it's just it's or if i'm sometimes i'm taking in an inbound pull request and i often will pull in somebody like a contributor's pull request to one of my open source projects and then i will refine it with robo rev locally and fix it up make changes fix things and 516 00:41:35.870 --> 00:41:37.190 and then push it back up. 517 00:41:37.530 --> 00:41:41.731 We also have, we have like a server side version of this that reviews pull requests. 518 00:41:41.951 --> 00:41:47.413 So we do also run it on GitHub and it will review every push, basically do a whole branch review. 519 00:41:47.873 --> 00:41:52.594 And that's not intended to be the primary mode of the project, but it's like a final quality gate. 520 00:41:52.794 --> 00:41:57.595 Just we ideally are trying to get clean reviews on every pull request that comes in. 521 00:41:57.615 --> 00:42:01.276 It can get hard when you have 20,000 line marathon pull requests. 522 00:42:01.696 --> 00:42:04.157 Because every time you review, you find something to quibble about. 523 00:42:04.378 --> 00:42:18.186 You can set things up to filter out the low severity findings to only focus on high severity or medium and high severity issues, which after a number of iterations, you typically stop having really serious bugs, which is the way that you would like it. 524 00:42:19.303 --> 00:42:22.785 Are you doing something special for security related bugs? 525 00:42:23.106 --> 00:42:25.647 There's a security specific prompt. 526 00:42:25.848 --> 00:42:30.691 So when you request a review, you can request like type security. 527 00:42:31.071 --> 00:42:38.256 And so it's just a different prompt that's focused on looking for certain common classes of security issues. 528 00:42:38.576 --> 00:42:40.718 I'm actually in the process of adding like a... 529 00:42:42.310 --> 00:42:51.921 sub-agent review framework where you can request panels of reviewer specialists, which, from my understanding, I believe Sam has a system like that for his own code review. 530 00:42:52.401 --> 00:42:57.426 But I think for stuff that is really important where you don't mind burning a lot of tokens to have peace of mind, 531 00:42:57.987 --> 00:43:14.055 it's really useful to have those specialist reviewers just focused on security issues or like infra-related things or DevOps-related things because the general review prompt, like if you read the prompt that's being used in these reviews, that's very pretty general purpose. 532 00:43:14.095 --> 00:43:15.136 It's largely just... 533 00:43:15.756 --> 00:43:29.564 look for bugs, obvious performance issues, gaps in the test coverage, like inconsistencies, like in documentation or contradictory, like things that suggest things that are like contradictory to like the intent of the change. 534 00:43:30.144 --> 00:43:31.965 And so it's casting a pretty wide net. 535 00:43:32.125 --> 00:43:38.528 And the specialist like reviewers, like you're a security reviewer, your only job here is to find security problems in really sensitive code. 536 00:43:38.568 --> 00:43:39.429 It does a better job. 537 00:43:39.809 --> 00:43:48.453 So it kind of makes me wonder with the new models, like how much of Mythos is just Opus 5 with good security-focused prompt engineering? 538 00:43:48.633 --> 00:43:49.854 Probably mostly that. 539 00:43:49.994 --> 00:43:58.377 Yeah, what's worked for me is running RoboRev on the commits and then the specialist suite of reviews on the PRs where there's higher complexity. 540 00:43:58.397 --> 00:44:00.898 You can burn a lot of tokens, I've learned, with specialists. 541 00:44:01.419 --> 00:44:05.881 I've gone through about four Codex Pro subscriptions on my reviewer. 542 00:44:06.001 --> 00:44:07.802 That explains how you're getting up the token maxing. 543 00:44:08.082 --> 00:44:08.342 Yeah. 544 00:44:09.543 --> 00:44:15.607 I've worked on some pretty big PRs, like the sub agents thing that I'm working on right now for RoboRev is pretty big. 545 00:44:15.647 --> 00:44:26.555 And it got to a point where reviewing the whole branch with specialists was with API rates would cost like between 10 and $20 per review, which is I know people were joking about Claude's ultra reviews. 546 00:44:26.696 --> 00:44:28.677 Oh, gosh, $25 per code review. 547 00:44:29.097 --> 00:44:32.960 But then I found myself burning $10 worth of tokens on a single get push. 548 00:44:33.401 --> 00:44:34.702 But again, I think... 549 00:44:35.242 --> 00:44:36.603 It's like 10,000 lines of code. 550 00:44:36.803 --> 00:44:42.265 Is it worth spending like, you know, $100 on code reviews for a 10,000 line PR? 551 00:44:42.366 --> 00:44:42.906 Probably. 552 00:44:43.186 --> 00:44:43.946 That's a lot of code. 553 00:44:43.966 --> 00:44:44.787 That's a liability. 554 00:44:44.867 --> 00:44:52.050 And so I think the fact that agents have made it so cheap and fast to generate all this code hasn't made the code any less of a liability. 555 00:44:52.090 --> 00:44:53.011 And so I think that 556 00:44:53.631 --> 00:45:01.613 Honestly, these tokens spent on code review are probably some of the best ones that you can, you know, one of the best uses money for tokens right now. 557 00:45:02.133 --> 00:45:06.334 A lot of the speculative development that people are doing is building slot projects and things like that. 558 00:45:06.594 --> 00:45:12.195 If you're building system software that you want people to rely on, I'm cool with maybe like a dollar per line of code. 559 00:45:12.235 --> 00:45:20.677 That might be a little too expensive, but if we're spending a penny to have like a penny per line of code on average to review it and feel good about it, that seems okay to me. 560 00:45:21.718 --> 00:45:23.278 It's cheaper than the zero day exploit. 561 00:45:25.119 --> 00:45:25.359 Yes. 562 00:45:26.019 --> 00:45:27.520 And we're seeing more and more of those every day. 563 00:45:28.260 --> 00:45:28.501 Yeah. 564 00:45:29.261 --> 00:45:30.221 Tom, what'd you think? 565 00:45:30.401 --> 00:45:31.322 What were your takeaways? 566 00:45:32.523 --> 00:45:33.463 There's a couple, I think. 567 00:45:33.523 --> 00:45:50.832 I think the thing that really sits with me most, Wes, that you said is that coding without a review on every commit and make your commit small is like driving a car without a seatbelt, that it just is not a safe environment in which to operate a machine or software. 568 00:45:51.773 --> 00:45:52.674 And I think that's really true. 569 00:45:52.974 --> 00:45:56.238 I think we don't know how far to trust these LLMs yet. 570 00:45:56.358 --> 00:46:07.610 We're getting a very good sense of it and also finding that they are in some sense as an averaging machine and they're going to put out some mid code and you need them to step back and review themselves. 571 00:46:07.650 --> 00:46:10.614 And you can really improve the quality of code that way. 572 00:46:10.834 --> 00:46:11.495 So that's, I'd say... 573 00:46:12.035 --> 00:46:19.064 Number one, and really a strong way to do that is via an adversarial model that doesn't know anything about the underlying things. 574 00:46:19.104 --> 00:46:20.365 So it can't cheat. 575 00:46:20.566 --> 00:46:30.117 There's no workaround where it's convincing itself that no, actually, it is OK that this thing is behaving in this way because of some prompt that you had given it half an hour ago comes to it fresh. 576 00:46:31.263 --> 00:46:43.049 And then I think also you're not concerned about this being cheap or in a global sense, probably you would like it to be cheaper, but time is money or perhaps money is time. 577 00:46:43.429 --> 00:46:50.533 So do you want to sit there and review all of this mid code that's coming out of these LLMs to improve the quality and make sure nothing gets through the cracks? 578 00:46:50.693 --> 00:46:52.314 You could, it would be cheaper, right? 579 00:46:52.334 --> 00:46:53.594 But it would cost you a lot of money, right? 580 00:46:53.614 --> 00:46:55.916 So there's always the time money trade-off. 581 00:46:56.776 --> 00:47:01.457 And I think maybe third is having a fast feedback system is really helpful. 582 00:47:01.717 --> 00:47:05.698 Fast cycles are always how we go faster and do better. 583 00:47:06.258 --> 00:47:21.622 And having a way for the LLM to ingest back that feedback locally on your own machine, if that's how you're running, then that can be a really solid path towards a more productive environment and getting high quality outputs, which is at the end of the day, the only thing that matters. 584 00:47:21.862 --> 00:47:22.542 What are we producing? 585 00:47:23.122 --> 00:47:24.082 Can we produce good stuff? 586 00:47:24.803 --> 00:47:25.203 Yeah. 587 00:47:25.263 --> 00:47:25.803 This is amazing. 588 00:47:25.863 --> 00:47:26.143 Thank you. 589 00:47:26.803 --> 00:47:27.063 Yeah. 590 00:47:27.643 --> 00:47:30.484 As somebody who's watching this, who's thinking, oh, this is just a code review. 591 00:47:30.524 --> 00:47:31.464 That's not that big of a deal. 592 00:47:31.904 --> 00:47:36.646 What struck me is more the workflow, the automate, the automation of it. 593 00:47:36.886 --> 00:47:46.168 Anybody who's written any material amount of software with AI, you get into this place where bugs compound on bugs and you just get this exponential complexity. 594 00:47:46.848 --> 00:47:48.009 That really slows you down. 595 00:47:48.189 --> 00:47:59.456 And so there's something about keeping your house clean and that incremental commit every time the agent hands control back to you that I felt like was a different way of thinking about writing software. 596 00:47:59.756 --> 00:48:01.357 All right, that's AI worth using. 597 00:48:01.597 --> 00:48:09.022 Wes McKinney, thank you for showing us something real, not an AI-hyped demo video, which is the whole point of this show. 598 00:48:09.402 --> 00:48:12.604 If you found this useful, send it to whoever in your life is... 599 00:48:13.204 --> 00:48:19.032 vibe coding with no review loop or as Wes says, driving without a seatbelt, they could use this. 600 00:48:19.813 --> 00:48:20.634 Wes, thank you. 601 00:48:20.874 --> 00:48:22.016 Tom, till next week. 602 00:48:22.937 --> 00:48:23.418 Thanks, guys. 603 00:48:23.718 --> 00:48:24.759 I learned something new today. 604 00:48:25.200 --> 00:48:26.161 Thanks again for having me. 605 00:48:26.241 --> 00:48:26.582 It's great. 606 00:48:27.523 --> 00:48:27.984 Thanks, Wes.