I recently sat down with Jane King of LilaMax Media to talk about what happens after AI writes the code. You can watch it below. A short recap follows, and the full transcript is at the end.
Code and production are two different worlds
Software lives in two realms: the code, and how that code behaves in production. The models behind coding agents learned from the first. The second is a different kind of data, and agents mostly don’t have it. As writing code gets cheaper, the bottleneck moves to shipping changes without hurting stability.
The shaky finger syndrome
Someone still has to be accountable for what ships. Today that’s an engineer pushing code an agent wrote, code they don’t know well. One of our customers calls the hesitation that comes with it the shaky finger syndrome. The name stuck because everyone got it right away.
Giving agents runtime context
Our approach at Hud is to give coding agents runtime context: how each function behaves in production, and a detailed record when something goes wrong. Coverage matters here. Sampling keeps costs down, but the rare events behind incidents tend to be the ones that get sampled out.
Where this is heading
I expect coding agents to work in a continuous feedback loop: learn from what happens after a change ships, and share responsibility for the health of the system they ship into. Code review is already changing. Monitoring and reliability are next. If we get this right, teams won’t have to trade velocity for reliability.
Full transcript
Jane King: Hud is a software startup that helps AI coding tools and human engineers see how their software is actually running in the real world. So when AI tools write code, they’re often guessing how that code will behave once it is launched. That could lead to bugs and crashes and software that doesn’t work in the real world. And Hud solves that problem. So with me is Roee Adler, the CEO and co-founder of Hud. So let’s talk about AI, because it’s just a fascinating, transformational technology. And it seems like more and more AI agents are writing and planning and doing the testing, all the software. But what do you think is still missing when it comes to AI and software?
Roee Adler: Software engineering world is divided into two realms. There’s the realm of code, and then there’s the realm in which the code actually behaves in production. Now, LLMs, the platforms on top of which AI coding agents were built, were trained on mountains of code, but the information related to how that code behaves in reality is just a different kind of data. Now, Gartner projects that 70% of software engineers in enterprises are going to be basing their work on AI in, say, a year or two. We think it’s lowballing, actually. But this advancement has come at a pretty terrible price, where the data is showing that about 78% of teams are saying their incidents increased in production, having adopted AI coding agents. And even Google’s DORA metrics say that as individual velocity goes up, so does system instability, which is a pretty interesting trade-off to be able to deal with. Somewhere between code review and incident response lies a world that needs to be reinvented for the age of AI, so that software engineers can wake up in the morning, manage armies of AI coding agents, but ship with confidence and not have this fear in their hearts and the shaky finger syndrome, as some of them are calling it, to ship this code that they aren’t deeply familiar with.
Jane King: Interesting. I’ve not heard of that, the shaky finger syndrome. Was that just like they’re not sure exactly how they’re going to behave in the real world?
Roee Adler: At the end of the day, someone needs to be accountable for the code that’s being shipped, right? So in the past, you know, someone made a terrible mistake. You knew who that person was. They could show remorse. Maybe they got fired. But today, if, you know, Anthropic’s agents write code that breaks my system, will they reimburse me for the tokens? Who’s accountable? So someone, a human, has to decide that they are shipping this code into production. And that’s where the… We heard, by the way, we heard from one of our customers use the term the shaky finger syndrome. So it was so powerful because everyone just got it, right? It’s, how can I take responsibility for code that I’m not deeply, intimately familiar with, but I’m going to be blamed if something breaks.
Jane King: Absolutely. No, those are all very big questions, a niche kind of solution. Now, you described the PR as the point where today’s AI SDLC effectively stops. So why is the gap between code that looks correct and code that behaves correctly in production becoming more important? I think we kind of touched on this a little bit, but could you expand on that?
Roee Adler: Sure. So code, in essence, it’s a set of instructions that are supposed to achieve some goal. You can think about it as a map. But then you put this code under real, actual conditions. Could be tens of thousands of pods running concurrently, databases that have various issues that emerge over time. It’s more like a live traffic situation. And, you know, we used to navigate with paper maps. And now the navigation is based on a traffic light that’s broken somewhere or an accident that occurred. Everything changes once you have live data. It’s just a fundamentally different kind of information.
Jane King: Yeah. So as engineering teams adopt AI coding agents at scale, do you expect the biggest challenge to shift from writing the code faster to understanding the real-world impact of the code?
Roee Adler: I think we’re already there. The writing code faster with agents has already been commoditized. Now, there’s no question it’s going to improve in amazing ways, but there are already, you know, a bunch of providers, amazing companies, luminaries in their field, and most software engineers are already basing their work on using one of those agents. The bottleneck has shifted in the past year from how fast am I as an engineer, to how quickly can my, can my company ship code without jeopardizing stability.
Jane King: So Hud’s approach is built around bringing the runtime context into the development process. So what does runtime context give an AI agent that codebase context and organizational knowledge does not?
Roee Adler: Sure. At Hud, we built a runtime code sensor, which is a technology that is based on two main pillars. One is function-level behavioral context, so the coding agent understands how every function behaves in production as it is reasoning over code. And the second is what we call a forensic engine. So when something goes wrong, it could be related to errors or slowdowns or CPU or memory, then there’s a very deep forensic document coming from production describing exactly what happened. This ubiquity and depth of information allows the coding agent to reason immediately, very quickly, against every piece of change that it is doing on the code.
Jane King: So one of Hud’s differentiators is its ability to observe the behavior at the function and invocation level without sampling. So why is that level of visibility necessary?
Roee Adler: Sampling is a crucial point to understand, because when you have a company operating at scale, like many of our customers, you can have billions and billions of events every day. And you can’t simply gather information about all of them and bring it to a huge data set. That would be ridiculously expensive and slow things down. So what companies usually do is they sample in various ways and only bring in small subsets of the data. But when something goes wrong, it’s always on the margins. It’s always this rare occasion that is always sampled out. And a lot of our IP is about being there, living and running together with the code, not missing anything, but also not sending everything. So being able to understand where an issue becomes something that impacts the business in the running software, gather the forensic information about it, and always send it back to the coding agent.
Jane King: Okay. And then also, Hud doesn’t simply monitor an application after deployment. It can gate changes based on production risk and verify the releases as they roll out. So how does that change what traditionally used to be done with deployment?
Roee Adler: Sure. Well, historically speaking, you’d have engineers write code and then other engineers or different processes do testing, then things would be deployed, and then usually other teams would monitor against incidents. And the gate between those has been a barrier that has existed for many, many years. Right now, I think many companies are trying to use the same pattern with AI coding agents, but we don’t think this is going to carry us into the future. We think coding agents need an iterative, continuous feedback loop from the code that they are shipping into their own behavior. So what happens is, whatever we can catch before a PR is shipped to production, we should. But sometimes the only way to catch an issue is when it hits the reality. And there we need to very quickly detect it, revert it, create the underlying fix, but then create a learning feedback loop so that the agent understands what went wrong, so that next time it will do a better job.
Jane King: Interesting. And your platform compares the rollout against its own production baseline and can roll back regressions before the blast radius grows. So what kinds of problems can this catch? And I assume that’s better than how we’ve seen things traditionally as well.
Roee Adler: Baselines are a critical component, I think, because historically, I could only know what I chose to put probes against. So there’s this pattern of an incident occurs and someone would say, “Oh God, if we only configured this alert, or if we only put this log in, we wouldn’t have gotten to this incident phase.” At Hud, we believe that AI coding agents need ubiquitous visibility into what’s going on, and this is why we believe that constantly comparing behavior against baselines of how the system currently behaves is the key to be able to have all this information without anyone needing to configure anything.
Jane King: Yeah. And so the forensics are an important part of this, and the constant kind of improvement, it sounds like. Yeah. And that’s what differentiates you.
Roee Adler: Yeah. That’s another critical component, because when things go wrong today, you usually have people trying to sift through different kinds of data sets. So you have logs and you have traces and you have the error monitors, and you’re trying to reconstruct what actually happened in production in order to understand what’s going on. And at Hud, part of our core beliefs is that people shouldn’t do sort of investigative police work to try to understand what happened at the scene of the crime. They should just have the information directly from there, which is exactly the forensic extension.
Jane King: Okay. Now, you’ve described Hud as providing the runtime context that AI agents need. So what does the software development look like when the AI agent not only can write the code, but also understands how it behaves in production?
Roee Adler: That would be the dream, right? It’s the dream of a continuous feedback loop where AI coding agents are not only accountable on producing code that’s a candidate to be merged into production, but they are also accountable for the health and stability and reliability of the system that they are shipping code into. And that would be a magical day.
Jane King: Okay, so that’s like the Holy Grail of what you’re looking for. Now, Hud can capture execution paths, parameters, dependencies, machine state when an issue occurs. So how do you think that this changes the relationship between the AI coding systems and the production systems?
Roee Adler: I think it goes back to who is accountable for the business operating in a healthy way at the level of reliability that’s expected. Today, it is definitely not the AI coding agents. And today, it’s not even easy to answer this question at all. We think that once we are able to bring this runtime context into the continuous workflow of AI coding agents, the question of who’s accountable for the health becomes resolved, which we think is exactly what is needed to balance between velocity and reliability and remove fear from the hearts of engineering organizations.
Jane King: So now, if AI continues to dramatically increase the amount of code that’s being produced, what parts of the traditional software workflow do you think will be obsolete first?
Roee Adler: It’s a fascinating prediction to make, right? And I think smart people are making some bold predictions or assumptions. I think that there’s no question the code review realm is going to change. It already is. The writing of software, of course, is already being revolutionized, because almost no one’s using IDEs anymore. They are not looking at code as much as they did in the writing of it. I think that the health, reliability, monitoring, stability, these are the next areas that need to be completely rethought and revolutionized as a result of those changes.
Jane King: Okay. And then finally, if Hud succeeds in closing the gap between AI-generated code and real-world production behavior, what fundamentally changes when it comes to engineering organizations? What becomes possible for the consumer and for the companies that is not possible today?
Roee Adler: Engineering leaders all over the world are trying to harness the power of the AI revolution in order to move with much, much higher velocity. That’s part of the goal. But right now, it’s impossible to do that without jeopardizing stability and reliability. We are hoping to look at these two as sort of yin and yang, to build this infrastructure, this platform where the more you increase velocity, you actually increase reliability. And that would, I think, unlock the power of AI code generation into engineering organizations at a much, much different level.
Jane King: Okay, well, it’s fascinating. And you’re right on the front lines of this.
Roee Adler: We feel that, we feel that.
Jane King: Yeah. Roee, thank you so much for coming and sharing the story of Hud.
Roee Adler: Thank you for having me.
Jane King: Thank you.
About authors
