Translation: Why Large Language Models Can't Truly Build Software Author: Conrad Irwin One thing I've spent a lot of time doing is interviewing software engineers. This is obviously a daunting task, and I wouldn't say I have any special tricks; but this experience has indeed given me time to reflect on what an efficient software engineer actually does. The Core Loop of Software Engineering When you observe a true expert, you'll find that they are always cycling through the following steps: * Building a mental model of the requirements. * Writing (hopefully?!) code that meets those requirements. * Building a mental model of the actual behavior of the code. * Identifying the differences between the two and then updating the code (or the requirements). There are many ways to complete these steps, but what sets efficient engineers apart is their ability to build and maintain clear mental models. How Do Large Language Models Perform? To be fair, large language models are quite good at writing code. When you point out where the problem lies, they also do a decent job of updating the code. They can do all the things that real engineers do: read code, write and run tests, add logging, and (probably) use a debugger. But what they can't do is maintain a clear mental model. Large language models can get caught in endless confusion: they assume that the code they write actually works; when tests fail, they can only guess whether to fix the code or the tests; when they feel frustrated, they simply delete everything and start over. This is the exact opposite of the qualities I expect in an engineer. Software engineers test as they work. When tests fail, they can refer to their mental model to decide whether to fix the code or the tests, or to gather more information before making a decision. When they feel frustrated, they can seek help by communicating with others. Although they sometimes delete everything and start over, that choice comes after they have a clearer understanding of the problem. But it will be fine soon, right? Will this change as models become more capable? Maybe?? But I think it requires a fundamental change in how models are built and optimized. The models needed for software engineering are not just those that can generate code. When a person encounters a problem, they can temporarily set aside all context, focus on solving the immediate issue, and then return to their previous thoughts and the larger problem at hand. They can also switch fluidly between the macro big picture and micro details, temporarily ignoring details to focus on the whole, and dive deep into specifics when necessary. We won't become more efficient just by cramming more words into our "context window"; that will only drive us crazy. Even if we can handle vast amounts of context, we know that current generative models have several serious issues that directly affect their ability to maintain clear mental models: * Context omission: Models are not good at identifying omitted contextual information. * Recency bias: They are severely affected by recency bias when processing context windows. * Hallucination: They often "hallucinate" details that shouldn't exist. These issues may not be insurmountable; researchers are working to add memory to models so they can employ similar thinking techniques as we do. But unfortunately, as it stands, they (beyond a certain level of complexity) cannot truly understand what is happening. They cannot build software because they cannot maintain two similar "mental models" simultaneously, identify the differences, and decide whether to update the code or the requirements. So, what now? Clearly, large language models are useful for software engineers. They can quickly generate code and excel at integrating requirements and documentation. For certain tasks, this is sufficient: the requirements are clear enough, and the problems are simple enough that they can be solved in one go. That said, for any task with a bit of complexity, they cannot maintain enough context accurately to iteratively produce a viable solution. You, as a software engineer, still need to ensure that the requirements are clear and that the code truly implements the functionality it claims. At Zed, we believe that in the future, humans and AI agents can collaboratively build software. However, we firmly believe (at least for now) that you are the driver holding the steering wheel, while large language models are just another tool at your disposal.