Scientists make a vocation of understanding the world, yet we still seem to lack a sufficiently clear answer to what exactly counts as “understanding.” One of the most basic questions is this: for any given system, what would count as understanding the system as a whole, and how would we know when that understanding was complete?
You can take any system you like: the brain, society, a large language model, and so on. At exactly what point can we say, all right, we understand this system? Often, we take it for granted that our inability to answer this question comes from our inability to fully observe and intervene in these systems. In doing so, we evade the real question: even if we had godlike powers to observe every detail of any complex system and control it at will, we would still not know what the endpoint of “understanding” was.
I am a computational neuroscientist, so let us begin with the brain. We often say that the goal of neuroscience is to understand the brain. But understanding the brain is difficult, and these difficulties show up in several ways. Consider the difficulty of observing it: early studies could often record from only a very small number of neurons simultaneously, and it took many years to reach the point where we could observe large populations of neurons at once in living animals. Whether through electrophysiology or imaging, what we observe is still a drop in the ocean of the brain. When it comes to intervening in and controlling brain activity, optogenetics, which has just been recognized with a Nobel Prize, is a remarkable achievement. Using light to control neurons was once the stuff of science fiction for neuroscientists. Yet an enormous gulf remains between this and precisely controlling every neuron in the brain at the same time. As for measuring the connections between neurons, we can now reconstruct a map of synaptic connections across the entire brain of a tiny animal such as a fruit fly. But these anatomical connections do not directly tell us how much one neuron actually influences another in a living brain. As for precisely setting the strengths of the connections between neurons throughout the brain, all at once, I do not know of a technology that can do this. Perhaps one day. For all these reasons, we neuroscientists seem able to comfortably attribute our failure to understand the brain to the limitations of our measurement and control technologies.
But here is the awkward part. For another kind of neural network, a large language model, these dreams of observing and controlling a system from within have already largely become reality. For a model whose complete code and parameters are in our hands, we can record its internal activations as it runs and manipulate those activations. We know the value of every connection parameter and can change those parameters at will. So, do we understand large language models? No. More importantly, we cannot even clearly say at what point we would be entitled to declare that we understand them. In the distant future, when we can do everything to the human brain that we can do to a large language model today, will we understand the brain? Or, rather, will we know what counts as “understanding the brain”?
So, let us talk about what “understanding” means.
One very natural answer is that to understand a system is to be able to predict its behavior. If, for any given input or environment, we can always predict what it will do, that seems quite good. If we can also change some part of the system and still predict how its behavior will change, then this seems to amount to understanding. We can use this “knowledge” to “transform” the world.
But here is the problem. We can already run large language models, copy them, modify their parameters, and run them again. When the computational conditions and sources of randomness are controlled, we can reproduce the same computational process. If this kind of simulation also counts as prediction, then, by the definition above, it seems that we acquire the ability to understand a model the moment we obtain it. Why, then, do we still feel that we do not understand these models? Why do we still call them “black boxes”? What exactly are we looking for?
Perhaps we should look in another direction: what is often called mechanistic understanding. One thing science often does is reveal relationships between variables. We do not merely know that things happen; we also try to understand why they happen, how one quantity changes with another, how changing one part affects the others, and under what conditions these relationships hold. These are the kinds of things people often have in mind when they talk about mechanisms.
Let us take another example. We built computers. Do we understand them?
I think it would be hard to answer “no” in this case. We know how circuits form components that store information and perform computations, and how those components work together to execute a program, step by step. If even a system like this cannot count as understood, what, then, are we prepared to say we understand?
But let us run a large language model on this computer. Do we still understand the computer?
You might say, of course we do. We just do not understand the model it is running. That is a perfectly reasonable answer, but it also makes something clear: when we say that we understand a computer, we are not claiming to understand everything that happens inside it. We understand how it executes programs without necessarily knowing why each program produces a particular result.
How, then, does this differ from knowing how a large language model performs matrix operations or computes attention? At the level of carrying out computations, we know how it works too. But we also want to know how it uses those computations to process language. If we compare “understanding a computer” in the former sense with “understanding a large language model” in the latter, we have quietly changed our standard.
So here is the question: on what grounds do we decide that a certain kind of explanation is sufficient, while other questions may be left outside the scope of “understanding”?
Let us think a little further. Suppose we have a computer’s complete circuit diagram and can calculate, step by step, how the voltage changes at every point, but we do not yet know which circuits work together to store a number, which perform addition, and which determine what to execute next. We would still feel that some understanding was missing. We would also want to know how these details are organized and why that organization makes computation possible.
This is what we hope to gain beyond the complete details of an implementation. For many components that we explicitly design, such an explanation exists before they are built. We first want to store a number, then design a circuit to store it; we first want to perform addition, then design the corresponding structure. But for a neural network produced through training, we have not specified in advance, detail by detail, how it should internally organize its abilities to solve different problems. We obtain all its parameters, yet still need to discover which variables we should use to describe it and what relationships exist between those variables.
There is also a view that understanding a system may require only a grasp of the process that produced it. For a large language model, for example, knowing how it was trained might already be enough; how the resulting model works might not be understandable at all.
But several things need to be distinguished here. Knowing the steps of training is not the same as explaining why those steps produce a particular ability. Even if we have achieved the latter, we still need to explain why that is enough. We can acknowledge that understanding how a system is formed is itself a kind of understanding. But why would this mean that we no longer need to understand how it works once it has been formed? Some explanations of how a system is formed also reveal its working principles, but this connection needs to be made explicit. We cannot assume that other questions no longer need answers simply because we have grasped the process that produced the system. Whether the product of training can be understood at all is another question that needs an answer of its own.
We also ask how the brain is shaped through development and learning, yet this does not lead us to stop asking how it remembers or makes decisions. Why, then, are we willing to treat the process that produces some systems as the endpoint of understanding, while demanding more of our explanations of others?
Then, even if we continue to ask how a system works, how should we choose the appropriate variables?
The same string of bits may be a number stored in a register in a description of the processor, and a score assigned to a position in a description of a chess program. When we ask, “Why did this number increase?”, we may be asking which instructions changed it, or why the program judged the current position to have become more favorable. The same machine requires us to uncover different relationships at different levels of abstraction.
For the brain, we can discuss the activity of a single neuron, the collective activity of a population of neurons, or the trajectories of that activity through some space. We can also describe it in terms of “which rule is being remembered” or “which two options are being compared.” There seem to be countless possible abstractions, and they need not fall neatly into successive levels: the same population of neurons can participate in multiple processes, and different processes can overlap.
Of course, we need to test whether these variables are well-founded. But even if they all are, why should we choose some of them rather than others? Which choices would help us understand the system as a whole? And how would we know whether some abstraction we have not yet thought of might reveal relationships we have never recognized?
Moreover, even after we have chosen our variables, the questions can keep extending further. Does this relationship still hold in a different environment? What happens if we change one of the variables? What if another process takes place at the same time, or we connect this system to another? So, on the one hand, we can keep redefining our variables; on the other, even if we keep our choice of variables fixed, we can keep asking new questions.
At what point, then, should understanding be complete?
I do not think we can simply answer, “There will always be more questions, so understanding can never be complete.” The computer example has already shown us that some unknowns do not prevent us from grasping how a system works. But neither can we respond to every question we cannot answer by saying that it belongs to another level. We need to know which unknowns mean that we have not yet understood, and which concern things we can continue to study even after we have understood.
Nor do I want to reduce this problem to the limits of human memory, or to a system being too complex to simplify. Suppose the eventual explanation ran to a million pages, even longer than the original code and parameters written out in full. Suppose no single person could read it all, but the scientific community could collectively grasp it. I would accept all of that. I just want to know: what, exactly, should those million pages contain? Why would writing all of that down be enough?
If the answer is “spell out the relationships between all the important variables,” what counts as important? If it is “explain all the key mechanisms,” how do we determine which mechanisms are key? If it is “find the rules that can generate all the system’s behavior,” the original code and parameters already provide those rules. These answers all seem reasonable, yet they leave the hardest part inside the definition.
If the further reply is that explanations are always relative to particular questions, why do we choose those questions? And why do we think answering them is enough? The other questions may still be important; perhaps they are simply not being studied this time. But if so, what justifies the move from “these questions have been answered” to “this system has been understood”? We can certainly prioritize particular questions for some practical purpose. But when that purpose is itself “understanding the system as a whole,” we still need a reason for deciding which questions must be answered and which can be left open.
At least for me, what remains unresolved is this: once we possess the complete details of a system’s implementation, what further knowledge do we need before we can reasonably say that we understand it? This knowledge need not answer every possible question. But it needs to give us grounds for judging that the remaining unknowns no longer stand in the way of our grasp of how the system works as a whole.
I do not think this makes existing scientific research meaningless. Discovering a variable, revealing a relationship, or demonstrating a mechanism can certainly give us real understanding. But knowing that we understand more than we did before is still different from knowing what would count as understanding the entire system.
I do not currently have an answer that satisfies me. Perhaps understanding is always relative to particular questions, with no single endpoint. Perhaps we can find a standard that specifies what knowledge is sufficient to constitute understanding of a system as a whole. I am unwilling to conclude that the question has no answer simply because we do not have one today. But large language models have at least shown us that we cannot keep looking to ever more powerful measurement technologies to settle this question.
When we say that the goal of science is to understand the world, what kind of knowledge are we hoping to obtain? In the distant future, when we can finally observe everything that happens in the brain and alter everything that happens in it, how will we know that we have finally understood it?