← Back to Essays
Reflection

AI Is Becoming More Capable

We want it to become more understanding of us.

Every month the models get better at something. Longer context. Stronger reasoning. Better code, better retrieval, better scores on whichever benchmark the field has agreed to care about this quarter.

None of that is the same as a system understanding what matters to you.

Capability is measured against tasks. Understanding is measured against a person, in a particular moment, with particular obligations and a particular sense of what would be the wrong thing to do right now. The first is improving quickly. The second is barely being attempted.

What a full room made clearer

On October 22 we put the idea in front of about eighty people at the Crow Museum of Asian Art at UT Dallas, and watched what happened to it.

What I had not expected was how little of the evening was mine. Five UT Dallas students had spent a summer with the hypothesis and had taken it somewhere. One traced its lineage and found that situation awareness research had been circling this for forty years, and that mapping it onto language agents produces an uncomfortable paradox: the more situationally capable an agent becomes, the better it gets at gaming the tests for situational capability. Another asked whether recognizing the situation first could cut the computation rather than add to it. Two built working systems — RiskPulse, which treats a late customer, a payroll run and an unpaid supplier as one cash situation rather than three alerts, and Sentinel, which turns a fraud alert into a set of candidate situations with the evidence for and against each, and refuses to let the language model choose. A fifth stepped back and asked whether any of this amounts to a new programming paradigm at all.

An idea that only its author can hold is not yet an idea. It is a preference. The first real test is whether other people can pick it up, disagree with parts of it, and build something you would not have built. That test was the evening’s actual result, and it mattered more than the attendance.

Recognition is the start, not the end

The argument this program has been making is that intelligence begins with recognizing the situation: what is happening, why it matters, and what the moment calls for. Context supplies the information. Situation supplies the meaning.

But recognition on its own is still something done to you. A system that reads your situation and acts on its reading, with no way for you to correct it, is not more understanding. It is more confident.

Teachable

So the thing we actually want is narrower and harder: a system you can teach.

Teach it that in your business a customer going quiet for a week means something different than it does in someone else’s. Teach it which situations you want handled and which you want handed back. Correct it when it reads a moment wrongly — and have the correction persist, rather than evaporating when the conversation ends.

That is a demanding requirement. It means the system has to hold an explicit account of what it thinks is happening, show the evidence for it, and let that account be argued with. Sentinel’s design makes the point in miniature: the evidence for and against each candidate situation is on the screen, and the model that writes the explanation is not allowed to choose the answer. You can disagree with a thing like that. You cannot disagree with a black box.

And eventually, yours

There is a further ambition underneath, and the evening made me willing to say it out loud.

If a system accumulates situational experience — what happened, what it understood, what was chosen, what followed, and where it was wrong — then over years that accumulation becomes something substantial. It is a record of how you work, what you prioritize, what you consider an acceptable response. It is, in a real sense, intelligence built through your own life.

That ought to belong to you. Not to whichever vendor happened to host it.

We are a long way from being able to deliver that. But it is worth naming the direction, because it changes what you build. A system designed to be teachable and portable looks different from one designed to be impressive.

This is a beginning

Prof. Ramesh Subramoniam closed the evening by saying that this is a beginning, and he was right in the way that is slightly inconvenient to hear. We have a hypothesis, five explorations, two working demonstrations, and a long list of things we cannot yet do.

There is still no formal definition of a situation precise enough to compute with. No benchmark. No empirical result showing that an explicit situation layer beats a well-engineered agentic baseline. Every student paper says so plainly, which is the part I am most pleased about.

What exists is a question worth the work, and now a few more people willing to do it.

If you want to argue with the hypothesis, test it, or bring a real situation that breaks it — that is the invitation.