AI agent decision quality improved most for me when I stopped treating every bad output as a prompt problem and started treating repeated human intervention as evidence that the working system itself needed to learn.
For a long time, the obvious explanation for disappointing AI work was that the model was not good enough. That explanation was often correct. Models hallucinated frequently, lost context, misunderstood ordinary instructions and required enough supervision that trusting them with anything consequential felt irresponsible. Better models genuinely changed what was possible, and better prompts genuinely helped.
But at some point in my own work, that explanation stopped being sufficient. The underlying models had become more than capable enough for many of the tasks I wanted to perform. Yet my workflows were still constrained by me. I was choosing every topic, applying every opportunity-assessment parameter, deciding which evidence mattered, correcting the same editorial patterns and reviewing decisions the system could have learned to prepare better.
The limitation was no longer primarily generation. It was the amount of judgment that still existed only in my head.
If every recurring decision returns to me, I have automated execution but I have not scaled judgment.
Better prompts were the wrong unit of improvement
I used to assume that good AI work started with better prompts. That is still a useful tactical idea. OpenAI’s own prompt-engineering guidance recommends clarity, specificity and examples because those things improve output. I still care about them. I simply no longer think the prompt is the most important unit of improvement for sustained work.
My current system works because it has been improved continuously. I do not silently brainstorm a topic for hours and then ask AI to write around the answer I eventually chose. I try to make the selection logic visible. What audience am I trying to reach? How close should the subject be to an active conversation? How much competition for attention is acceptable? How does the topic advance the existing Open Source Consulting architecture rather than chase whatever is fashionable this morning?
The same pattern applies elsewhere. If I find an opportunity and personally spend hours applying the same evaluation criteria, I remain the evaluation engine. If I receive information from multiple sources and repeatedly decide which evidence deserves attention, I remain the evidence filter. If every editorial correction comes back to me without changing the production rules, I remain the quality-control system.
AI makes the execution around those decisions dramatically faster. That sounds useful until execution becomes fast enough that my unchanged decision process is exposed as the constraint. The interesting work then moves one level up: can the reasoning behind a repeated decision become part of the system?
This extends the operating principle I described in Redesigning Work With AI Starts When I Stop Trying to Use AI. The larger opportunity is not to place a better tool inside the old workflow. It is to redesign the workflow around what the technology changes. In this case, the redesign is about making judgment more transferable.
Judgment becomes transferable by working through it
There is a practical problem with telling an AI system to “use my judgment.” Judgment is rarely available as a clean rulebook before the work begins. It carries the residue of previous decisions: patterns noticed over years, exceptions that became important only after something went wrong, preferences that are difficult to explain until a trade-off makes them visible.
I encounter the same problem when developing people. If I make the decision for someone every time, the work may move faster today, but their learning potential is reduced. If we walk through the decision together, they can apply their perspective, I can expose the line of thought behind mine, and the differences become material for the next decision. What was previously tacit starts becoming discussable.
Dale Carnegie’s modern delegation material describes delegation as more than moving work off a manager’s plate; it is also about developing others while achieving an outcome. His leadership principles also emphasize asking questions rather than defaulting to direct orders. The analogy is imperfect, but the mechanism matters: capability develops through a shared process, not only through instructions delivered at the beginning.
A recent human example made that mechanism tangible for me. In a Product Operating Model development group, I introduced a vision for using generative AI to draw on broad datasets to validate hypotheses, with people concentrating on judgment and nuance rather than manually processing every input. I did not have capacity to build the idea myself.
At the next meeting, a colleague who had worked with me for longer arrived with a prototype. It was not exactly what I would have built, which was part of the value. The prototype already reflected enough of our shared decision patterns that I could add a few attributes and move it toward presentation rather than begin from zero. Enablement had distributed part of the cognitive load.
That is what I now try to do with AI as well. I guide it through the decision rather than merely replace its answer. We explore why one option is stronger, which parameter mattered, where two principles conflict and what information would have changed the result. The white spots become questions. The repeated patterns become candidates for instructions, examples, criteria, tools or escalation rules.
When the AI starts working the way I imagined
The first improvements were unsurprising. A few adjustments reduced the number of interventions I needed to make. Outputs arrived closer to the form I wanted. Repeated corrections became less repetitive. That was useful, but it still felt like optimization.
The tone changed when the same logic started overcoming limitations that were plainly mine. At the end of a publishing cycle, for example, the system can synthesize several days of arguments and evidence into a consistently formatted Executive Briefing newsletter in seconds. The surprising part is not that a language model can summarize text. It is that the output can reflect an accumulated set of editorial rules, source boundaries, component standards, channel roles and approval gates that were refined through earlier work.
Topic selection changed in the same way. I no longer need to begin from a blank page and silently decide what deserves a week of attention. The system can work within parameters: avoid subjects so hot that OSC has no differentiated angle, avoid subjects so narrow that only a tiny specialist audience will care, check overlap with existing content, preserve the weekly architecture, and give me a small number of choices with explicit trade-offs.
I still decide. But choosing between A, B and C is a different use of my attention from manually generating and filtering the entire option space. More importantly, when I override the recommendation, the override can become evidence for improving the next recommendation.
This is where the system described in AI Agent Roles: Why I Gave Mine Ridiculous Names becomes more than role naming. A role is useful only when its decision patterns, handoffs, evidence standards and authority mature with use. Otherwise the memorable name merely hides a static prompt.
AI agent decision quality improves through iteration
Engineering practice is moving in a similar direction. Anthropic argues that evaluations become more valuable across the lifecycle of an agent because they make failures and behavioral changes visible rather than leaving teams in a reactive loop. Its 2026 guidance on evaluating AI agents explicitly recommends turning real failures and manually checked behaviors into reusable tests and iterating continuously.
Google’s current agent-evaluation documentation describes a similarly iterative cycle: define cases, run the agent, capture traces, compute metrics, analyze problems, propose improvements and then verify those improvements through another run. Its agent evaluation workflow is obviously more formal than my personal operating system, but the underlying idea is recognizable. Improvement comes from closing the loop between observed behavior and the next version of the system.
I do not maintain an industrial evaluation suite for every personal agent. I do something simpler. When I intervene, I ask whether the intervention was a one-off executive judgment or evidence of a recurring decision pattern. If it is recurring, I try to improve the system before I ask it to perform the next chunk of similar work.
The Agent Decision-Quality Loop
- Delegate a bounded decision. Give the AI enough authority to reveal how it interprets the task without pretending the risk is unlimited.
- Observe where judgment was needed. Notice not only whether the output was wrong, but why human intervention changed the result.
- Make the reasoning explicit. Work through the trade-off, missing context or principle rather than simply replacing the answer.
- Encode the reusable part. Convert recurring judgment into a simple instruction, example, criterion, control, evaluation or escalation condition.
- Test the next iteration. Let the system face a similar decision and check whether the quality actually improved.
- Expand autonomy selectively. Move toward “act unless something is unusual” only when the system has demonstrated that it understands the normal operating envelope.
The loop looks simple because it should be simple. Generative systems do not require every nuance to be expressed as a giant if-this-then-that tree before they can act. They can infer patterns from context. That means the practical challenge is often to start with a clear principle, use it, notice where reality creates ambiguity, and then make the next version slightly better.
The best escalation is a conflict between valid parameters
The maturity test for me is not whether the AI stops asking questions. It is whether the questions become better.
Consider topic selection. I might propose spending a week on how Siri AI could reduce what I call Interface Tax (as I did). The system could reasonably warn that the subject is attracting substantial attention and that entering a crowded conversation may reduce the chance that OSC earns distinctive attention. That recommendation can be correct under the distribution parameter.
I might still override it because another parameter has greater strategic weight: OSC’s topic architecture requires us to address a structural development from our own perspective, even if the expected impression count is lower. The useful escalation is therefore not “I do not know what to do.” It is “two valid rules point in different directions; which one takes precedence here?”
That distinction matters. “Unusual” does not have to mean that something is broken. It can mean the decision sits outside the established hierarchy of trade-offs. Once we resolve it, the next rule can remain simple: avoid excessively hot topics unless strategic narrative continuity makes the topic necessary.
The same logic changes how I think about trust. In When AI Makes a Decision I Do Not Like, I argued that confidence grows when I can trace where a decision came from, understand the applicable rules and see how correction changes future behaviour. Continuous enablement adds another layer: a correction should not merely repair the current output when it reveals a reusable decision principle.
Autonomy should grow with decision quality, not enthusiasm
I am most comfortable with the operating mode “act unless something is unusual.” That sentence sounds casual, but it depends on a surprising amount of prior work. The AI and I need a sufficiently similar understanding of what normal looks like, which parameters matter, what evidence is acceptable and which conflicts deserve escalation.
This is also why I resist the idea that better AI use is mainly about granting more autonomy. The goal is not maximum autonomy. It is appropriate autonomy supported by demonstrated decision quality. Human involvement should recede where the pattern is stable and the consequences are bounded, while remaining available where the decision is ambiguous, strategic, consequential or outside the operating envelope.
There is a useful leadership parallel. Research summarized by Harvard Business Review notes that delegating decision-making can free managerial capacity and give employees meaningful agency, while also recognizing that delegated decisions can become burdens when responsibility and readiness are poorly matched. The point is not simply to delegate more. It is to delegate decision-making strategically.
The AI version deserves potentially more restraint. An agent should not receive authority because a leader is excited about autonomy. It should receive authority because the class of decision is understood, the boundaries are explicit, failures are observable, escalation is meaningful and previous iterations have demonstrated sufficient quality.
The leadership analogy has an important limit
I do not think the analogy fails because AI can never participate in decision-making. The limitation I notice today is different. Human development is bilateral in a way my interaction with AI is not.
A strong colleague can go home, think about a problem I did not have time to solve, connect it with an experience I do not have, and return the next morning with something that surprises me. The Product Operating Model prototype was valuable partly because another person contributed their own perspective rather than merely absorbing mine. Senior leaders sometimes destroy that value when hierarchy becomes so strong that the junior person is treated only as a receiver of instruction.
AI can certainly surprise me. It can generate options I had not considered and combine information at a scale I cannot match. But in my current working system, it is still mostly on the receiving end of the developmental relationship. I supply the objectives, context, feedback, boundaries and permission to improve the process. That difference matters.
It is also consistent with the argument in AI Agents Are Not Digital Employees. Organizational metaphors can illuminate a mechanism without erasing the differences between people and software. A human colleague brings independent experience, motivation, relationships, accountability and a life outside the operating system. Treating agents and employees as interchangeable would flatten exactly the human contribution the analogy helps expose.
Three habits make the learning compound
The practical changes I would recommend are not sophisticated prompt techniques. They are operating habits.
First, never neglect process improvement. If a workflow creates the same unnecessary intervention twice, improve the workflow before beginning the next chunk of similar work. The temptation is always to finish today’s task and fix the system later. Later is how repeated friction becomes an operating model.
Second, think in iterations. Retrospectives exist because the value of an iteration is not only what the team delivered. It is what the team learned about how to perform the next iteration better. Humans may record those lessons and then ignore them under pressure. AI can be remarkably diligent about applying a rule that has actually been incorporated into its instructions, evaluation criteria, tools or working context.
Third, use AI to improve your work with AI. Continuous improvement has always suffered from weak documentation. What exactly went wrong last Tuesday? Which correction have I made four times? Where do most escalations originate? AI is unusually well suited to reviewing traces of its own work, identifying recurring interventions and helping turn them into candidate improvements. That does not remove human judgment from the loop; it directs human judgment toward improving the loop.
The prompt is not the system
I still write prompts. I still correct outputs. I still make consequential decisions. The change is that I try not to accept recurring intervention as the natural price of using AI.
When the system asks me the same kind of question repeatedly, I now see a design opportunity. Can we expose the judgment behind the answer? Can we encode the reusable part simply? Can the next iteration recognize the normal case and escalate only when valid parameters conflict? Can the system help us remember where it failed previously?
That is how AI agent decision quality began compounding in my own work. The model improved, certainly. But the larger shift came when I stopped waiting for the model to become better and started improving the relationship between model, instructions, evidence, tools, feedback and my own judgment.
The best leaders I have worked with did something similar with people. They did not scale by being present at every decision. They made their reasoning increasingly legible, created room for others to contribute, corrected without monopolizing the work, and expanded responsibility as capability became visible.
AI is not an employee. But the leadership lesson travels surprisingly well: if you want more leverage, do not become better at approving everything. Become better at developing a system that needs your approval less often, and makes much better use of your judgment when it does.
