Imagine an AI-supported healthcare system flagging someone for follow-up after reading their glucose trends. The recommendation appears decisive. A clinician is asked to approve it, yet several days of readings are missing because a device stopped syncing. If the interface hides that gap, what exactly is the clinician approving? The human is in the loop, but that does not mean the human is in control.
The scenario is illustrative, not a reported clinical incident. It comes from the kind of problem that preoccupies Sangeeta Parakala, a product designer whose work has spanned Slope’s AI-enabled clinical-research platform, Omnicell’s pharmacy systems and Optum’s clinical decision-support products. These are environments where a software decision can reverberate far beyond the screen.
In an email interview with DesignWhine, Parakala makes a careful distinction: her experience includes AI-powered systems with bounded autonomy, not the design of fully autonomous AI products. She does not claim to have solved human oversight. Instead, she asks whether familiar design principles, visibility, feedback, recovery and control, can still protect human judgment as machines assume more responsibility.
Approval Is Not Authority
When product teams talk about keeping a person involved, the design response is often another approval step. But an approval button cannot compensate for missing context or an inability to contest a decision. A person may be held accountable for an outcome while having almost no practical power over it.
“Human control does not mean approving every decision; that creates the appearance of oversight without making it meaningful. It means understanding the system at the level required, recognizing when it moves outside acceptable boundaries, and having the authority to question, redirect, pause or override it.”
The distinction is about what the reviewer can do, not whether their name appears in an audit trail. Can they inspect the evidence, identify a breached boundary and redirect the workflow? Can they stop an action before it becomes irreversible? Without those capabilities, the interface may be documenting compliance rather than enabling judgment.
“If a person’s role is only to approve a decision they cannot understand or change, that is not human control. It is oversight theater.”
This is also the design problem behind NIST’s treatment of human-AI interaction: human involvement can take different forms, and its effectiveness depends on the role and authority people actually have. The product needs to make that authority visible and usable.
Risk Should Determine Visibility
Making every automated step visible may sound responsible, but it can create another failure: constant supervision of low-risk actions that trains people to ignore every notification. Parakala describes oversight as proportional, using an everyday analogy. Tying shoes needs little supervision; walking to a park may need monitoring; crossing a busy road needs direct attention. The point is not to compare AI with children, but to show that scrutiny must follow the consequences.
“Visibility should increase with risk, uncertainty and irreversibility. For me, the decisions that must be surfaced are those that materially change outcomes, cross agreed boundaries or are difficult to undo.”
For designers, that is a more practical brief than making an AI system infinitely explainable. Let reversible, routine actions remain quiet. Surface unexpected behaviour and missing information. Require meaningful intervention when a decision crosses a limit or becomes difficult to undo. Attention should be reserved for moments when human judgment can alter the result.
We have explored a related consequence in our analysis of why an AI agent’s Stop button must actually interrupt execution. Parakala’s point goes one step earlier: the system must know when to invite that intervention in the first place.
Fluent Chat Can Hide the Machinery
Conversational AI offers a seductive simplification. People can ask for an outcome in ordinary language without navigating the underlying application. But a polished answer can conceal the sources consulted, the assumptions made and the actions still underway. A chat window may make complexity easier to approach while making the system itself harder to inspect.
“Conversational UI does not have to mean text alone. The conversation can be paired with plans, sources, status views and confirmation steps that make the system’s state visible.”
A useful interface might combine conversation with a visible plan, source references, status updates and a chance to change course before a consequential action. If an assistant says it has handled a task, users should be able to tell what it changed and what remains pending. Fluency helps people communicate intent; it does not establish that the underlying workflow is safe, complete or correct.
The concern is not alien to traditional UX. Nielsen Norman Group’s usability heuristics already foreground system-status visibility and user control. AI turns those principles from useful interface habits into safeguards against actions that may proceed while the user is looking elsewhere.
Uncertainty Must Change the Workflow
In healthcare, confidence can be a dangerously reassuring presentation choice. A score may look precise without explaining the uncertainty behind it. Parakala instead asks designers to show what evidence is available, what is missing and what a person can do with that knowledge.
“If several days of data are missing because a device stopped syncing, the interface should explain that the recommendation is based on incomplete data and help the user decide what to do next.”
That example is more revealing than a generic warning label. The problem is not simply that the recommendation has a lower confidence value. The problem is that the clinician may need to verify data, seek additional context or defer a decision. In other words, uncertainty should alter the path through the product, not merely the colour of a metric.
“Uncertainty should change the workflow, not merely the color or value of a score.”
That demands work from product teams beyond a new component in the design system. They must decide what information gaps should pause automation, which exceptions require expert review and how a person can reverse or escalate an action. As our examination of UX evaluation for AI products argues, a superficially successful interaction can still conceal an unreliable system. If a warning cannot lead to a different decision, it risks becoming interface decoration.
Autonomy Needs Clear Boundaries
Parakala is wary of declaring one new design principle for agentic products. Status visibility, error prevention, recovery and clear feedback remain essential. What changes is how much a system can do before it has to ask for help.
“I would also add bounded autonomy: the system should act independently only within clear limits and involve a person when it reaches those limits. This does not mean inserting someone into every decision.”
The limits might define what information an agent can access, which decisions it may take independently and when it must hand control back. These are not abstract governance settings. They determine whether the user sees a suggestion, a pending action or an irreversible outcome, and whether the interface offers a real way to intervene.
In a later email, Parakala described people as the connective tissue of complex systems: they bridge incomplete information, interpret ambiguity and make judgment calls across workflows. AI is entering that same connective layer, but its behaviour may be probabilistic and difficult to anticipate. Her concern is therefore not simply whether a machine can explain itself.
“People need more than an explanation of what the system did. They need to understand uncertainty, recognize when their judgment is required and have a meaningful way to respond.”
Human Agency Has to Be Designed
For DesignWhine, the force of Parakala’s argument is that human agency cannot be added at the end of a workflow as a ceremonial checkpoint. It has to shape permissions, boundaries, escalation paths, reversibility and the visibility of uncertainty. The same question appears in our argument for the right to say no to AI: a choice means little unless the product can honour it.
Before describing an AI system as human-controlled, product teams should ask three questions. Can a person understand the system’s next consequential action? Can they recognise the uncertainty or missing evidence that matters? And can they pause, redirect or reverse the action while their judgment can still make a difference?
Parakala does not offer a comforting formula for autonomy. She offers a tougher standard for its designers. The measure of human control is not how often the interface asks for approval. It is what a person is empowered to change when that approval matters.
A human in the loop is not necessarily a human in control. What is the clearest example you’ve seen of AI oversight that gives people real power to challenge or stop a decision?