Psychological Acceptability in AI Agent Policy Design

A hospital rolls out an AI scheduling agent that can also flag early signs of patient deterioration. The tool works. The accuracy numbers hold up in testing. Six months later, nurses are double-checking every flag by hand, and a smaller group has started ignoring the system altogether, routing around it with their own spreadsheets. Nobody told them to distrust it. Nobody told them to lean on it blindly either. The policy assumed trust would follow competence. It didn't, and that gap between what a system can do and what people will actually let it do is where most AI agent policy breaks down.

This is a design problem that starts with a wrong assumption: that trust is a byproduct of good engineering rather than something that has to be built on its own terms. Policies written as if trust is already there, or will arrive once people get used to the tool, tend to produce one of two outcomes. Either people rely on the system too much, handing off judgment they should keep, or they avoid it, building workarounds that quietly undo whatever the policy was meant to accomplish. Both outcomes are predictable responses to a mismatch between how much autonomy a policy grants an agent and how much trust the people living with that policy actually have. Get the mismatch right, and the rest of agent governance, the autonomy levels, the oversight checkpoints, the transparency dashboards, has a foundation to sit on. Get it wrong, and no amount of technical polish fixes it.

The two ways trust goes wrong: automation bias and algorithm aversion

Trust in AI agent policy doesn't miscalibrate in one direction. It miscalibrates in two, at the same time, often inside the same organization, and fixing one failure mode can quietly make the other one worse.

The first is automation bias: people start treating an agent's output as correct simply because it came from the system, and they stop checking it the way they'd check a colleague's work. The resistance is about giving up decision authority to something, or someone, a person doesn't trust.

These two failure modes can sit side by side in the same company. Neither group is being irrational. Each is responding to its own relationship with the system, its own stakes, and its own sense of what happens if the agent gets it wrong.

A third dynamic makes both worse. That expectation turns out to be false more often than policy designers want to admit, and the result is a kind of oversight illusion: people believe a safety net exists, so they stop acting as if they need to catch problems themselves. That illusion will come back later in this piece, because it's the central weakness in human-in-the-loop design. For now, the point is narrower: designing for "more trust" or "less trust" as a single dial misses what's actually happening. Some parts of an organization need less friction. Others need more. A policy that treats trust as one number for the whole system will get it wrong for most of the people using it.

What makes a policy psychologically acceptable (and why fairness is central to it)

Getting people to comply with a policy and getting them to actually accept it are different achievements, and the gap between them is usually fairness, not usability.

Trust in AI, seen clearly, isn't a general attitude toward technology that a cleaner interface can shift. Procedural fairness sits inside all four of those factors, and it carries more weight than most policy designs give it credit for. An organization that strips humans out of a decision loop to save time may be saving time at the direct cost of its own legitimacy in the eyes of the people it serves.

Interviews with patients at UMass Chan Medical School, drawn from 29 adults recruited through the UMass Memorial Health system, found this pattern directly. Patients saw real value in AI conversational agents for mental health support, but consistently wanted a human to stay involved, and that preference held even when the AI was clearly capable of handling the task well. It's about needing a human presence as a condition for the interaction to feel fair and accountable.

Acceptability also moves with the task. A patient who's perfectly comfortable with an AI agent handling appointment scheduling may refuse outright to let one anywhere near a diagnosis or a therapy session. That means a policy can't treat acceptability as a single setting applied uniformly across everything an agent does. It has to be assessed function by function. The 2026 policy brief from Schoene and colleagues, published through Frontiers in Public Health, makes this explicit for mental healthcare: informed consent and human oversight aren't paperwork requirements layered on after the fact, they're preconditions for people to trust the system at all, and skipping either one produces resistance that no amount of technical refinement will undo.

Why graduated autonomy is the right structure

If trust has to be earned rather than assumed, then the structural answer is autonomy that expands as reliability is demonstrated.

Graduated autonomy means laying out explicit levels of independence for an agent, starting with systems that only execute specific sub-tasks when directly told to, and working up toward systems that operate under broad human monitoring rather than step-by-step approval. An agent earns its way up the ladder. It doesn't start there.

What should govern where an agent sits on that ladder isn't just what the system is technically capable of doing. An agent that can undo its own mistake warrants far less oversight friction than one making a call that can't be taken back, and policy should say so explicitly instead of treating every action an agent takes as equally weighty.

The autonomy level shouldn't be fixed once reliability is demonstrated, either. That keeps the framework alive rather than a one-time classification made at launch and forgotten.

The human-in-the-loop problem: when oversight mechanisms undermine the oversight they promise

Human-in-the-loop design is the default answer to nearly every AI agent governance question today, and it carries a flaw that is visible only after the system has been running for a while: the appearance of oversight and the reality of oversight can pull apart from each other.

The oversight illusion introduced earlier produces this effect. The checkpoint still exists on paper. The judgment it was supposed to exercise has quietly stopped happening.

A position paper from September 2026 pushes this further, arguing that relying on human-in-the-loop oversight can degrade the cognitive capacities people need to oversee a system well. It wears down the exact judgment it exists to exercise, the same way a skill atrophies from disuse. This echoes a broader pattern already visible in regulatory debates over automated decision systems: oversight requirements written into policy can create strong incentives for reviewers to rubber-stamp quickly rather than slow down and actually think, especially when volume is high and the system has been reliable so far.

That leaves policy designers with a real dilemma. Adding more checkpoints doesn't fix the underlying issue if the cognitive conditions around those checkpoints stay the same. It just relocates the failure somewhere else in the process. The honest answer, as far as current thinking goes, isn't to strip oversight out. Getting the frequency and placement of those checkpoints right is still an open, difficult question, and treating it as settled would undersell how real the tension is.

How transparency mechanisms build or destroy trust

Transparency gets treated in most AI agent policy as a single lever: more explainability, more visibility, more trust. That's not how it works in practice. Transparency only builds trust when it's matched to what the person receiving it actually needs to know, and generic explainability can feed automation bias just as easily as it corrects it.

There's a real difference between transparency that helps someone ask sharper questions about what an agent is doing, and transparency that just reassures them the system is functioning. A decision nobody can trace isn't a decision anyone can meaningfully trust, and that traceability has to be something people can actually use in the moment, not something that exists in theory. It tells you what happened. It doesn't stop anything from happening.

Real-time visibility into what an agent is doing right now, and what it has touched or accessed, is a fundamentally different capability from a log you check after the fact. Schoene and colleagues point to exactly this gap in their 2026 policy brief, identifying the lack of transparency in how AI systems make decisions as a direct threat to accountability, informed consent, and clinical trust. Explainability, in their framing, is a structural requirement for the relationship between a person and the system making decisions that affect them.

A useful parallel comes from a 2026 CHI paper by Cooper and colleagues on responsible design for mental well-being tools. They argue that an AI tool's "active ingredients", what it actually does to produce its effect, and whether it reliably delivers that effect, need to be specified and disclosed, because responsible design depends on knowing precisely what a tool is accountable for. That same logic holds for an enterprise AI agent approving a transaction or flagging a risk. Research on trust assurance backs this up directly: mechanisms like clear human oversight structures, accountability policies, and adherence to recognized standards produce meaningfully more willingness to trust a system than technical explanations of how the model works ever do on their own.

How domain and user population shape the acceptability threshold

None of the design principles above work as a universal formula. Where the acceptability threshold sits moves depending on the domain and depending on who's using the system, and policy built around a single standard will misjudge both the low-stakes and high-stakes ends of that range.

Consumer comfort with AI agents varies sharply by domain, and discomfort runs highest in exactly the areas where AI deployment has been most ambitious: medical advice, mental health support, financial decisions. Most Americans say they'd rather deal with a human in these settings, even when an AI alternative is available and functional. That preference isn't a temporary unfamiliarity that will fade with exposure. It tracks with how much is at stake if the system gets something wrong.

Vulnerable users carry the sharpest version of this risk. An AI interaction that reinforces a delusion, or deepens an unhealthy emotional dependency, or produces outputs skewed by training data that doesn't represent the population using it, turns acceptability from a design preference into a safety question. Legislative responses have started to track this logic. Utah's SB 149 requires businesses to disclose when consumers are interacting with generative AI, and the state has also moved to regulate mental health chatbots specifically, including requirements around licensed professional involvement in their development. Illinois has gone further, restricting fully autonomous AI therapy. These moves, taken together, reflect a broader regulatory instinct: some domains carry enough risk that human involvement isn't negotiable, regardless of how capable the underlying system is.

Cooper and colleagues offer a useful way to think about this variation with their framework of design analogies. An AI tool might function more like a nutritional supplement, loosely helpful, low stakes, minimal guarantees, or more like an over-the-counter drug, or a yoga instructor offering guidance, or a primary care provider making decisions with real consequences. Each analogy carries a different set of guarantees, a different risk profile, and a different standard of accountability for the outcome. The same underlying AI tool, deployed in two different contexts, can reasonably need two entirely different levels of governance. Acceptability is a judgment that has to be remade for every domain and every population a system touches.

Agency decay as the long-run cost of policies that don't sustain human competence

The failure mode that gets the least attention in AI agent policy is what happens to human judgment over the months and years after a system has already been accepted and is working well.

Call it agency decay. As AI agents take on more of a workflow, the people working alongside them often show early productivity gains, the tool genuinely helps, the output looks good, the task gets done faster. But those same people can show declining engagement with the underlying task over time, a gradual loss of the skills needed to evaluate the agent's work critically, and a spreading sense that responsibility for the outcome belongs to someone else now, maybe the system, maybe whoever approved its deployment. It is invisible in a six-month review. It appears years later, when something finally goes wrong and the people meant to catch it no longer have the sharpness to do so.

This is what makes acceptability a moving target rather than a box checked at deployment. The cognitive skills that real oversight depends on, noticing when something looks off, questioning an output instead of accepting it, holding a clear sense of personal responsibility for the result, don't stay constant just because a policy once earned genuine trust. They need to be actively exercised to stay sharp, the same way any skill does. A policy that succeeds completely at launch can still be setting up a failure five years out, if nothing in its design keeps the humans in the loop actually practicing the judgment the loop depends on. Acceptability, in other words, is a condition that has to be sustained, on purpose, for as long as the system stays in use.