A radiologist at a large teaching hospital gets a new AI triage tool. It reads each incoming chest scan, flags the ones that look urgent, and pre-marks suspected findings on the image before she ever opens it. Her job is unchanged on paper: she reads every scan and signs off. In practice her job has changed entirely. She is no longer the first pair of eyes. She is the second — the one confirming or overruling a machine that, in validation, agreed with expert consensus more than nine times in ten.
For the first fortnight she reads each scan as she always has, eyes tracking the lungs edge to edge, treating the AI’s marks as a colleague’s opinion rather than a verdict. Then the pattern of the work teaches her what the pattern of the work always teaches. The tool is right. It is right on Monday and right on Thursday and right for a hundred scans in a row. Her overrules get rarer. Her reads get faster. Somewhere around scan four hundred, without a single decision to do so, she stops reading the image and starts reading the marks — checking whether the AI’s boxes look plausible rather than searching the scan for what the boxes might have missed.
Then comes the scan the tool reads as clear, on a patient whose tumour sits in exactly the region the model is weakest on. The finding is faint but visible — to a radiologist who is looking. She is no longer looking. She is confirming. She signs off in nine seconds and moves to the next.
Nothing about this radiologist is negligent. She is highly trained, highly motivated, and doing precisely what the system was designed to make her do. That is the point. The failure was not a lapse in her character. It was the predictable output of asking a human to do the one thing humans are engineered to fail at — and then calling that arrangement a safety control.
A problem we solved, and named, in 1948
During the Second World War, the British military had a problem with radar and sonar operators. The equipment worked. The operators, staring at screens for hours waiting for the rare blip of an enemy submarine, kept missing the blips. Not through incompetence — through something more fundamental, and more troubling, because it could not be trained or disciplined away.
A young psychologist named Norman Mackworth was asked to study it. He built an apparatus now known as the Mackworth Clock: a plain clock face with a hand that ticked in regular steps and, at random and infrequent intervals, jumped two steps instead of one. The observer’s only task was to catch every double-jump across a two-hour watch. It is about the simplest sustained-attention task imaginable.
The result, published in 1948 and replicated thousands of times since, founded an entire field. Detection accuracy did not hold steady and then tail off with fatigue at the end. It collapsed early. Performance fell by ten to fifteen per cent within the first thirty minutes — and kept declining from there. This is the vigilance decrement, and its most important property is the one that makes it so dangerous: it afflicts motivated, rested, well-trained people who know they are being tested. Willpower does not touch it. It is not a flaw in particular workers. It is a property of the attentional system every human is issued at birth.
The cruel structure of it is this. Human attention is not built to be sustained on a source of information where almost nothing happens. The rarer the signal, the faster and deeper the decrement. A monitoring task where events are frequent keeps the mind engaged. A monitoring task where the thing you are watching for almost never occurs is precisely the task the brain cannot hold.
Which sets up the central irony, articulated in 1983 by the cognitive psychologist Lisanne Bainbridge in a four-page paper, “Ironies of Automation,” that has aged better than almost anything written about technology since. Her argument was deceptively simple: we automate the tasks humans are bad at, and in doing so we hand the human the residual task of monitoring the automation — a task humans are even worse at.
“We know from many ‘vigilance’ studies that it is impossible for even a highly motivated human being to maintain effective visual attention towards a source of information on which very little happens, for more than about half an hour.”
— Lisanne Bainbridge, “Ironies of Automation,” Automatica, 1983
Bainbridge’s paper has survived for four decades because it does not describe a fact about 1983 technology. It describes a structural relationship between humans and reliable machines — one that gets worse, not better, as the machine improves. And we have just handed that exact relationship to two billion knowledge workers, and given it a reassuring name.
Why AI is the most dangerous automation we have ever monitored
Every previous generation of automation that put a human on watch at least had the decency to look mechanical. A cruise control, an autopilot, an assembly-line sensor — you could tell you were supervising a machine. Generative AI removed that tell. It produces work that looks exactly like the work of a competent, careful colleague. It writes in complete sentences. It cites. It hedges where a thoughtful person would hedge. It sounds certain where a confident person would sound certain. And it is right often enough that scrutinising it begins to feel not just unnecessary but faintly insulting, like re-checking the arithmetic of someone who has never once got the sum wrong.
This fluency is not a cosmetic feature. It is the mechanism by which the trap springs harder. The research literature on automation bias — our tendency to over-trust automated systems — identifies two distinct failure modes, and fluent AI aggravates both simultaneously. The first is the omission error: you fail to notice a problem because the system did not flag it. The radiologist who stops searching the scan and searches the boxes instead is committing omission errors by the hundred; she has outsourced not just the analysis but the very act of looking. The second is the commission error: you actively follow the system’s recommendation even when other evidence in front of you contradicts it. The more authoritative the system sounds, the more its confident wrong answer overrides your quiet correct doubt.
What makes this worse than the old radar screen is that the AI is not a source of information on which very little happens. It is a source on which a great deal happens — a torrent of plausible, polished output — almost all of which is fine. That is the vigilance decrement’s ideal habitat, dressed up as its opposite. The operator feels busy, engaged, productive. There is no dead screen to fight. There is a stream of competent work sliding past, and the rare defect is a single deformed shape in a river of well-formed ones, moving at the speed of your approval clicks.
And here is the reliability paradox in its purest form: the better the model gets, the more certainly the human oversight fails. A model that is wrong ten per cent of the time keeps you at least intermittently alert, because you catch errors often enough to stay in the game. A model that is wrong a tenth of a per cent of the time lulls you completely — and then delivers its rare error to a reviewer who has not truly reviewed anything in weeks. Improving the AI does not reduce the risk of the human-plus-AI system. Past a certain point, it increases it.
What gets lost, and the debt nobody records
The evidence that we have already surrendered the oversight role is not subtle. In the 2025 global study of trust in AI conducted by KPMG and the University of Melbourne — roughly 48,000 people across 47 countries, one of the largest surveys of its kind — 66% of respondents said they rely on AI output without evaluating its accuracy. 56% admitted they have made mistakes in their work because of AI.
These are not the numbers of a species diligently keeping AI in the loop. They are the numbers of a species that has quietly stepped out of it while leaving a mannequin in the chair.
The cost of this does not appear anywhere it can be easily seen, and that is precisely what makes it accumulate. Every hour AI saves is booked immediately, visibly, and enthusiastically — in dashboards, in time-saved metrics, in the genuine relief of people whose drudgery just evaporated. But that saved hour is not free. A portion of it is a loan drawn against future vigilance, and the repayment falls due unpredictably, all at once, on whoever happens to be holding the rare error when it finally ships.
Call it the vigilance debt: the silent, compounding liability created every time speed is purchased by withdrawing attention, recorded on no ledger because a monitoring failure produces no artefact until the moment it produces a catastrophe.
Ordinary work leaves a trail of near-misses. You notice yourself almost making a mistake, and the noticing keeps you sharp. Vigilance failure leaves no such trail. The scans you signed off without reading look identical to the scans you signed off after reading. The letters you approved on rhythm look identical to the letters you approved on judgement. There is no feedback, no wobble, no warning tremor — right up until the one that was wrong goes out to four thousand customers or one patient, and the incident review discovers, to everyone’s genuine surprise, that the human control everyone was relying on had quietly stopped controlling anything months ago.
Regulators have noticed what most organisations have not. The European Union’s AI Act, in its provisions on human oversight, does something almost unheard of in legislation: it names a specific cognitive bias. Automation bias is the single psychological phenomenon the Act explicitly calls out as a threat to effective human oversight of high-risk systems. The lawmakers understood what the average deployment plan does not — that writing “a human will review the output” into a process does not make the review happen. Vigilance is not summoned by an org chart. It is defeated by one.
The four faces of the vigilance trap
The trap does not present the same way to everyone. It has recognisable archetypes, and naming them is the first step to noticing yourself becoming one.
The Rhythm Clicker. This is the reviewer whose approvals have become motor memory. The task has a cadence — open, glance, approve, next — and once the cadence sets, the content stops entering conscious attention at all. The Rhythm Clicker is not lazy; they are efficient, and their efficiency is the disease. They have optimised a task down to the point where the one part that mattered, the actual looking, has been optimised away. The tell is that they could not tell you anything specific about the last ten things they approved.
The Fluency Truster. This reviewer still reads — but reads for polish rather than for truth. Because the AI’s output is articulate, structured and confident, the Fluency Truster’s brain accepts fluency as a proxy for correctness, which it never was. They will catch a typo and miss a fabricated statistic, because the typo disrupts the surface and the fabrication does not. This is the most insidious archetype, because it feels like diligence. They are working hard. They are just checking the wrong layer.
The Volume Drowner. Before AI, this person reviewed ten things a week and could give each real attention. AI raised the inflow to a hundred, and nobody adjusted the reviewing capacity, because reviewing was not the thing the rollout was about. The Volume Drowner is trying to be vigilant and is structurally prevented from it. Their failure is not attentional but arithmetic: genuine scrutiny of a hundred artefacts a week was never on the menu, so what gets called review is triage at best and theatre at worst.
The Lonely Sentinel. This is the single human designated as the check on a system that runs thousands of times a day — the one radiologist, the one approver, the one “human in the loop” whose sign-off legitimises the entire pipeline. The Lonely Sentinel carries the full weight of a safety story they cannot possibly bear, and everyone upstream and downstream behaves as though the sentinel’s presence has made the system safe. Their existence is what allows the organisation to stop worrying. Their existence is the reason it should worry more.
What to actually do about it
The first and most important move is to stop treating vigilance as a matter of will. Bainbridge told us in 1983 that you cannot exhort your way out of the vigilance decrement, and seventy-five years of data agree. A policy that says “reviewers must carefully check all AI output” is not a control. It is a wish, and worse, it is a wish that transfers liability onto individuals for a failure the system made inevitable. Every intervention that actually works is a design intervention, not a motivational one.
If you are an individual reviewer: your value has quietly inverted. The scarce, rising skill is no longer producing output faster — the machine wins that outright and permanently. It is being the person who genuinely reads the thing when everyone around you has stopped. Cultivate adversarial review as a deliberate practice: begin from the assumption that the plausible answer in front of you is wrong, and spend your attention hunting for where. Read for truth, not polish. And protect your own attentional conditions, because a Rhythm Clicker is just a good reviewer who was never given a reason to stay awake.
If you manage the work: the highest-leverage thing you can do is design the reviewing task so that a normal human can actually perform it. Cap the length of unbroken monitoring stretches — the decrement bites at thirty minutes, so rotate reviewers before that, not after. Deliberately re-introduce signal: seed a known error rate into the stream so that the reviewer catches something often enough to stay engaged, the way pilots train on simulated failures and airport screeners are shown planted threat images to keep detection rates up. Measure catch rates, not throughput. And never let one person be the whole control for a high-volume system; the Lonely Sentinel is an org-design failure wearing a job title.
If you lead the organisation: internalise the reliability paradox before it internalises you. The safety of a human-plus-AI system does not rise monotonically with model quality — it can fall as the model improves, because improvement erodes the human check. This means your risk is highest precisely when your metrics look best and your people feel most confident. Instrument for it. Ask not “how much time did AI save?” but “when did a human last catch something the AI got wrong, and how would we know if they had stopped?” If you cannot answer the second half of that question, you do not have oversight. You have a green button and a person to press it.
If you design these systems: the goal is not to add a human to the loop. It is to build a loop a human can actually stay awake inside. That means surfacing the model’s uncertainty rather than hiding it behind uniform fluency, varying the presentation so the reviewer cannot settle into rhythm, forcing genuine engagement at the moments that matter rather than requesting blanket attention that cannot be given, and — hardest of all — resisting the commercial pull to make the output so smooth and confident that trusting it becomes the path of least resistance. Every increment of polish you add is an increment of vigilance you take away.
The uncomfortable truth
We told ourselves a story about AI and human judgement. In the story, the machine handles the volume and the human provides the wisdom — the oversight, the discernment, the final responsible glance that keeps the whole thing safe. It is a comforting story, and it is the story written into a thousand deployment plans and at least one major piece of European legislation. The trouble is that it depends entirely on a capability we have known for three-quarters of a century that humans do not reliably possess: sustained vigilance over a system that rarely fails.
The deeper irony is that we are running the experiment backwards. We are pouring effort into making the models more reliable, and each gain in reliability quietly weakens the human check that our safety story rests on. The better it gets, the less we watch. The less we watch, the more the rare failure matters. And the rare failure is the only kind a highly reliable system produces.
“Human-in-the-loop” was never a solution. It was the problem, restated as a reassurance. The organisations that come through this well will be the ones honest enough to admit that a human posted to watch a nearly-perfect machine is not a safeguard but a sedative — and to redesign the work around the reviewer we actually are, rather than the tireless sentinel we keep pretending to be.
The machine’s job is to be right almost always. Ours was supposed to be catching the moment it isn’t. We should probably find out whether anyone is still watching.