Illuminate AI’s Kirsty Mac and Claire Carmichael recently hosted a webinar on Human-Centred AI Enablement. During the Q&A, a question landed that stuck: “can you explain your human in the loop ick?” This article is the longer answer. Watch the full session here
The “Human in the Loop” Problem
By Claire Carmichael, Innovation Director, Illuminate AI
At a recent webinar on human-centred AI enablement, I was asked a question:
Can you explain your “human in the loop” ick?
It is a fair question. Because it’s a fairly commonly used phrase in my line of work and every time the phrase appears in a conversation, I have the same immediate reaction — if the recording is accurate, I probably visibly shuddered when it came up.
The reaction is not really about the words themselves. It is about the image they trigger.
Whenever I hear “human in the loop”, I picture a circuit.
The machine does most of the work. A human is inserted somewhere along the line so that if something goes wrong, the system does not fail completely.
But in most circuits, the component designed to fail first is the fuse. It absorbs the failure. It burns through. Then it gets replaced so the system can continue running.
That is the mental image that sits behind my discomfort with the phrase.
And the uncomfortable truth is that this is not a new problem.
Technology governance recognised the issue decades ago. As IBM put it in 1979:
A computer can never be held accountable. Therefore a computer must never make a management decision.
Machines cannot be held accountable. Which means that when AI becomes part of a process, accountability still has to sit somewhere within the system.
Very often, “human in the loop” quietly answers that question by placing a person at the end of the workflow.
Not necessarily because they have been given the tools, time, or realistic ability to do the job well — but because their presence means ‘someone’ can be held responsible if ‘something’ goes wrong.
That is not a safety mechanism.
It is risk transfer with a reassuring name.
When the “human in the loop” fails
There is an example I come back to regularly. An article was published in a newspaper where the final paragraph was clearly a ChatGPT prompt asking whether the user would like a shorter, snappier version for the front page. It had been left in. It went to print.
It circulated on LinkedIn, shared largely for a laugh. “How could anyone possibly miss that?!”
For me, that question points in the wrong direction. Because when something like that can be allowed to happen, I assume a failure in process design.
In this particular case, the editorial team reviewing content for that publication had been reduced from seventeen people to one.
One person. Reviewing at least seventeen times as much output as before.
That is not a human in the loop. That is a person being asked to do a structurally impossible job while the organisation tells itself it has a review process. A gate that existed on paper, that scale made meaningless in practice.
There is a clinical parallel worth taking seriously. When radiographers work alongside AI diagnostic tools and routine cases are handled by the system, what remains for the human is a relentless stream of difficult, ambiguous, high-stakes decisions. Everything that arrives is an edge case. There is no routine, no breathing space, no flow.
Research shows a significant increase in burnout in these situations — not because the technology made the work harder in the obvious sense, but because it hollowed out the manageable middle and left only the edges.
There is a second effect running alongside this. The better a system performs over time, the more humans trust it. And the more we trust it, the less critically we engage with it.
A study published in JAMA Network Open found that when AI diagnostic tools made errors, many clinicians accepted the incorrect suggestion rather than questioning it. The human review did not catch the mistake. In some cases, the presence of an AI recommendation actually made the error less likely to be caught.
This is often called automation bias. It is not a character flaw. It is a predictable response to working in systems that produce large volumes of output while only occasionally failing.
And when systems are designed without accounting for how human attention actually functions under load, the problem is not the humans.
It is the design.
We are all leading from the front of something
Kirsty Mac made a point in the run-up to the webinar that I keep coming back to: there is a greater risk that our humans begin to think, sound and act like machines than that the machines start to think, sound and act like people.
That is what happens when people spend their working days doing only exception handling — only the hardest and most draining decisions, with no routine, no breathing space, and no room to bring the judgement and care they are actually good at. We do not just risk burning people out. We risk flattening the distinctly human qualities that make them valuable in the first place.
And that is not just a technical design problem. It is a leadership one.
I think about this not only professionally. I am also a parent. My son is twelve. He will enter a workforce that is being shaped right now, by decisions being made right now, by people who are — whether they feel like it or not — leading from the front of something.
That is true of everyone reading this.
If you are making decisions about AI in your organisation — what to automate, how to structure oversight, where to place people, what review realistically means — you are setting precedents. And once those precedents are embedded in processes, culture and expectation, they will be far harder to undo than they were to create.
This is not an argument for slowing down. It is an argument for making eye contact with the decisions we are actively making.
Because “human in the loop” is an imprecise shorthand that hides a design decision. Where the human sits, what they are responsible for, and whether that responsibility is realistic matters enormously — both for the quality of the output and for the wellbeing of the people involved.
The alternative is not complicated to describe, even if it takes real work to build. Whether we are designing a system, an agent, a workflow, or an entire programme of change, the principle should be the same: the human sits at the centre, supported by technology — not the other way around.
Not the fuse in the circuit.
The heart of it.
Research from Harvard Business Review suggests this matters more than many organisations realise. Their work indicates that AI is amplifying workloads rather than reducing them for many people. If we do not make these design decisions deliberately, the distance between systems built for people and systems patched together around them will only grow.
Pay attention to “the ick”
So, the next time you are in a room and someone shrugs and says, “don’t worry, there’s a human in the loop”, pause. Feel uncomfortable. Ask the follow-up questions.
What exactly is that human responsible for?
Can they realistically do it?
How much are they expected to review?
What does “review” actually mean in this process?
What happens when volume increases?
And what happens when the system is right most of the time — will anyone still be looking closely enough when it isn’t?
If the answers to those questions are unclear, the loop is not a safeguard. It is simply a place to park accountability.
And the answers will tell you whether the system was genuinely designed with people in mind, or whether the person was inserted to make the accountability less visible.
We talked about all of this in our recent webinar on human-centred AI enablement — the one that prompted this piece. If it has landed and you want to go deeper, that recording is a good place to start.
And if you are currently trying to get an AI initiative or an agentic solution into production and it is not landing the way you expected — adoption is low, the humans in your process are struggling, or something is not working the way it looked on paper — it might be worth a conversation about design.
Not because the technology is wrong, but because the people might not yet be in the right place in the circuit.
That is exactly the kind of conversation we are here for.
References and further reading
JAMA Network Open (2024) — clinician decision-making and automation bias in AI-assisted diagnosis