# Automation Blindness (and why it matters when you Vibecode)

There's this problem I have been encountering that I could not quite put into words. Something specific, that had been bothering my about the AI workflow. I recently read an article that talks about a psychological penomenon, which explains my feeling quite well.

By noone | 2026-09-27 | Technology
Canonical URL: https://the-tall-order.blog/posts/automation-blindness-and-why-it-matters-when-you-vibecode/

Here is the corrected version of your text with typos, spelling errors, and minor punctuation issues fixed:

There's this problem I have been encountering that I could not quite put into words—something specific that had been bothering me about the AI workflow. I recently read an article that talks about a psychological phenomenon, which explains my feeling quite well.

The idea that AI does work and humans check it is fundamentally flawed. It leads directly to automation blindness (also known as automation inattention). At that point, the human cannot properly contribute the value they have, so you might as well not have them in the loop.

So what does automation blindness mean? A good example is to think about airport security (I got this analogy from Cory Doctorow's newest book, which is quite an interesting read, even though I do disagree with some points). The personnel at airport security are world champions at recognizing water bottles. But in actual tests where people actively try to smuggle weapons or explosives, the human monitors consistently fail to catch it. How could that be? It should not be hard, should it?

It turns out humans are simply not built for eternal, passive vigilance. We are terrible at catching rare but critical errors in automated systems. If some automation works 99% of the time, just telling someone to pay attention just does not work. The mind naturally wanders off, and no amount of discipline can work against that.

Now, if, say, 20% of people at the airport had guns on them, that would seem pretty dangerous. But also, they would probably catch them every time. Good on them!

Let's say we know AI is highly accurate at spotting cancer cells on X-rays. If we tell a radiologist to just evaluate the AI's output, that doctor will lose their effectiveness. If the AI is right 90% of the time and the doctor is right 75% of the time, the logical assumption is: let the AI analyze, let the doctor review. In reality, the doctor becomes completely blind to the rare mistakes the automated system makes.

Now if hospital management hears that AI is right 90% of the time, they might just be inclined to fire all radiologists but one, who now is doomed to fail to review AI slop all day. Doctorow talks about this in his book too.

The correct way to implement new technologies is to have employees integrate them into their workflow. Maybe, when the AI disagrees with the doctor's decision, it nudges them to look again. That way, you can actually improve the error rate well beyond what either can do on their own.

Approaching AI from a "human reviews AI" perspective just doesn't work, at least in its current form.

This leads straight into the Out-of-the-Loop (OOTL) Performance Problem. When people passively observe automated systems, their situational awareness collapses. We get complacent, our skills decay, and it becomes dangerous or incredibly difficult to assume manual control when things fail. This phenomenon is well understood in airline pilots.

This collapse in situational awareness happens in three stages:

- **Perception:** You fail to notice baseline parameters or critical cues.
- **Comprehension:** You struggle to understand the meaning of observed events or failures.
- **Projection:** You become unable to anticipate the future state of the system or make quick decisions.

In software engineering, the result is that you lose context. Your mental image of the codebase becomes whatever Claude or ChatGPT handed you. When you review it, you might see sensible changes, but you completely miss the big picture.