# Recognizing failure

> Model failures rarely announce themselves; they arrive as fluent, confident text. This topic shows what hallucination looks like in real documents and code, how bias enters and repeats, and how people drift into overreliance and automation complacency once a tool is usually right.

Page: https://schubergphilis.github.io/ai-training/topics/safety/failure-modes/
License: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/

*Safety* · topic `safety/failure-modes`

Model failures rarely announce themselves; they arrive as fluent, confident text. This topic shows what hallucination looks like in real documents and code, how bias enters and repeats, and how people drift into overreliance and automation complacency once a tool is usually right.

## Concepts

- **Hallucination in practice**: What invented output looks like in daily work: a statute with a real number and wrong contents, a library method that almost exists, a quote attributed to the right person from a speech never given, a summary that adds a point the source did not make. The tell is specificity without a checkable source. Treat precise details from a model as claims to verify, not facts. [glossary](https://schubergphilis.github.io/ai-training/glossary/#hallucination-in-practice)
- **Bias**: Systematic slant in a model's output inherited from its training data and its tuning: which names it assumes are senior, which dialect it calls professional, which candidates it ranks higher for the same record. Bias is hard to see in a single answer and shows up in patterns across many. Anything a model does that affects people should be sampled and compared across groups. [glossary](https://schubergphilis.github.io/ai-training/glossary/#bias)
- **Overreliance**: Trusting a model's output beyond what you have checked or could check. It starts with tasks you know well and then spreads to ones you do not, where you can no longer tell a good answer from a plausible one. The remedy is to keep verifying in proportion to stakes and to keep enough of your own competence to recognize when the tool is wrong. [glossary](https://schubergphilis.github.io/ai-training/glossary/#overreliance)
- **Automation complacency**: The habit of waving through a tool's output because it has been right many times before. Attention decays fastest when the tool is good, so the rare failure passes unchecked. It is a known effect from aviation and medicine, not a personal weakness, and is countered by process: designed checks, sampling and rotating who reviews. [glossary](https://schubergphilis.github.io/ai-training/glossary/#automation-complacency)

## Links

- **Builds on:** [Capabilities and limits](https://schubergphilis.github.io/ai-training/topics/concepts/limits/)
- **Leads to:** [Verifying outputs](https://schubergphilis.github.io/ai-training/topics/safety/verification/)
- **Competencies drawing on it:** [Verifies AI output before relying on it](https://schubergphilis.github.io/ai-training/competencies/safety/verifies-output/)

## Lessons

- [Seeing bias across many answers](https://schubergphilis.github.io/ai-training/safety/bias-in-patterns/) (tutorial)
- [What an invented fact looks like](https://schubergphilis.github.io/ai-training/safety/spotting-hallucination/) (tutorial)
- [When the tool is usually right](https://schubergphilis.github.io/ai-training/safety/when-the-tool-is-usually-right/) (explanation)

## Sources

- `Academy ai-capabilities-and-limitations` [AI capabilities and limitations](https://academy.claude.com/courses/ai-capabilities-and-limitations), Claude Academy (course)
- `Parasuraman 2010` [Complacency and Bias in Human Use of Automation: An Attentional Integration](https://doi.org/10.1177/0018720810376055), Human Factors 52(3) (2010), 381-410 (paper)
- `Goddard 2012` [Automation bias: a systematic review of frequency, effect mediators, and mitigators](https://doi.org/10.1136/amiajnl-2011-000089), Journal of the American Medical Informatics Association 19(1) (2012), 121-127 (paper)
- `Bainbridge 1983` [Ironies of automation](https://doi.org/10.1016/0005-1098(83)90046-8), Automatica 19(6) (1983), 775-779 (paper)
- `NTSB Asiana 214` [Descent Below Visual Glidepath and Impact With Seawall, Asiana Airlines Flight 214, Boeing 777-200ER, HL7742, San Francisco, California, July 6, 2013 (AAR-14/01)](https://www.ntsb.gov/investigations/AccidentReports/Reports/AAR1401.pdf), National Transportation Safety Board (reference)