# Following a claim back to its source > Walk two references in an AI-written paragraph back to the reports they name, find the one that does not exist and the one that says something else, and learn what to do with a claim you cannot check in time. Page: https://schubergphilis.github.io/ai-training/safety/following-the-source/ License: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/ In this lesson we take a paragraph a chat assistant wrote for a briefing, with two references after its claims, and walk each reference back to the report it names. One of the two reports turns out not to exist, and the other says something different from what the paragraph says it says. The previous lesson's routine has "open one link" as its third check. This lesson is what you do once the link is open, and what you do with a claim you cannot open at all. ## References that look right A colleague asks a chat assistant for "a paragraph on AI adoption at work for the intro of the briefing, with sources". No assistant wrote this paragraph. The author of this lesson wrote it, and its two references with it, so that one names a real report and one does not. > Adoption is now the norm rather than the exception. The Stanford AI Index > found that 78% of organizations that used AI in 2024 reported a > measurable productivity gain (Stanford HAI, *The 2025 AI Index Report*). > The effect reaches the desk: workers who use AI daily save an average of > 5.4 hours a week (Nordholm Institute, *Desk Work and AI Survey 2024*). The paragraph has everything a careful writer would put in. Every claim comes with a number, and every number comes with a report, a publisher and a year. One of the two reports is real and one is not. Before reading on, decide which is which, and write down what you would do to find out. #### Checkpoint: Which citation is which? Which statement about the two references is true? - Both reports exist, and both findings are in them. - The AI Index exists and states the finding. The Nordholm survey does not exist. - The AI Index exists with a different finding. The Nordholm survey does not exist. - The AI Index does not exist. The Nordholm survey exists with a different finding. A short program written for this lesson looks each citation up in a small list of reports, and for the ones it finds, prints what the report states next to what the paragraph says it states. This is what it prints. ```text FOUND Stanford HAI, The 2025 AI Index Report text says: 78% of organizations that used AI in 2024 reported a measurable productivity gain report says: 78% of organizations reported using AI in 2024, up from 55% the year before NOT FOUND Nordholm Institute, Desk Work and AI Survey 2024 text says: desk workers who use AI daily save an average of 5.4 hours a week citations: 2 reports found: 1 claims that match the report: 0 ``` The Nordholm Institute and its survey do not exist. The name was invented for this lesson, and a web search for it finds nothing. The 2025 AI Index Report by Stanford HAI (the Stanford Institute for Human-Centered Artificial Intelligence) is real, and you can open it (The 2025 AI Index Report, Stanford Institute for Human-Centered Artificial Intelligence (Stanford HAI)). Its public page says that 78% of organizations reported using AI in 2024, up from 55% the year before. That is an adoption figure. The paragraph turned it into a productivity figure, "78% of organizations that used AI reported a measurable productivity gain", which the report's page does not say. The page does say, in general terms, that research finds AI boosts productivity. No percentage is attached to that sentence, and the 78% is not it. The finding in the paragraph is invented, and the number in it is real. This is the harder case of the two, because the number checks out if you only search for it. The lesson [What an invented fact looks like](https://schubergphilis.github.io/ai-training/safety/spotting-hallucination/#specificity-is-the-tell) showed that a specific without a source is the tell. Here every specific has a source, and the question moves from "is there a source?" to "does the source say this?". The first question a search answers. The second takes the walk in the next section. ## The walk **Source checking** is the walk from a claim to the passage that is supposed to support it, and back. It has four steps, and each one can end the walk. 1. **Open the source itself.** Open the report or the law or the article as published. A page that mentions it, or the assistant's summary of it, is a step short. If you cannot find it by its title and publisher, the walk ends here, and the claim is unverified. 2. **Find the passage.** Search the source for the number, the name or the quoted words. A 300-page report has a summary page and a search box, and the specific in the claim is what you search for. If the specific is nowhere in the source, the walk ends here too. 3. **Read what the passage says.** Read the sentence around the number and the sentence before it. What does the number count? Who was asked, and when? The 78% in the AI Index counts organizations that use AI. The paragraph made it count organizations that gained from AI. 4. **Compare the two sentences.** Write down what the source says and what the text says it says, one sentence each. If they match, the claim stands. If they differ, the text changes to match the source, or the claim comes out. The previous lesson, [Checking habits that run when you are in a hurry](https://schubergphilis.github.io/ai-training/safety/checking-habits/#the-routine), said to open one link. That was to find out whether the rest deserve the same treatment. Both references failed here, so every other reference in that text would get the same walk before it went out. Anthropic's AI Fluency course puts this kind of judgment under discernment, the skill of evaluating what a model produced rather than accepting it (AI Fluency: Framework and foundations, Claude Academy). The walk is discernment applied to one reference at a time. #### Pitfall: Stopping at the title match Your colleague searched for "2025 AI Index Report", found the Stanford page in the first result, saw "78%" in its summary, and marked the reference as checked. The report exists, the number is in it, and the claim is still wrong, because the number counts a different thing. A title match tells you the shelf is real, and a number match tells you the book is on it. Only reading the passage tells you the book says what the text says it says. When you find the source, keep going until you have the sentence, and compare it with yours. ## When the walk cannot finish Sometimes the walk stops before the comparison. The report is behind a paywall your team does not have. The source is a book with no page number and no search. The publisher exists but the report is not on its site. The briefing goes out in an hour. In each case the claim is one you have not checked, and the text should say so or stop making it. The choice is between two honest states. **Mark it unverified.** Leave the claim in with a note that names what you could not do: "cited to the AI Index, not confirmed against the report". The reader then knows what weight to give it, and the note is a task for whoever picks the text up next. Or **take it out.** If the text stands without the claim, remove the claim and its reference. A briefing with one fewer statistic loses less than a briefing with one wrong statistic that a reader repeats in a meeting. What you never do is leave the reference in as it stands. A reference that nobody has opened looks the same as one that has been checked, and the reader cannot tell them apart. Harvard's CS50 course has an AI helper that answers questions on the course forum (CS50's Introduction to Computer Science, Harvard CS50) (Teaching CS50 with AI: Leveraging Generative Artificial Intelligence in Computer Science Education, Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 (SIGCSE 2024), 750-756). A staff member can endorse, amend or delete each of those answers, so students can see which answers a person has checked and stands behind (Teaching CS50 with AI: Leveraging Generative Artificial Intelligence in Computer Science Education, Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 (SIGCSE 2024), 750-756). Your text needs the same distinction, and "unverified" is how you make it when you have no time to check. #### Checkpoint: The report you cannot open The briefing goes out in an hour. One claim cites a consultancy report. The consultancy exists, the report is not on its website, and nobody on your team has a copy. What do you do with the claim? - Leave the claim and the reference as they are. The publisher exists, so the report probably does too. - Replace the reference with a link to a news article that quotes the same number. - Keep the claim and add a note that says the report could not be confirmed before sending. - Ask the assistant to confirm the report exists and paste its answer as the check. ## Exercise Open the 2025 AI Index Report's public page (the References section below links it) and find the passage the paragraph in this lesson relies on. Write one sentence with what the report says, and one sentence with what the paragraph says it says. Then write one line that names the gap between them. Ten minutes is enough. A good result is three lines. The first says what the 78% counts in the report. The second says what the paragraph made it count. The third names the change, in the words you would use to tell the colleague who wrote the briefing. If you had only searched the page for "78%", would you have found the gap? Stretch: Do the same walk for one reference in a document you or a colleague wrote with an assistant this month, and write the same two sentences. ## Recap 1. A reference can fail in two ways. The source does not exist, or the source exists and says something else. The second is harder to catch, because the title and the number both check out. 2. Source checking is a walk: open the source itself, find the passage, read what it says, and compare it with what the text says it says, one sentence each (AI Fluency: Framework and foundations, Claude Academy). 3. Stopping at a title match or a number match is the pitfall. Only the sentence around the number tells you what it counts. 4. A claim you cannot check before the deadline is marked unverified or taken out. It never stays in looking like a checked one. 5. Anything with a page number, URL, statute or study behind it gets opened before it goes to anyone else. ## References - The 2025 AI Index Report, Stanford Institute for Human-Centered Artificial Intelligence (Stanford HAI), https://hai.stanford.edu/ai-index/2025-ai-index-report - AI Fluency: Framework and foundations, Claude Academy, https://academy.claude.com/courses/ai-fluency-framework-foundations - CS50's Introduction to Computer Science, Harvard CS50, https://cs50.harvard.edu/x/ - Teaching CS50 with AI: Leveraging Generative Artificial Intelligence in Computer Science Education, Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 (SIGCSE 2024), 750-756, https://doi.org/10.1145/3626252.3630938