Why your AI answers one question well and fumbles the next
A 30-person insurance brokerage asks their assistant what the renewal window is for commercial auto. Four seconds later they have the right answer and the right clause. Ten minutes after that, someone asks which clients have renewals falling inside a rate change. The same assistant returns something vague and slightly wrong.
The owner draws the obvious conclusion. The AI is unreliable. It is not. Nothing about the system changed between those two questions. What changed is what the system managed to find before it started writing.
It is usually a retrieval problem, not a broken model¶
Two things happen when you ask a question. The system searches your documents for relevant passages. Then it writes an answer from those passages.
Almost every inconsistent answer comes from the first step, not the second. The writing step is usually fine. It was just handed the wrong three paragraphs.
That distinction matters because it tells you where to look. You do not have a model problem. You have a findability problem.
How you phrase the question changes what gets found¶
The search runs on meaning, and your words are the only signal it has.
"Renewal window" worked because that phrase sits in the policy document. "Rate change exposure" is internal shorthand that appears nowhere in the files. Same person, same knowledge base, very different result.
Specificity works in both directions. Ask something too broad and you get a shallow answer stitched from ten weak matches. Ask using a term only your team uses and you get nothing solid at all.
How the document is written decides what can be found¶
Retrieval pulls out pieces of documents, not whole ones. So the question is whether a piece still makes sense on its own.
A section with a clear heading and a self-contained paragraph survives that. A paragraph that opens with "in that case, the same rule applies" does not. It is meaningless once separated from the page above it.
This is why Start with documents, not AI is the practical starting point. Structure is not tidiness. It is what makes an answer retrievable.
Questions that span several documents are the hard ones¶
A question that sits in one place needs one successful lookup. A question that spans four documents needs four, plus the knowledge that those four relate to each other.
Nothing in your files states that relationship. So the system finds the strongest single match and answers from that. You get a partial answer that reads like a complete one, which is the version most likely to slip past.
What you can fix without touching a single setting¶
Four things, all on your side of the screen:
- Use the words your documents use, not your internal shorthand
- Ask one question at a time instead of bundling three
- Name the document when you know which one holds the answer
- Add a short plain summary at the top of any file people query often
None of that requires configuration. It changes what the search has to work with.
Telling "it could not find it" apart from "it made it up"¶
Ask where the answer came from. The response tells you which problem you have.
A retrieval miss hedges. It stays generic, offers no source, or points at a document that does not cover the topic. A fabrication does the opposite. It is confident and specific, and it cites a real document that does not actually say that.
Not the same fault. Not the same fix. The first one is a content and structure problem you solve by rewriting the source. The second is a guardrail problem, and no amount of tidying your documents will touch it.
Keep exploring¶
Most inconsistent answers are a findability problem wearing a reliability costume, and the fix usually starts with the source document. RAG in Plain English covers how the lookup works underneath, or you can just contact FIT and talk through what your system is getting wrong.