Your AI reads the company's documents and the answers aren't good enough. Before you spend another quarter on prompts, spend ten minutes finding out which problem you actually have.
At forty documents the answer is always in the context, so every problem really is a prompt problem. At a hundred thousand it often isn't, and from there your model has two available behaviours.
Prompting slides you between those two failures. It never reaches right, because that needs the correct document in front of the model. An architecture question, not a wording one.
Real questions it answered badly, from the people who use it. Include the embarrassing ones.
Locate the document that contains it. If you can't either, stop: nobody wrote it down, and no system was ever going to produce it.
Was the right document in what retrieval returned? Glance at the extracted text, not the original file. That's what the model read.
Paste the correct document in yourself. Same question, same prompt, one document, no noise.
How many come back right now? Intermittent is a result in itself, and one pass will lie to you.
The model was always capable. It never received what it needed. No amount of prompt engineering fixes this.
Upstream is wider than most teams expect, and each layer costs something different to fix.
Instruction clarity, output format, question framing and task design are the right spend, and you're making it with evidence.
One check first: if your pipeline turned that document into soup, you weren't testing the model at all.
Most teams reach for RAG without asking whether their data was ready to be searched. It looks fine, right up until the run where it isn't. Working out which upstream layer you're in, and what each costs, is the first hour we spend with any client.
A standing rule worth having: nobody accepts an "improve the AI's answers" ticket without running this first. Ten minutes, and it routinely redirects a quarter of planned work.
The same question sits under every AI idea in your business, including the ones still on a whiteboard: does this get more expensive and less accurate as the data grows, or does it hold? We triage that on your list in a free call, and we'll say so if there's nothing there for you.
