A sales RAG system can be used, appreciated by its users, and still leave a hard business question unanswered.
What did it change?
I had the opportunity to look at one after roughly a year in production. People used it. The feedback was good. A salesperson could ask a question in plain language and receive a relevant document instead of working through years of folders and documents. That is what the review established. It did not establish universal answer accuracy, corpus freshness, or a clean commercial attribution.
The system cost roughly $30K in capital expenditure and about $4K a year to operate.
It was not a failed project.
But it was not possible to make a rigorous claim about the sales impact either.
That distinction is easy to miss in AI work because we are usually very good at measuring the technology after it arrives. Queries. Latency. Retrieval quality. Active users. Feedback. Infrastructure cost.
Those are useful measures. They do not automatically answer the question the business originally cared about.
Useful Is Not the Same as Attributed
The sales team needed evidence quickly.
What have we delivered that looks like this customer’s problem?
Which projects demonstrate this capability?
What capacity is available?
Who inside the company knows the history behind the proposal?
Before the RAG layer, the answers existed, but they were distributed across documents, folders, and people who had accumulated the context over years. The system reduced the time needed to find a relevant starting point. It also reduced the need to interrupt the colleague who happened to know where the answer lived.
That is useful.
It may even be commercially valuable.
But a sales result sits inside a noisy system. Pricing changes. Competitors change. A major account appears or disappears. A new sales leader changes qualification. The market moves. A sales RAG layer can be part of a better outcome without being the part that can be cleanly isolated afterwards.
The wrong reaction is to declare that the project has no value because revenue attribution is difficult.
The other wrong reaction is to declare success because people like the interface.
A serious operating review has to leave room for a third answer: the system is useful, the value hypothesis is plausible, and the evidence is insufficient to prove the result it was funded to create.
That is not failure. It is a measurement finding.
The Benefit May Have Moved
One effect was easier to see than a revenue change.
Experienced salespeople used the system differently from new ones. Someone who has spent ten years in a business often knows what the company has delivered, who can answer a difficult question, where the exceptions are, and which customer story is relevant. They may use the system for confirmation or speed.
A new salesperson starts without that private map.
For them, the important questions are more basic:
- What have we delivered?
- For whom?
- What can we realistically take on?
- Where is the evidence for the claim I am about to make?
The RAG system appeared to help a new person reach a credible customer conversation sooner. That is an onboarding hypothesis, not a completed proof. We did not establish the baseline that would let us calculate the effect with confidence.
But it is a different value hypothesis from sales productivity.
That matters because a project can be judged against the wrong outcome long after the project has started producing a different kind of value. A knowledge system may not move win rate in a measurable way. It may reduce ramp time, preserve knowledge through turnover, improve proposal quality, reduce interruptions to senior people, or make commercial claims easier to support.
If the sponsor only asks for revenue attribution, all of that may become invisible.
The Baseline Was the Missing Asset
The system was measured after deployment. The work was not measured before deployment.
That is the real lesson.
Before a knowledge or RAG project starts, the team should decide what it expects to change and record the current condition. Not a generic statement that people spend too much time searching. A specific operational picture.
How long does a salesperson spend looking for evidence before a customer call?
How often do they interrupt a senior colleague for a project reference, capacity answer, or technical fact?
How much senior-sales time is consumed by repetitive retrieval work?
How long does it take a new salesperson to hold a credible conversation independently?
How often does a proposal omit evidence that already exists somewhere in the company?
The answers do not need false precision. They need to exist before the intervention changes the workflow.
Otherwise, twelve months later, a sponsor can see that the system is active and useful but cannot compare it with the work it replaced. The organization has created a technology metric without an operating metric.
Choose the Outcome Before the System Arrives
Not every AI project should be forced into a direct revenue calculation. A sales knowledge system may be intended to reduce search time, shorten the path to an independently credible customer conversation, reduce interruptions to senior people, or improve the evidence available for a proposal. Those are different claims and need different evidence.
The sponsor should choose the primary outcome before deployment. Secondary effects can be recorded, but they should not be promoted afterwards to rescue a primary claim that cannot be proven. In this case, the production review suggested an onboarding hypothesis that the original sales-productivity framing had missed.
The answers do not need false precision. They need to exist before the intervention changes the workflow.
The Cost Is Also More Than the Build
The visible cost was not the whole cost.
The project had a build cost and infrastructure cost. A continuing system also has a corpus to maintain, source material to update, and a technical stack that will change as models and surrounding tools change.
For a sales corpus, production success has another boundary as well. The system needs to return material only to people authorised to see it, preserve the source and freshness of what it presents, respect confidentiality and retention obligations, and leave enough of an audit trail to explain what was retrieved for a commercial decision. Fast retrieval that crosses those boundaries is not a successful knowledge system.
Those are not arguments against the system. They are arguments against reporting only the cost that appears in the original project plan.
A useful AI business case should therefore have two boundaries:
First, the outcome boundary. What will the business count as evidence that the system improved useful work?
Second, the operating boundary. What will it cost to keep the knowledge current, quality-controlled, permissioned, and available after the launch team moves on?
Without both, the organization can easily compare a one-time build cost with an imagined permanent benefit.
Measure Before You Need to Defend It
The pressure to measure usually arrives after deployment, when someone asks whether the investment worked.
By then, the cleanest comparison may already be gone.
The team has changed its habits. New hires have arrived. Documents have been reorganized. The people who used to answer every question have adjusted their work. The company may still learn from a retrospective, but it cannot reconstruct the baseline with the same confidence.
This is why I would start a new AI project with a simple question:
What number should be different if this succeeds?
Then ask what it is today, who owns it, how it will be measured, what will count as an acceptable change, and which other factors might make attribution difficult.
Only after that should the team decide whether the intervention is RAG, a workflow redesign, conventional search improvement, training, or no project at all.
A working AI system deserves more than applause from the people who use it.
It deserves an honest account of the work it changed, the evidence it produced, and the value that still has to be proved.