RAG Doesn't Make AI a Better Tester. It Makes It a Better-Briefed One.

Like any other IT company on the planet we have been continuously running our experiments with AI. We wired up an AI to generate tests for one of our services, pointed it at the repository, and let it work. The tests looked good. They compiled, they read cleanly, a reviewer skimming them at the end of a long day would have approved most of them.

Few weeks later we understood why they were quietly wrong.

The model kept writing assertions against a component we had deprecated months earlier. It missed a business rule that lived in a Jira ticket nobody had surfaced to it. It cheerfully duplicated coverage we already had, because it had no idea our existing suite existed.

None of that was an intelligence problem. The model was perfectly capable. It was a briefing problem. I had handed a smart stranger a system they had never been told anything about, and then acted surprised when they got the details wrong.

That is the problem Retrieval-Augmented Generation is meant to solve. And after a good deal of experimenting, I have come to think of RAG for testing as exactly that: an automated briefing. How well you brief the model decides how useful its output is. Like any briefing, there is a ceiling on what it can buy you. I will come back to that ceiling at the end, because it is the part most people building this stuff would rather not discuss.

Here is what I found along the way.

What RAG is actually doing here

Strip away the acronym and RAG (Retrieval Augmented Generation) is simple. Instead of relying only on what the model learned during training, you fetch relevant material at the moment you ask the question and hand it to the model alongside the prompt.

For testing, that matters because the model, on its own, does not know:

• how your sale component actually behaves

• that ticket PROJ-4521 changed the checkout flow last sprint

• that your suite already covers the happy path but not the timeout case

• that the design spec says the button stays disabled until validation passes

  
  

Without retrieval, the model guesses. With good retrieval, it works from roughly the same context a senior engineer on your team would have before writing a single test.

The trouble is that "we added RAG" has become as hollow a phrase as "we added AI." People connect a repository, embed some files, and call it done. That is not a briefing. That is dropping a stack of unsorted files on someone's desk and walking away.

Chunking: the part that decides everything

Before you can retrieve anything, you have to break your sources into pieces. This step gets almost no attention, and it is where most of our early attempts fell apart.

The naive approach is to split on fixed character counts: every thousand characters, with a small overlap. It is easy, and it destroys meaning. You end up with fragments that begin mid-sentence, lose track of which function or class they belong to, and embed as noise.

Two things fixed this for us.

For prose (documentation, tickets, existing test cases), split on structure first: headings, section boundaries. Then apply a sliding window with a modest overlap so an answer never gets amputated at a chunk boundary. And prefix every chunk with its breadcrumb. Rather than embedding a bare line like “Returns 401 if the token is expired”, embed it as:

[Auth Service > Token Validation > Error Handling]

Returns 401 if the token is expired

Now the fragment carries its own address. Without that, the same sentence appearing in two different modules becomes indistinguishable in the embedding space, and retrieval starts confusing them.

For code, fixed-character splitting is worse than useless. It severs signatures from bodies and cuts class definitions in half. Split on declaration boundaries instead (function, class, interface, top-level const) and keep merging adjacent declarations until you hit a token budget, then start fresh. The result is chunks that are complete units of meaning. The model gets a whole function, not the first sixty percent of one. This is what we call contextual chunking. 

Get chunking wrong and nothing downstream can save you. You are briefing the model with torn pages.

The four kinds of RAG

This is where the real learning happened, and where I want to spend the most time. There is a tendency to treat RAG as one thing you either have or you do not. In practice it is a ladder, and each rung is a meaningfully better briefing than the one below it.

1. Naive RAG

Chunk the documents, embed them, retrieve the top handful by vector similarity, paste them into the prompt. Done.

This is where nearly everyone starts, and it works well enough on prose that people convince themselves the problem is solved. It is not. Naive RAG has three failure modes we hit repeatedly:

• It retrieves once and never reconsiders. If the first pull is thin, the answer is thin.

• It has no sense of the terms that matter. A query phrased conceptually will miss a chunk that only matches on an exact identifier.

• It ranks purely on semantic closeness, so a chunk that merely sounds related outranks one that is precisely relevant.

Naive RAG is a briefing assembled by someone who skimmed the request, grabbed the first few documents that felt on-topic, and left the room. Sometimes that is enough. On a real codebase, it usually is not.

2. Hybrid and "advanced" RAG: fetch on meaning and on terms

The single biggest jump in quality for us came from admitting that semantic search alone fails on code.

Take the query "how does checkout handle a payment failure?" Vector search finds things that mean something similar: handlePaymentFailure(), onPaymentError(), and so on. Good. But PAYMENT_DECLINED_CODE = 4021 might not surface at all, even though it is exactly the context a test needs.

That is what keyword ranking, specifically BM25 algorithm, catches. It has been quietly powering search engines for decades. It matches exact terms, which is precisely what code is full of. The catch: you have to tokenise intelligently, splitting camelCase and snake_case into their parts, or handlePaymentFailure will never match a query containing the word "payment."

Run both retrievers in parallel and fuse their rankings with Reciprocal Rank Fusion.

A chunk that ranks well on both lists rises above one that ranks first on vectors alone and is absent from the keyword list entirely. One practical habit: over-fetch. Ask each retriever several times your target, fuse, then trim. Neither method has full coverage, so the overlap is where the good material lives.

Beyond fusion, this rung is where the techniques people now label advanced RAG sit, and we used two in our implementation:

• Re-ranking. After retrieval, pass the candidates through a cross-encoder that scores each chunk against the query directly. It is slower per item, so you only run it on the shortlist, but it reorders that shortlist far more accurately than the first-pass score.

• Query rewriting. A tester's question is often underspecified. Rewriting it, or generating a hypothetical answer and retrieving against that (the HyDE - Hypothetical document embeddings trick), pulls back material the literal query would have missed.

  
  

Hybrid RAG is a briefing from someone who understood both the idea you asked about and the exact terms that matter, and who took a second pass to put the most relevant material on top.

3. Graph RAG: fetch what is connected, not just what is similar

This is the rung that turns decent test generation into genuinely useful test generation, and it is really two distinct ideas that people carelessly lump together. Separating them mattered.

The similarity graph handles implicit relationships. After embedding every chunk, compute the cosine similarity between all pairs and connect each chunk to its closest neighbours. At retrieval time, when you pull a chunk, you also pull its neighbours from this graph. So a query that retrieves the login redirect logic also surfaces the session-token handling and the redirect-URL validation, even if those did not score highly on their own.

This is powerful for testing precisely because test behaviour is never isolated. A login test is not only about the login endpoint. It touches session management, auth middleware, redirect rules, and the fallback states. The similarity graph surfaces those connections without anyone hand-curating them.

The knowledge graph handles explicit relationships. Your tickets have parent, subtask, and epic links. Your test cases cover relationships to requirements. Your commits reference ticket IDs. Extract those and build a real graph:

• PROJ-4521 (epic) links to PROJ-4522, PROJ-4523 (subtasks)

• test_checkout_flow.ts links to PROJ-4521 via an // implements PROJ-4521 comment

• test_case_887 links to PROJ-4523 

Now, when a chunk is retrieved, do a one-hop expansion and pull its direct neighbours. The code file arrives with its ticket. The ticket arrives with its acceptance criteria. This is the point where retrieval stops behaving like a search engine and starts behaving like a test engineer who owns the traceability matrix.

Graph RAG is a briefing from someone who, when they hand you a ticket, instinctively hands you the epic it belongs to and the code that implements it, because they know you will need all three.

4. Agentic RAG: fetch, judge the gap, fetch again

The rung we are still experimenting and most cautious about. Instead of retrieving once, the model retrieves, inspects what it has, notices what is missing, and goes back for more, in a loop, until it decides it has enough.

When it works, it is impressive. Ask it to test a flow that spans three services and it will chase the thread across all three rather than stopping at the first. It behaves less like a lookup and more like an investigation.

The loop is only as good as the model's own judgement about the gap in its knowledge. It can convince itself it is done when it is not. It can also spiral, retrieving in circles and burning tokens on a question that a keyword match would have answered in one pass. Agentic retrieval is a genuine step forward. It is not a substitute for knowing when to stop, and right now that self-awareness is the least reliable part of the whole stack.

What it looks like when the rungs stack up

Say you prompt your AI test framework to generate a test for "user checkout with an expired payment card." On a mature setup, here is roughly what fires:

1. The query is recognised as a specific retrieval, not an enumeration, so the retrieval budget is set accordingly.

2. Vector search finds the checkout, payment, and error-handling chunks by meaning.

3. BM25 finds the chunks containing "expired," "card," and "decline" by term.

4. Fusion promotes the chunks that appeared on both lists.

5. The similarity graph expands to the neighbours: the post-failure session state, the retry logic, the UI error component.

6. The knowledge graph pulls in PROJ-4521, "Handle expired card gracefully," which carries the acceptance criteria naming the exact message the test should assert.

The model now writes a test that calls the right endpoint, expects the right error code, asserts the criteria from the ticket, and checks the message the graph-linked front-end chunk specifies. That is not magic. That is a well-briefed engineer.

The ceiling of AI

Here is the part which most people know but struggle to explain properly.

Everything above makes the model better informed. None of it makes the model wiser. Retrieval decides what the model knows about your system. It does not, and cannot, decide what is worth testing in the first place.

A perfectly briefed junior engineer, handed every ticket, every design doc, and the full existing suite, will still write the tests that are easy to see and miss the one that matters: the edge case that only shows up under load, the interaction nobody wrote a ticket for, the assumption baked so deep into the system that no document states it out loud. Knowing what could break, and what it would cost the customer if it did, is not a retrieval problem. It is judgement. It is the part of testing that is cognitive, not clerical.

That is why I keep coming back to the same position on AI in testing. It is a genuinely useful tool, and RAG is a genuinely useful way to make it less ignorant. Use it. Build the pipeline properly. But do not confuse a better-briefed model with a better tester. The briefing is engineering. The judgement about what is worth the briefing is what makes a great test engineer.

Get the retrieval right and your AI stops hallucinating deprecated components and starts writing tests that read like someone who knows your system.

Deciding which tests are worth writing at all? That job has not moved. It is still sitting on your side of the desk, and I suspect it will stay there for a long time. But breaking down what AI can do and what it can’t is an important step for test engineers to prove their value.