Skip to main content
This evaluation compares GraphRAG with traditional RAG on the same benchmark corpus and the same question set. The goal was to measure whether graph-aware retrieval improved multi-hop question answering.

​
What was implemented

The evaluated system did not use flat chunk retrieval alone. It implemented GraphRAG with these stages:
  1. Split source documents into chunks.
  2. Extract entities and relationships from those chunks.
  3. Build a knowledge graph from the extracted entities and relationships.
  4. Run standard retrieval to get seed evidence for a user question.
  5. Expand through graph links to collect connected evidence.
  6. Re-rank the final evidence set before answer generation.
That is the GraphRAG behavior that was evaluated.

​
What it was compared against

The comparison used a traditional RAG baseline with the same corpus and the same answer model. The only difference between the two runs was retrieval mode:
  • Baseline RAG: flat chunk retrieval only
  • GraphRAG: graph-seeded expansion plus graph-aware reranking
This matters because it isolates the effect of GraphRAG itself.

​
Benchmark design

The benchmark was designed to test multi-hop retrieval rather than simple one-chunk lookup. It used:
  • three target knowledge graphs
  • nine semantically similar distractor graphs
  • 108 short factual documents
  • 24 questions that required linking facts across multiple chunks
This design is important. If every answer already appears in one obvious chunk, GraphRAG will not show much benefit over standard RAG.

​
Evaluation method

Both retrieval modes were run over the same benchmark corpus and the same question set. Both then used the same answer-generation model and the same answer prompt. The evaluation recorded three metrics:
  • Exact match: whether the final answer exactly matched the gold answer
  • Token F1: token overlap between the final answer and the gold answer
  • Evidence recall@5: how much of the required supporting evidence appeared in the top 5 retrieved chunks

​
How the evaluation was run

The evaluated flow was simple, and identical for both runs — the same store, the same documents, and the same question set. Only the retrieval mode changed.
  1. Create a graph-enabled store. The store was configured with an embedding model, an entity/relationship extraction model, a chunk size of 800 tokens with 120 overlap, and a maximum graph traversal depth of 2 hops.
  2. Ingest the corpus. Each of the 108 benchmark documents was chunked and embedded, and entities and relationships were extracted from every chunk to build the knowledge graph. For example, “Rivera Logistics won the Northern Bridge Sensors contract” and “Atlas Infrastructure Group owns Rivera Logistics” come from different documents, but extraction links them through the shared Rivera Logistics entity.
  3. Run every question in baseline mode. Flat retrieval: embed the question, retrieve the top-scoring chunks, and answer from them.
  4. Run every question in graph mode. Same question, same store: retrieve seed chunks, identify the seed entities they mention, expand up to 2 hops through the graph to collect connected evidence, and re-rank the expanded evidence set with graph signals before answering.
Step 3 versus step 4 is the key comparison. The query, corpus, and answer model stayed the same — only the retrieval mode changed:
  • baseline = traditional flat retrieval
  • graph = graph-seeded expansion plus graph-aware reranking

​
Measured results

The live benchmark run produced the following results: Improvement:
  • Exact Match: +37.5 points
  • Token F1: +35.4 points
  • Evidence Recall@5: +16.0 points

​
Acceptance threshold

The benchmark used the following pass criteria:
  • exact match improvement of at least +5.0 points
  • evidence recall@5 improvement of at least +10.0 points
The evaluated GraphRAG implementation passed both thresholds.

​
Representative question-level outcomes

Examples where GraphRAG succeeded and baseline RAG did not:
  • “Who is the chief financial officer of the company that owns the Northern Bridge Sensors contract winner?”
    • Baseline RAG: unknown
    • GraphRAG: Javier Nanda
  • “Which company acquired the firm that prepared a risk report for Meridian Ports Authority?”
    • Baseline RAG: unknown
    • GraphRAG: Atlas Infrastructure Group
  • “Which company owns the company that won the Delta Reach Sensors contract?”
    • Baseline RAG: unknown
    • GraphRAG: Bluepeak Transit Group
These are multi-hop questions. They require linking facts across connected entities rather than retrieving a single directly matching chunk.

​
Why GraphRAG performed better

Baseline RAG retrieved semantically similar chunks, but it sometimes failed to retrieve the connected evidence needed to complete the reasoning chain. GraphRAG improved performance by:
  • identifying the relevant seed entities from the question
  • traversing graph relationships to find linked evidence
  • reranking the final evidence set with graph signals in addition to semantic similarity
That is why the improvement appears most clearly on multi-hop questions.

​
Summary

On this benchmark, GraphRAG outperformed traditional RAG on both final-answer accuracy and supporting-evidence retrieval. The strongest gains appeared on questions that required linking facts across multiple connected entities. In the MKA1 platform, graph-aware retrieval is delivered through vector stores — the platform’s retrieval surface, which owns tenancy, usage metering, and billing — as a retrieval mode alongside semantic and hybrid search, rather than as a separate product API. To turn it on, create a store with retrieval_mode: "graph". See Create a graph store for the full request and for what changes once a store is in graph mode.