Evaluating Codebase-Oriented RAG through Knowledge Graph Analysis [Part 1]

One of the emerging applications of retrieval-augmented generation (RAG) systems is RAG for codebase analysis. 

In this setting, static analysis pipelines – combining parsing, AST construction, symbol resolution, and dependency extraction – are used to construct a knowledge graph from the structure of a source code repository, including files, packages, classes, functions, and other structural elements. 

This knowledge graph enriches the context provided to connected large language models, enabling them to answer questions about the code with improved grounding and significantly reduced structural hallucination. In more advanced workflows, such systems can also assist in proposing concrete modifications to the code.

In this two-part blog series, our goal will be to analyze the knowledge graph produced by such systems. We’ll approach this through a series of queries that pursue two main objectives:

• In Part 1 (today), we perform a codebase architectural analysis, assuming the generated knowledge graph is correct, to understand what structural insights can be derived from it. 
• In Part 2 (next week), we critically evaluate the structural integrity of the knowledge graph itself (constructor validation) to assess how faithfully the graph represents the original codebase and explain some notes on expressiveness and semantic completeness, investigating whether the graph captures sufficient semantic richness to support deeper reasoning tasks.

Throughout this post, we’ll use Code-Graph-RAG as a case study, as it represents a recent and notable tool in this space. As the subject codebase, we’ll use Soufflé, a well-known Datalog language and engine. This choice is intentional: it provides a relatively complex C++ codebase with diverse input/output workflows, parallel execution mechanisms, and computationally intensive logical algorithms, making it a suitable benchmark for evaluating the strengths and limitations of codebase-oriented RAG systems.

All right, let’s begin.

Using gdotv for Knowledge Graph Analysis & Evaluation

Knowledge graphs generated by RAG-based code analysis tools are typically used behind the scenes. 

Although these systems usually allow exporting or inspecting the graph (often through command-line interfaces) the graph itself is rarely treated as a first-class, explorable artifact. With a general-purpose graph exploration tool such as gdotv, however, these knowledge graphs can be analyzed visually and interactively, enabling deeper structural inspection and exploratory validation. In this experiment, we use the latest version of gdotv to systematically analyze and evaluate the generated knowledge graph.

Setup

In this analysis, we first need to set up Code-Graph-RAG, which in turn requires configuring the connected language model, preparing the target codebase, and generating the corresponding knowledge graph. To explore and analyze the resulting graph interactively, we also require gdotv.

Setup Code-Graph-RAG

First, clone the repository and install the required system dependencies:

clone the repo `git clone https://github.com/vitali87/code-graph-rag.git`

Then install required native tools:

install prerequisites: `brew install cmake ripgrep`

And now make the project:

cd code-graph-rag
uv sync
make dev

You may need to install uv in your system via brew or your package manager.

Configure Code-Graph-RAG

Create and configure your environment file:

cp .env.example .env
vi .env

The.env file contains pre-written and commented configuration templates. Depending on whether you use a local small language model (SLM) via Ollama, or a hosted LLM API such as OpenAI, you can uncomment and adjust the appropriate section. For this example, we’ll use local models through Ollama:

# Model Provider Configuration

# Example 1: All Ollama (local)
ORCHESTRATOR_PROVIDER=ollama
ORCHESTRATOR_MODEL=llama3.2
ORCHESTRATOR_ENDPOINT=http://localhost:11434/v1

CYPHER_PROVIDER=ollama
CYPHER_MODEL=codellama
CYPHER_ENDPOINT=http://localhost:11434/v1

Here the orchestrator model handles reasoning and high-level task planning and the Cypher model generates graph queries for Memgraph. Please note that using a high-capacity hosted LLM (e.g., GPT-class models) generally improves reasoning quality and query accuracy compared to small local models.

Initialize Your Language Models with Ollama

Since the configuration specifies Ollama, we must install and prepare the models locally.

curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.2
ollama pull codellama

Ensure Ollama is running:

ollama serve

Ollama serves models at http://localhost:11434, which matches the .env configuration.

Parsing a Codebase: Soufflé (C++)

Now we prepare the target codebase. As I said before, we’ll use Soufflé as our case study. 

First, clone the repository:

git clone https://github.com/souffle-lang/souffle.git

Then, make sure Memgraph is running (from inside the code-graph-rag directory):

docker-compose up -d

Memgraph is a lightweight and high-performance property graph database used here as the graph engine that stores the extracted knowledge graph.

Generate the Knowledge Graph

Now we parse the Soufflé repository and build the knowledge graph:

./.venv/bin/cgr -q start --repo-path ~/GitHub/souffle/ --update-graph --clean

When the process finishes the knowledge graph is fully stored inside Memgraph and you can query it via the cgr CLI or much better, you can connect to Memgraph using gdotv for interactive data exploration and validation.

In the following video, you can see how to connect gdotv to the running Memgraph instance and explore the basics: