Building a RAG Chat System: The Hardest Part Was "Hallucination Detection"

In an earlier blog post, I wrote about building a chat system using RAG. This is the follow-up — a story about judging how far you can actually trust the answers that come back.

Previous post: here

Chunk internal documents, embed them, do similarity search with pgvector, pass the results to an LLM and let it answer.

That much is pretty standard, I think.

But once we actually started running it in production, the biggest problem wasn’t retrieval accuracy — it was that

“the LLM smoothly says things that aren’t in the context.”

That’s what people call hallucination.

”Plausible Lies” Are Genuinely Tricky

For example, the LLM would fill in things not written in the search results, on its own.

What made it nasty was that these weren’t outright nonsense. They were often:

• Half correct
• Correct as general knowledge
• Natural in context

In other words, even a human reader can’t tell right away.

RAG is, at the end of the day, still an LLM. Even if retrieval brings back the right information, the model still “embellishes” at generation time.

That was harder than I expected.

At First, I Thought “Just Let Another LLM Judge It”

My first idea was simple:

let another LLM judge whether the answer is consistent with the context.

I tried several evaluation libraries that take this approach.

But in practice, this alone wasn’t stable.

Given the same answer, the judgments would subtly drift.

Even with temperature set to 0, it didn’t fully stabilize.

And of course, the API cost was heavy.

Around this point, my thinking shifted toward:

“relying on a single evaluation axis is dangerous.”

In the End, We Combined Multiple Metrics

Ultimately, we ended up combining:

• LLM-based consistency evaluation
• NLI-style models
• BERTScore-style similarity

and scoring from multiple perspectives.

BERTScore in particular looks at semantic closeness via embeddings rather than plain string matching, so it was reasonably robust to paraphrasing.

But the flip side is that BERTScore alone misses cases that are “semantically close but actually false.”

So the architecture ended up mixing:

• Statistical similarity
• Inference models
• LLM judgment

Using XGBoost Was a Pretty Engineering-Driven Decision

The interesting part was that for the final judgment we used XGBoost, not a neural network.

The reasons were simple:

• Training data wasn’t that large
• We wanted lightweight inference
• We wanted to look at “feature combinations”

For example, patterns like:

• BERTScore is low
• but NLI is high

come up surprisingly often.

Tree-based models were easier to work with for these subtle patterns.

This part felt less like building an “AI system” and more like assembling an ordinary machine learning system.

The Unexpected Part: Handling “Legitimate Refusal”

Another interesting thing during implementation was how to treat “answer refusal.”

At first, I thought

“I don’t know”

was a failure.

But in reality, it was the opposite.

If something isn’t in the context, returning

“no information available”

is more correct than padding the answer with something plausible.

So partway through, we started splitting the categories:

• Hallucination
• Abnormal
• Normal

You don’t really notice this kind of thing until you’ve actually run the system in production.

Looking Back: “RAG Is Harder at Evaluation Than at Retrieval”

When people talk about RAG, the conversation tends to focus on:

• Vector search
• Embeddings
• Chunking

These matter, of course. But what really hurt in production was evaluation.

Even if retrieval is a bit off, the LLM sometimes compensates well.

Conversely, even when retrieval is correct, the LLM may quietly mix in falsehoods.

So in the end, it all comes down to:

“how much can we trust this answer?”

Even now, I think that’s the hardest part of RAG.

Afterword

Career-wise, I’m not an AI specialist. I’ve spent most of my career on the full-stack side — infrastructure, frontend, backend.

But back in my student days, I had some experience with numerical computing, and that’s what let me join this project.

At the start, there were so many things I didn’t understand. Honestly, it was tough.

Even so, once I dug in, it was genuinely interesting.

To be honest, the architecture I described here was largely modeled on AI textbooks — there’s nothing original about it. I’m an engineer, not a researcher, so I made my peace with that.

These days, I’m on the side of “using AI in implementation,” not building AI itself.

If the chance comes around again, I’d like to do this kind of work once more.