CODE IN TOKYO

The RAG System I Was Building Until Last Year — Customizing AI Chat for Our Own Use

Until around May of last year, I was building an AI-powered internal chat system.

AI is evolving so fast these days that even a short break makes it feel like ancient history. So before I forget, I wanted to write down the architecture I was working with at the time.

Note: I can’t write about the actual business domain as-is, so I’m framing it as an “internal company system.” The reality was a bit different :)

This post walks through the general structure of RAG (Retrieval-Augmented Generation), based on the system I was actually building.

What is RAG, really?

Put simply, RAG is a mechanism that lets the AI read company documents before answering.

A typical LLM (the kind that powers ChatGPT) knows general knowledge from the internet, but obviously has no idea about company-specific information.

For example:

None of this is in the training data.

That’s where RAG comes in.

The architecture we actually used

The setup was pretty orthodox.

The flow looked like this:

  1. Ingest internal data
  2. Split documents (chunking)
  3. Vectorize (embedding)
  4. Store in pgvector
  5. Embed the user’s question
  6. Search for similar chunks
  7. Pass to the LLM for answer generation

Looking back, it’s textbook. But in practice, this setup was quite powerful.

1. First, gather the internal data

The source data lived mostly in PostgreSQL.

For example:

The key point here: “stored in the DB” does not mean “understandable by AI.”

Data written for humans is hard for AI to consume as-is.

2. Chunking the documents

Enter LangChain.

It was heavily used at the time.

If you send a long manual directly to the LLM, you run into:

So you break the text into smaller pieces.

For example:

About Product A
========

Feature description...

Incident response...

Notes...

Becomes:

Chunk 1: Feature description
Chunk 2: Incident response
Chunk 3: Notes

Where you cut matters more than you’d think.

Too fine, and context disappears. Too coarse, and retrieval accuracy drops.

A lot of RAG quality actually comes down to this kind of unglamorous tuning.

3. Embedding (vectorization)

Next, you embed the text.

Embedding simply means converting text into a numerical vector.

For example:

"I want to change my password"

becomes a numerical array of several hundred to several thousand dimensions.

The key property is that the closer two texts are in meaning, the closer their vectors.

So phrases like:

end up mathematically close.

This is the heart of RAG.

4. Store in pgvector

We stored the embedded vectors in PostgreSQL + pgvector.

This was very practical.

The reasons were simple:

That said, one caveat.

Chunking blows up the data volume fast, so the vector DB itself is better kept separate from the production business DB.

Same Postgres, but a different instance — that kind of split.

Operational practices like backups and monitoring still carry over directly.

But I really wanted to avoid a setup where the vector workload had to fight the production DB for I/O.

You know there are dedicated vector DBs in the world like:

But in enterprise systems, “we don’t want to add another DB” is a strong force.

So pgvector was a very pragmatic choice.

5. Embed the user’s question

The user asks a question.

For example:

What's the cutoff date for expense reports?

We embed this with the same embedding model.

This part is important: you must use the same embedding model for both indexing and querying.

Otherwise, the semantic space doesn’t line up.

Then we run a similarity search with pgvector.

Something like:

SELECT *
FROM documents
ORDER BY embedding <-> :question_vector
LIMIT 5;

That returns chunks like:

— things that are semantically close.

This is the part that differs from regular full-text search.

It’s not keyword matching. It’s closeness in meaning.

So it’s relatively robust to:

7. Finally, pass it to the LLM

We hand the retrieved chunks to the LLM.

Something like:

Please answer using the information below.

[Reference]
- Expense reports close on the 25th of each month
- Approvals are due by month-end

[Question]
What's the cutoff date for expense reports?

And the LLM responds with:

The cutoff date for expense reports is the 25th of each month.

— a natural-sounding answer.

That’s RAG.

The reality of RAG, as I felt it back then

What I felt after actually doing it was this: RAG isn’t “AI magic.” It’s a fairly down-to-earth search system.

In practice, what really moves accuracy is:

The unsexy parts.

More than the LLM itself, what matters is how you make it search.

And now

I haven’t built any hands-on AI system for nearly a year now, but these days there are techniques like:

Still, the core idea hasn’t changed since back then: don’t rely on AI alone — search external knowledge and augment it.

I think that’s the essence of RAG.

Closing thoughts

When people hear “AI chat,” they tend to assume the LLM is doing everything cleverly.

But in real enterprise systems, the foundation is built on fairly conventional technologies:

It might be closer to reality to see RAG as an evolved search engine.

And honestly, those unglamorous parts are where the real fun is.

— By the way, in my experience, the people who say “AI will solve everything” tend to have internal documents that haven’t been updated in 10 years.

Before asking AI to get smarter, organize your own materials first.

Faster than waiting for magic: taking inventory.