How to Build a RAG Chat App: Chat with Your Documents Using AI

Retrieval-Augmented Generation (RAG) is one of the most practical applications of large language models. Instead of relying solely on the model's training data, RAG allows your AI to query your own documents and provide answers based on your specific content.

In this guide, we will build a RAG-powered chat application using Next.js, LangChain, and OpenAI. Users can upload documents, ask questions, and get answers with citations.

Understanding the RAG Architecture

RAG works in three phases:

• Ingestion: Documents are split into chunks, embedded into vectors, and stored in a vector database
• Retrieval: When a user asks a question, the system converts it to a vector and finds the most similar document chunks
• Generation: The retrieved chunks are injected into the LLM prompt as context, and the model generates an answer with citations

Setting Up Document Ingestion

import { RecursiveCharacterTextSplitter } from 'langchain/text_splitter'
import { OpenAIEmbeddings } from 'langchain/embeddings/openai'
import { MemoryVectorStore } from 'langchain/vectorstores/memory'

export async function ingestDocument(content: string) {
  const splitter = new RecursiveCharacterTextSplitter({
    chunkSize: 1000,
    chunkOverlap: 200,
  })

  const chunks = await splitter.createDocuments([content])

  const embeddings = new OpenAIEmbeddings({
    openAIApiKey: process.env.OPENAI_API_KEY,
  })

  const vectorStore = await MemoryVectorStore.fromDocuments(chunks, embeddings)
  return vectorStore
}

Building the Chat Interface

'use client'

import { useState, useRef } from 'react'

export function RagChat({ vectorStore }: { vectorStore: any }) {
  const [messages, setMessages] = useState<{ role: string; content: string }[]>([])
  const [input, setInput] = useState('')
  const [loading, setLoading] = useState(false)

  const handleSend = async () => {
    if (!input.trim() || loading) return
    const userMessage = input.trim()
    setInput('')
    setMessages(prev => [...prev, { role: 'user', content: userMessage }])
    setLoading(true)

    const response = await fetch('/api/rag-chat', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ message: userMessage }),
    })

    const data = await response.json()
    setMessages(prev => [...prev, { role: 'assistant', content: data.answer }])
    setLoading(false)
  }

  return (
    <div className="flex flex-col h-[600px] rounded-xl bg-gray-900">
      <div className="flex-1 overflow-y-auto p-4 space-y-4">
        {messages.map((msg, i) => (
          <div key={i} className={`flex ${msg.role === 'user' ? 'justify-end' : 'justify-start'}`}>
            <div className={`max-w-[80%] rounded-xl px-4 py-2 ${
              msg.role === 'user'
                ? 'bg-brand-600 text-white'
                : 'bg-gray-800 text-gray-100'
            }`}>{msg.content}</div>
          </div>
        ))}
        {loading && (
          <div className="flex items-center gap-2 text-gray-400">
            <span className="animate-pulse">...</span>
            <span className="text-sm">AI is thinking</span>
          </div>
        )}
      </div>
      <div className="border-t border-gray-800 p-4">
        <div className="flex gap-2">
          <input
            value={input}
            onChange={e => setInput(e.target.value)}
            onKeyDown={e => e.key === 'Enter' && handleSend()}
            placeholder="Ask a question about your documents..."
            className="flex-1 rounded-lg bg-gray-800 px-4 py-2 text-sm text-white placeholder-gray-500 border border-gray-700"
          />
          <button onClick={handleSend} className="rounded-lg bg-brand-600 px-4 py-2 text-sm text-white">Send</button>
        </div>
      </div>
    </div>
  )
}

The API Route

import { NextRequest, NextResponse } from 'next/server'
import { OpenAIEmbeddings } from 'langchain/embeddings/openai'
import { ChatOpenAI } from 'langchain/chat_models/openai'
import { createStuffDocumentsChain } from 'langchain/chains/combine_documents'
import { createRetrievalChain } from 'langchain/chains/retrieval'

export async function POST(req: NextRequest) {
  const { message } = await req.json()
  const vectorStore = await getVectorStore()
  const retriever = vectorStore.asRetriever()

  const llm = new ChatOpenAI({ modelName: 'gpt-4o' })
  const chain = await createRetrievalChain({
    retriever,
    combineDocsChain: await createStuffDocumentsChain({ llm }),
  })

  const result = await chain.invoke({ input: message })
  return NextResponse.json({ answer: result.answer })
}

Cross-Selling: Build Your RAG App Faster

Our AI RAG Chat template provides a complete, production-ready implementation of this architecture. It includes document upload, vector search, source citations, conversation history, and multi-document support. Deploy in hours and customize for your use case.

Combine with AI Document Parser for advanced document processing, or AI Form Builder to create smart data collection forms that feed into your knowledge base.