Building DocQA: Zero-Hallucination RAG Document Q&A with FastAPI & ChromaDB
A complete architectural walkthrough of building a local RAG pipeline with ChromaDB vector search, Groq Llama 3.3, and strictly grounded source citations.
### Overview
When building document Q&A assistants for HR policy guidelines, technical runbooks, or project specification docs, hallucination is unacceptable. If an answer is not present in the uploaded document, the assistant must state that clearly rather than inventing details.
Core RAG Architecture
1. **Local Document Chunking & Embedding**: Chunks are embedded locally using `sentence-transformers`—preventing unnecessary third-party embedding API costs. 2. **Vector Storage in ChromaDB**: Dense vector embeddings are indexed in ChromaDB with semantic similarity scoring. 3. **Re-Ranking & Similarity Filtering**: Matches below similarity threshold 0.68 are filtered out before reaching the model. 4. **Grounded Generation**: Groq's Llama 3.3 generates the answer strictly from retrieved context snippets, returning exact citations.
# FastAPI RAG Retrieval Endpoint
@app.post("/api/qa")
async def answer_question(request: QARequest):
docs = vector_store.similarity_search_with_score(request.query, k=4)
relevant_snippets = [d[0].page_content for d in docs if d[1] >= 0.68]
if not relevant_snippets:
return {"answer": "The answer is not available in the uploaded document.", "citations": []}
response = llama_client.complete(
prompt=build_grounded_prompt(request.query, relevant_snippets)
)
return {"answer": response, "citations": relevant_snippets}