Building a High-Quality RAG Application

Introduction

InfoWiz, a Retrieval-Augmented Generation (RAG) application built to unlock the knowledge hidden within a vast collection of corporate documents. As an AI engineer on this project, I want to walk you through the journey of how we brought InfoWiz to life, from the initial stages of data processing to the final, sophisticated application you see today. This is a tale of challenges, innovative solutions, and a relentless focus on quality.

Chapter 1: The Foundation - A Deep Dive into Data Ingestion and Processing

Every RAG application begins with the data. Our challenge was to take a diverse set of documents—PDFs, DOCX files, and more—and transform them into a format that a language model could understand and use effectively. This is where our data ingestion pipeline came in, and it's far more than a simple script.

The heart of this process lies in the utilities/chunking directory. We knew that simply splitting the documents into fixed-size chunks wouldn't be enough. We needed to preserve the context and structure of the original documents. To achieve this, we developed a sophisticated, multi-stage chunking strategy:

• High-Resolution Document Parsing: We chose the powerful Unstructured.io library for our document parsing. Specifically, we used the "hi_res" partitioning strategy with the "yolox" model. This is a computer vision-based approach that allows us to accurately identify and extract elements from complex documents, including tables, titles, and lists. We also configured it to infer table structures, ensuring that tabular data is preserved correctly.
• Local Inference for Performance and Privacy: A key architectural decision was to use local inference models for unstructured. We overwrote the default behavior to load the yolox and table-transformer-structure-recognition models from the local repository. This not only improved performance and reduced costs but also ensured that our sensitive documents were not sent to any external services.
• Advanced Text Processing: Our TextChunkProcessor is a pipeline in itself. Before chunking, it performs:
  • Content Analysis: It analyzes the text to detect the presence of tables, images, and other content types, enriching the metadata for each chunk.
  • Text Cleaning: It normalizes whitespace and cleans up the text to ensure consistency.
  • Detailed Metadata Extraction: It uses a series of regular expressions to extract critical metadata like page numbers, revision numbers, and publication dates directly from the document's content.
• Context-Aware Chunking: We used a title-based chunking strategy (chunk_by_title) to create semantically meaningful chunks. We configured our chunking with a max_chunk_size of 1024 tokens and a chunk_overlap of 96 tokens to ensure that each chunk is both comprehensive and contextually linked to its neighbors.
• LLM-Powered Summarization: To further enhance our chunks, we integrated a gpt-4o-mini model to generate a summary for each chunk. This provides a condensed representation of the chunk's content, which can be invaluable for certain retrieval strategies.

This meticulous and multi-faceted approach to data processing laid a solid foundation for the next stage: building our knowledge base.

Chapter 2: Building the Knowledge Base - Sophisticated Indexing

With our documents processed into high-quality, metadata-rich chunks, the next step was to create a searchable index. For this, we turned to two powerful tools: LlamaIndex and ChromaDB.

• LlamaIndex: We used LlamaIndex as the primary framework for orchestrating the indexing process. Its high-level abstractions and rich feature set allowed us to build a robust and efficient pipeline.
• ChromaDB: As our vector store, we chose ChromaDB for its performance and ease of use.

The utilities/Indexes_creation/produce_index_from_chunks.py script is where the magic happens. Here's how it works:

1. Iterating Through Chunks: The script processes the JSON files from our chunking pipeline.
2. Creating Nodes with Relationships: For each chunk, it creates a TextNode in LlamaIndex. Crucially, we preserved the relationships between chunks. Each node is linked to its PREVIOUS and NEXT nodes, and back to its SOURCE document. This allows us to retrieve not just a single chunk, but a window of surrounding context, which dramatically improves the quality of the generated responses.
3. Storing in ChromaDB: The nodes are then indexed and stored in a ChromaDB collection, creating a highly searchable and contextually aware knowledge base.

Chapter 3: The Heart of the Application - An Advanced RAG Engine

With our knowledge base in place, it was time to build the application itself. The core logic resides in infowiz/app.py, and it's built on the Viktor framework, which allowed us to create a sophisticated, multi-page web application.

But the true innovation of InfoWiz lies in its advanced RAG engine:

• Dynamic Query Engine Selection: We recognized that not all queries are created equal. Simple queries can be handled by a standard retrieval engine, but complex questions often require a more sophisticated approach. That's why we implemented a dynamic query engine selection mechanism. Using an LLMSingleSelector, the application analyzes the user's query and intelligently chooses the best engine for the job:
  • For simple queries: It uses a standard ContextChatEngine.
  • For complex queries: It switches to a SubQuestionQueryEngine. This powerful engine breaks down a complex question into smaller, more manageable sub-questions, which are then answered and synthesized into a final, comprehensive response.
• ReAct Agent for Advanced Reasoning: The SubQuestionQueryEngine is powered by a ReActAgent. ReAct is a cutting-edge agent framework that combines reasoning and acting, allowing the agent to iteratively think through a problem, use tools to gather information, and refine its answer. This is a significant step beyond simple retrieval and generation.
• Federated Search with Automated Citations: Our hybrid search is more than just a combination of keyword and vector search. It's a federated search system that can query across all our ChromaDB collections simultaneously. But the standout feature is our automated citation and referencing mechanism. After generating a response, the system performs a sentence-level match between the answer and the source documents. This allows it to:
  • Inject citations directly into the text of the answer.
  • Provide a list of references with direct links to the source documents in SharePoint. This provides full traceability and allows users to verify the information for themselves, which is a critical feature for any enterprise application.

Chapter 4: Ensuring Quality - A Multi-Faceted Approach to Testing and Evaluation

A high-quality RAG application requires a rigorous approach to quality assurance. We implemented a multi-faceted strategy that combines traditional testing with cutting-edge, metric-driven evaluation.

• Comprehensive Test Suite: We developed a suite of tests (in the tests directory) that cover various aspects of the application, from text extraction to the functionality of our advanced SubQuestionQueryEngine.

• End-to-End Retrieval Diagnostics: Our test_context_retrieval.py script is a powerful diagnostic tool. It allows us to perform an end-to-end test of the retrieval pipeline by taking a test query, calculating its similarity against every single chunk in our knowledge base, and then passing the top results to the LLM. This has been invaluable for debugging and fine-tuning our retrieval process.

• Metric-Driven Quality Assurance with deepeval: The pinnacle of our quality assurance process is our use of the deepeval library. In tests like test_sub_query_engine_quality, we programmatically assess the quality of our RAG system's output. We define test cases with an input, the actual output, and an expected output, and then use deepeval's metrics to score the response on:

  • Answer Relevancy: Is the answer relevant to the question?
  • Faithfulness: Is the answer factually accurate and supported by the retrieved context?

  We then assert that these scores meet a certain threshold. This data-driven approach to quality assurance ensures that InfoWiz is not only functional but also consistently accurate and reliable.

Chapter 5: The Technology Stack

InfoWiz is built on a modern and powerful technology stack:

• Core Framework: Viktor, LlamaIndex
• Vector Database: ChromaDB
• LLMs and Embeddings: Azure OpenAI (GPT-4o, GPT-4o-mini), LangChain
• Data Processing: Unstructured.io, PDFPlumber, PDFMiner.six, Tiktoken
• Evaluation: DeepEval, NumPy, SciPy
• ML Models: ONNX (for local document processing)

Conclusion

The journey of building InfoWiz has been a challenging but rewarding one. From the meticulous processing of our source documents to the development of a sophisticated, multi-page application and a rigorous evaluation framework, we've built a powerful tool for unlocking the knowledge hidden in our corporate data.

The dynamic query engine selection, the use of a ReAct agent, the federated search with automated citations, and the metric-driven approach to quality all demonstrate a commitment to engineering excellence. InfoWiz is more than just a RAG application; it's a testament to what can be achieved with a clear vision, a powerful technology stack, and a relentless focus on quality.