Question answering over your own documents

Ask questions about a folder of documents and get answers with citations, plus an evaluation set that tells you if it works.

Why build it

Retrieval-augmented generation is common in industry, and most demos skip evaluation. Measuring answer quality is what makes this portfolio-worthy.

Features

• Ingest PDFs and markdown
• Chunking and embeddings
• Vector search with metadata filters
• Answers with source citations
• Evaluation harness with a question set

Roadmap

1. p1

   Ingest

   • Parse documents and keep page and section metadata
   • Compare chunk sizes and overlaps
   • Store embeddings in a vector database
2. p2

   Retrieve

   • Top-k vector search
   • Add keyword search and combine
   • Re-rank results
3. p3

   Answer

   • Prompt that requires citations
   • Refuse when sources do not contain the answer
   • Stream the response
4. p4

   Evaluate

   • Write 50 questions with expected sources
   • Measure retrieval hit rate and answer accuracy
   • Report which change moved each number

Common pitfalls

• Judging quality by trying a few questions by hand
• Chunking that splits tables and code blocks in half

Skills you will practise

• Embeddings
• Retrieval
• Evaluation
• Prompting