RAG Compliance & Incident Intelligence Assistant
A question-answering system over UK health and safety, SIA licensing and fire safety regulations, where every answer carries citations back to the source clause. The differentiator is the evaluation harness: retrieval quality is measured in CI, not assumed.
Why this project
I have spent the last three years working in security and compliance roles alongside my studies, and I have seen how much time goes into looking up what the regulations actually require. An LLM that paraphrases regulation from memory is dangerous in that setting; an answer is only useful if it cites the exact clause it came from. That constraint, groundedness over fluency, is what this project is built around.
What it does
- Ingests and chunks UK regulatory documents into pgvector with metadata for precise citation
- Retrieves with hybrid search (dense vectors plus BM25) and a re-ranking step, because naive vector search alone is not good enough for legal-style text
- Answers through both a hosted model (OpenAI) and an open-weight model via Ollama, so cost and privacy trade-offs are demonstrable
- Evaluates every change with RAGAS (faithfulness, answer relevance, context precision) in GitHub Actions, failing the build if quality regresses
- Caches frequent queries in Redis and streams responses through a FastAPI backend
Architecture
The evaluation harness
Most RAG demos show a chat box. This one ships a comparison table: the same golden question set run against naive vector search, hybrid search, and hybrid plus re-ranking, scored on faithfulness and context precision. That table is the proof that each pipeline stage earns its complexity, and it is the artefact I want to discuss in interviews.
Skills demonstrated
- RAG architecture: chunking, hybrid retrieval, re-ranking, citation grounding
- LLM evaluation as an engineering discipline (RAGAS in CI)
- Working with both hosted and open-weight models
- Backend engineering: FastAPI, PostgreSQL/pgvector, Redis, Docker
- Domain knowledge from three years in UK security and compliance work