ASU LLM Evaluation Pipeline

An automated LLM quality monitoring system for an ASU university RAG chatbot. Every GitHub push triggers a full RAGAS evaluation: GPT-4o answers 100 test questions, GPT-4o-mini judges each response for faithfulness, relevancy, and context precision — 25× cheaper than GPT-4o for evaluation tasks. Six automated quality gates block deployment if any metric drops below threshold (hallucination rate ≤ 10%, faithfulness ≥ 0.80, answer relevancy ≥ 0.75). Vectors persist in Pinecone cloud so CI runners skip re-embedding on every push. Results are auto-committed back to the repo with [skip ci] to prevent infinite trigger loops, then streamed to a React dashboard on Vercel showing trend charts across the last 50 runs, a per-question breakdown table with category and difficulty filters, and real-time cost tracking with 30-second auto-refresh. Latest run: 6/6 gates PASS, faithfulness 1.00, hallucination 0%, cost $0.087.