LLM-as-a-judge

LLM-as-a-judge (LaaJ) uses a large language model to autonomously evaluate the output of another model, delivering scalable, cost-effective, and nuanced quality assessment.

This technology deploys a sophisticated LLM (like GPT-4, which shows 81% agreement with human judgment in some studies) as a judge to score or rank AI-generated content. It moves past traditional, brittle metrics (BLEU, ROUGE) that fail on open-ended text. LaaJ excels in subjective evaluation: assessing faithfulness, tone, or relevance via prompt-engineered criteria. Use cases span from pairwise comparisons (Chatbot Arena style) to direct scoring on a 1-5 scale, providing structured, interpretable feedback that dramatically cuts the cost and time of human annotation.

https://github.com/CSHaitao/Awesome-LLMs-as-Judges