LLM-as-a-judge: having one AI grade another
LLM-as-a-judge is an evaluation method in which a language model scores, compares or checks another system's answers against a rubric. It makes it possible to evaluate open-ended answers at scale, which classic automated tests cannot judge.
Data as of October 22, 2026
Key figures
| Level | Advanced |
|---|---|
| Origin | 2023 |
| Introduced by | Zheng et al. |
Definition
Put simply: instead of paying human graders to read thousands of answers, you ask a language model to score them according to precise instructions. Three formats dominate:
- Single-answer grading: the judge scores one answer against a rubric (accuracy, completeness, tone).
- Pairwise comparison: the judge says which of two answers is better.
- Reference-guided: the judge compares the answer with an expected solution.
Origin
The term took hold with the paper “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” by Zheng and colleagues, posted on arXiv in June 2023. It examines both the method’s value and its limits: position, verbosity and self-enhancement biases, and limited reasoning ability on some questions.
A concrete example
A bank tests its FAQ assistant on several hundred real customer questions. For each answer, an LLM judge receives the question, the retrieved documentation excerpts and a rubric: “1. Is the answer based only on the excerpts? 2. Does it contain made-up information? 3. Would a non-expert customer understand it? Explain, then score.” The team also has customer advisers grade a sample, to confirm the judge agrees with them before trusting it to evaluate every new version.
Why it matters
Closed-answer benchmarks can’t measure the quality of free text, a summary or a conversation. An LLM judge makes large-scale agent evaluation and assistant testing possible, every time a prompt or model changes. Connected to production traces through LLM observability, it also enables continuous quality monitoring.
Best practices and misconceptions
- “The judge is always right.” Wrong: calibrate it against a human-graded sample.
- Swap the order of compared answers to cancel out position bias.
- Ask for reasoning before the score, and use precise criteria rather than “is this a good answer?”.
- Watch out for long answers: at equal quality, the judge may overrate them, a form of bias worth tracking.
Related terms: reward model, benchmark, agent evaluation.
Related terms
- Agent evaluation · Methods for measuring whether an AI agent really completes a task end to end, reliably, safely and at an acceptable cost.
- Benchmark · A standardised test, a fixed set of scored questions or tasks, used to compare how different AI models perform.
- LLM observability · The traces, metrics and logs that show what an AI application is doing: prompts, responses, tools called, cost and latency.
- AI bias · A systematic error that makes an AI treat some groups unfairly or repeat stereotypes, often inherited from its training data.
- Reward model · An auxiliary model that scores an answer by predicting human preference, used as the training signal for another model.
Full glossary
Frequently asked questions
Is an LLM judge as reliable as a human?
On some tasks it comes close. The 2023 paper that popularized the method reports that a strong judge such as GPT-4 reaches over 80% agreement with human preferences, about the same level as agreement between humans. That does not hold for every domain, so check it on your own data.
What are its known biases?
Researchers have described position bias (favoring the first or second answer), verbosity bias (favoring longer answers) and self-enhancement bias (favoring answers from its own model). The judge can also get math or logic reasoning wrong.
Can the same model be both candidate and judge?
It's possible but risky because of self-enhancement bias. Using a judge from a different model family, or several judges, reduces that risk.
How is it different from a reward model?
A reward model is trained specifically to output a score used in reinforcement learning. An LLM judge is a general-purpose model guided by a prompt and a rubric, mostly used for evaluation.
Data sources
Data as of October 22, 2026
Report an error
Does a figure, a date or a fact look wrong? Tell us: every report is checked.