Claude Certified Architect – Foundations
LLM-as-judge: grading Claude evals with a model Use an LLM judge only when code cannot grade the output. When you do, give it a detailed rubric, ask for a fixed output such as correct or incorrect or a 1 to 5 score, let it reason first, and use a different model from the one that produced the answers.
Pick the grading method first Anthropic's guidance is to choose the fastest, most reliable, most scalable method the task allows. Code-based grading, such as an exact match or a string match, is fastest and most reliable but cannot handle nuanced judgments. Human grading is the most flexible and highest quality, but slow and expensive, so avoid it where you can. LLM-based grading is fast, flexible and handles complex judgment. Test it for reliability before you scale it.
If every expected answer is one of positive, negative or neutral, compare strings in code after normalizing whitespace and case. A judge would only add cost and variance.
Writing the judge prompt Give a detailed, clear rubric. The docs' example: the answer must mention 'Acme Inc.' in the first sentence, or it is graded incorrect. One use case may need several rubrics.
Make the output empirical or specific. Ask for only correct or incorrect, or a score on a 1 to 5 scale, rather than prose. Purely qualitative output is hard to assess quickly and at scale. A vague request like 'rate quality 1 to 10' leaves the judge to decide what each number means.
Encourage reasoning. A grader model with thinking on reasons before it produces a score, which improves results on complex judgments.
Which model judges Anthropic's eval examples note that it is generally best practice to use a different model to evaluate than the model that generated the output. Also favor volume: many automatically graded cases beat a few hand-graded ones. The docs' example grader prompt asks for correct or incorrect inside <result> tags, and the code checks for that exact string.
Where the exam catches people Believing an LLM judge is the default: if code can check the answer, code-based grading is faster and more reliable. Believing an open-ended score is fine: ask for a fixed label or a defined scale with a clear rubric. Believing the same model should grade its own output: use a different model as the judge. Believing human grading is the gold standard to use everywhere: it is slow and expensive, so avoid it where you can. Written from Anthropic's documentation Try 3 questions on this Pick an answer to see why each option is right or wrong.
Your release gate scores a sentiment classifier against a golden set of 1,000 human-labelled tweets, where every expected answer is one of positive, negative or neutral. Each build sends all 1,000 model outputs to an LLM judge with a rubric, costing eleven minutes and a meaningful token bill.
Which approach should you take? Right. In the game that’s Renown for your Context Management and Reliability building.
Not that one: C is right. In the game your Context Management and Reliability building would lose a little Renown, and this question would come back later.
Your answer A. Have two domain experts hand-grade every run's outputs, since human grading is the most flexible and the highest-quality method available.There are three documented ways to grade an eval and they trade off differently. Human grading is the most flexible and highest quality of the three, and it is also slow and expensive enough that the guidance is to avoid it where possible. It earns its cost when a rubric needs judgment no rule captures, not when the correct answer for every case is already written down beside it. Your answer B. Ask the judge to reason in <thinking> tags before producing its verdict, then discard the reasoning and keep only the verdict.Asking the grader to reason first and then throwing the reasoning away is genuine, documented advice for LLM-based grading, and it helps most on tasks requiring complex judgment. It also makes each grading call longer and more expensive. Applied to a three-label comparison it buys nothing, because there is no judgment in the grading step to improve. The answer C. Compare each output against its stored golden answer with an exact-match check in code, after normalising whitespace and case.The docs rank grading methods and say to choose the fastest, most reliable, most scalable one the task allows. Code-based grading sits at the top of that ranking, and an exact-match eval measures whether the output matches a predefined correct answer, typically after normalising whitespace and case, which makes it a simple, unambiguous metric that suits clear-cut categorical answers such as positive, negative and neutral. Your answer D. Constrain the judge to output only correct or incorrect rather than prose, so that every verdict parses unambiguously.Instructing the grader to be empirical or specific - emit only correct or incorrect, or a score on a defined scale - is the documented fix for purely qualitative evaluations that are hard to assess quickly and at scale, and it would be the right move if a model had to do the grading. It does not change the fact that the eleven minutes, the token bill and the run-to-run variance all come from using a model where a comparison would do.
Your judge prompt asks for an overall quality score from 1 to 10 with no further guidance. Two engineers running the same suite on the same outputs obtain means that differ by more than a point, and almost every individual score is a 7 or an 8.
What change would most effectively improve the scores' reproducibility? Right. In the game that’s Renown for your Context Management and Reliability building.
Not that one: D is right. In the game your Context Management and Reliability building would lose a little Renown, and this question would come back later.
Your answer A. Widen the scale to 1 through 100 so the judge can express finer distinctions.A finer scale without anchors gives the judge more room to vary, not more information. Reproducibility usually falls rather than rises. Your answer B. Average five separate judge calls per case so that random scoring variation cancels out.Averaging several calls narrows the spread around whatever the judge is doing. It does not tell the judge what a 7 is supposed to mean. Your answer C. Ask the judge for a score and a justification, and keep only the numeric score.Asking for a justification does tend to improve adherence, but discarding it removes the audit trail, and the underlying problem is that the levels were never defined. The answer D. Replace the scale with a few discrete levels, each anchored by a written description.An unanchored numeric scale forces the judge to invent what each number means, which is why scores cluster in the middle and shift between runs.
Show the full explanation and doc quotes A small set of discrete levels, each with a written description of what qualifies, turns scoring into a classification against stated criteria and makes it far more reproducible.
Your team evaluates a summarization system built on Claude Opus 5 (claude-opus-5) using a judge that is also Claude Opus 5 with the same system prompt. The judge scores the system's own summaries higher than summaries a competing vendor's model produced, while a blinded sample of human raters scores the two roughly equally. You need eval scores that track human judgment.
What change would most effectively address this issue? Right. In the game that’s Renown for your Context Management and Reliability building.
Not that one: A is right. In the game your Context Management and Reliability building would lose a little Renown, and this question would come back later.
The answer A. Use a judge from a different model family than the system under test, hold it fixed, and calibrate it against the human-labeled sample.Self-preference bias appears when the judge and the system under test share a model family, so the first structural fix is to choose a judge from a different family. Your answer B. Increase the number of judge samples per item and average the scores, so that the systematic gap regresses toward the human result.Sampling more judgments per item reduces variance, not bias. A judge that systematically prefers outputs from its own family will produce the same preference on every draw, so the averaged score converges precisely on the wrong number with a narrower error bar. Your answer C. Instruct the judge in its system prompt to avoid favoring any particular model, and record the instruction in the rubric version.A judge cannot introspect a preference it does not represent as a preference. Instructing it not to favor a model changes the wording of the prompt without changing the underlying scoring behavior, and it still leaves the judge uncalibrated against human labels. Your answer D. Discard the LLM judge and score every item with human raters, accepting the reduced throughput of the evaluation suite.Human scoring is the calibration standard, not a substitute for the automated suite. The workable pattern is a small human-labeled holdout used to measure agreement with the judge, plus deterministic assertions for anything checkable, so that the judge runs on every commit and humans review a sample.
Show the full explanation and doc quotes The judge model, its prompt and the rubric are then held fixed for the life of the suite, and its labels are validated against a human-labeled sample so that inter-rater agreement is measured rather than assumed.
Written from platform.claude.com/docs/en/test-and-evaluate/develop-tests
Related Certopolis is an independent study game. It is not affiliated with, endorsed by or sponsored by Anthropic, and its questions are not the exams' questions. Exam names are trademarks of their owners.