Bedrock model evaluation: automatic, human, LLM judge (AIP-C01)

Try 3 questions on this

Pick an answer to see why each option is right or wrong.

1. A legal publisher must choose between two FMs in Amazon Bedrock to summarize court rulings for a weekly client newsletter. Editors have written an approved summary for each of 400 past rulings. For each model, the publisher needs a score that shows how closely the model's summaries match the approved summaries. The score must be computed by an algorithm, so that the same responses always receive the same score. The scoring must not use human reviewers or another LLM. Which solution will meet these requirements?

   Right. In the game that’s Renown for your Testing, Validation, and Troubleshooting building.

   Not that one: A is right. In the game your Testing, Validation, and Troubleshooting building would lose a little Renown, and this question would come back later.

   The answerA.Create an automatic model evaluation job per model with the text summarization task type and the accuracy metric. Use a custom prompt dataset with each ruling as the prompt and its approved summary as the reference response.An automatic evaluation job computes fixed algorithmic metrics, and the metric depends on the task type.Your answerB.Create an automatic model evaluation job for each model with the general text generation task type and the accuracy metric. Use a custom prompt dataset with each ruling as the prompt and its approved summary as the reference response.Accuracy means something different for each task type. For general text generation it is the RWK score, a test of world knowledge, so it does not compare a summary with the approved one.Your answerC.Create an Amazon Bedrock model evaluation job that uses an LLM as a judge for each model with the Builtin.Correctness metric. Use a custom prompt dataset with each ruling as the prompt and its approved summary as the reference response.Correctness would read the approved summaries, but the grader is another LLM. The publisher wants an algorithmic score and no LLM in the scoring.Your answerD.Create an automatic model evaluation job for each model with the text summarization task type and the robustness metric. Use a custom prompt dataset with each ruling as the prompt and its approved summary as the reference response.Robustness perturbs each ruling and reports how much the summary changes. It says how stable a model is, not how close its summaries are to the editors' versions.

   Show the full explanation and doc quotes

   For text summarization, the report page (https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-report-programmatic.html) says: "Accuracy : For this metric, the value is calculated using BERT Score." and "It matches words in candidate and reference sentences by cosine similarity." The approved summaries go in the referenceResponse key of a custom dataset. The datasets page (https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-prompt-datasets.html) lists referenceResponse as "required to indicate the ground truth response against which your model is evaluated" for accuracy and robustness evaluations.

   Written from docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-report-programmatic.html

2. A company is deciding whether to replace its self-hosted summarization model with an FM in Amazon Bedrock. The company has already stored the self-hosted model's summaries of 400 support tickets. Five product specialists must read each ticket, compare the two summaries side by side, and rate which summary is more accurate. The ratings must be saved in Amazon S3 for audit. Which solution will meet these requirements with the LEAST operational overhead?

   Right. In the game that’s Renown for your Testing, Validation, and Troubleshooting building.

   Not that one: D is right. In the game your Testing, Validation, and Troubleshooting building would lose a little Renown, and this question would come back later.

   Your answerA.Create an Amazon Augmented AI human review workflow for the specialists. Have a script call the FM for each ticket, and start a human loop with both summaries for the specialists to rate.A human loop reviews one prediction at a time and needs a custom script to create each loop. It is more work for the same side-by-side rating.Your answerB.Create an Amazon SageMaker Ground Truth labeling job with a custom task template that shows both summaries. Write an AWS Lambda function that adds the summary of each ticket from the FM to the input manifest.This can work, but it rebuilds by hand what a Bedrock human evaluation job already provides: the side-by-side form, the FM calls, and the stored results.Your answerC.Create a model evaluation job in Amazon Bedrock that uses an LLM as a judge. Supply the stored summaries as the inference responses, and define a custom metric that rates the accuracy of each summary.An evaluator model rates accuracy quickly, but the requirement is for the five specialists to rate each pair of summaries.The answerD.Create a human-based model evaluation job in Amazon Bedrock with a work team of the specialists. Select the FM as one inference source, and provide the stored summaries as inference response data for the second source.The human-based evaluation page (https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-jobs-management-create-human.html) says "You can evaluate the performance of Amazon Bedrock models, or of other models by providing your own inference response data in your prompt dataset." and "You can select one Amazon Bedrock model and provide your own inference response data for the second source." One job therefore generates the FM summaries, shows them next to the stored summaries to the specialists' work team, and writes the results to the S3 destination set for the job.

   Written from docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-jobs-management-create-human.html

3. A company runs a RAG application that retrieves policy passages from its own search service and sends them with each customer question to an FM in Amazon Bedrock. Reviewers found answers that contain details that are not in any of the retrieved passages. The company has saved 1,000 production prompts, each with the question and the retrieved passages. No reference answers exist. The company must score how often the FM's responses include information that is not in the passages of their prompt, and must repeat the scoring after each prompt change. Which solution will meet these requirements?

   Right. In the game that’s Renown for your Testing, Validation, and Troubleshooting building.

   Not that one: C is right. In the game your Testing, Validation, and Troubleshooting building would lose a little Renown, and this question would come back later.

   Your answerA.Create a model evaluation job that uses an LLM as a judge, with the FM as the generator model and the saved prompts as the dataset. Select the Builtin.Completeness metric.Completeness judges whether the response answers every question in the prompt, not whether it adds information that the passages do not contain.Your answerB.Create an automatic model evaluation job with the question and answer task type, with the FM as the model and the saved prompts as the dataset. Select the accuracy metric.The accuracy metric of an automatic job compares responses with reference answers, and the company has none; it does not look for content missing from the passages.The answerC.Create a model evaluation job that uses an LLM as a judge, with the FM as the generator model and the saved prompts as the dataset. Select the Builtin.Faithfulness metric.The judge metrics page defines Faithfulness (Builtin.Faithfulness) as a metric that "Identifies whether the response contains information not found in the prompt to measure how faithful the response is to the available context." Because the saved prompts carry the retrieved passages, that is exactly a score for details that are not in the passages, and it needs no reference answer.Your answerD.Create a model evaluation job that uses an LLM as a judge, with the FM as the generator model and the saved prompts as the dataset. Select the Builtin.Correctness metric.Correctness judges whether the response is right, not whether each detail comes from the passages in the prompt; a correct detail from the model's own knowledge still passes.

   Written from docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-metrics.html