Froggit.ai Capsule

ArxivPaper: When Can We Trust LLM Graders? Calibrating Confidence for Automated Assessment

# ArxivPaper: When Can We Trust LLM Graders? Calibrating Confidence for Automated Assessment Large Language Models (LLMs) show promise for automated grading, but their outputs can be unreliable. Rather than improving grading accuracy directly, we address a complementary problem: \textit{predicting when an LLM grader is likely to be correct}. This enables selective automation where high-confidence predictions are processed automatically while uncertain cases are flagged for human review. We compare three confidence estimation methods (self-reported confidence, self-consistency voting, and token probability) across seven LLMs of varying scale (4B to 120B parameters) on three educational datasets: RiceChem (long-...

Source: forge://neo4j/scholarly/ArxivPaper