LLM Judge Reliability: Transitivity Violations and Conformal Prediction Sets

Why It Matters

As AI tools become more integrated into design workflows, understanding their reliability is crucial for making informed decisions. This research provides methods to diagnose and quantify the trustworthiness of LLM-based evaluations, ensuring that designers can rely on AI-generated feedback.

Key Finding

LLM judges show significant inconsistencies in their evaluations, but by analyzing the width of prediction sets generated through conformal prediction, designers can get a reliable measure of how trustworthy each individual evaluation is, and this measure is consistent across different judges.

Key Findings

• Widespread per-input inconsistency in LLM judgments, often masked by low aggregate violation rates.
• Split conformal prediction sets provide theoretically guaranteed coverage, with set width serving as a reliable indicator of per-instance trustworthiness.
• Prediction set width shows consistent cross-judge agreement, indicating it captures document-level difficulty rather than judge-specific noise.
• Criterion significantly impacts reliability, with relevance judged most reliably and fluency/consistency least reliably.

Research Evidence

Aim: How can transitivity violations and conformal prediction sets be used to diagnose the per-instance reliability of LLM judges in natural language generation evaluation?

Method: Diagnostic toolkit combining transitivity analysis and split conformal prediction.

Procedure: Applied a transitivity analysis to identify inconsistencies in LLM judgments and used split conformal prediction to generate prediction sets with guaranteed coverage, with set width as a reliability indicator. This was performed on the SummEval dataset across multiple judges and criteria.

Sample Size: 1,918 (for prediction set width correlation)

Context: Natural Language Generation (NLG) evaluation using LLM-as-judge frameworks.

Design Principle

Quantify and account for uncertainty in AI-driven design evaluation.

How to Apply

When using LLM-based tools for design feedback or analysis, incorporate a secondary diagnostic step to assess the reliability of individual outputs, perhaps by looking at the variance or confidence scores provided by the AI, or by using techniques like conformal prediction if feasible.

Limitations

The study focused on specific LLM judges and datasets; findings may vary with different models or evaluation tasks. The definition of 'difficulty' is implicitly tied to the LLM's judgment process.

Student Guide (IB Design Technology)

Simple Explanation: When using AI to judge designs, it's not always consistent. This study shows how to check if the AI is being reliable for each specific design, by looking at how confident it is in its judgment.

Why This Matters: Understanding the reliability of AI tools is essential for any design project that uses them for feedback, analysis, or decision-making.

Critical Thinking: To what extent can we trust AI judges in subjective design evaluation, and what are the practical implications of their inherent inconsistencies for design iteration and decision-making?

IA-Ready Paragraph: The reliability of AI-driven design evaluation, particularly for complex tasks like natural language generation, is a critical consideration. Research by Gupta and Kumar (2026) highlights that LLM judges exhibit per-instance inconsistencies that are not always apparent in aggregate statistics. Their work introduces diagnostic tools, including transitivity analysis and conformal prediction sets, to quantify this unreliability. The width of prediction sets, in particular, serves as a robust indicator of judgment confidence, correlating well across different judges and reflecting underlying document difficulty rather than judge-specific noise. This suggests that designers should implement mechanisms to assess the reliability of individual AI outputs, rather than solely relying on averaged scores, especially when evaluating subjective criteria like fluency or coherence.

Project Tips

• Consider how you will validate the outputs of any AI tools used in your design process.
• Explore methods to quantify the uncertainty or reliability of AI-generated feedback.

How to Use in IA

• Reference this study when discussing the limitations or validation of AI-generated design feedback.
• Use the concepts of per-instance reliability and confidence intervals to justify design decisions based on AI input.

Examiner Tips

• Demonstrate an awareness of the potential for bias and inconsistency in AI-generated design feedback.
• Show how you have attempted to validate or cross-reference AI outputs.

Independent Variable: ["Criterion (e.g., relevance, coherence, fluency, consistency)","Judge (specific LLM model)"]

Dependent Variable: ["Transitivity violation rate","Conformal prediction set width","Per-instance inconsistency"]

Controlled Variables: ["Dataset (SummEval)","Evaluation criteria (Likert scale)"]

Strengths

• Introduces novel diagnostic tools for LLM evaluation reliability.
• Provides theoretically guaranteed coverage for prediction sets.
• Demonstrates cross-judge agreement in reliability indicators.

Critical Questions

• How do these findings generalize to other AI models or different types of design evaluation tasks?
• What are the computational costs and practical feasibility of implementing these diagnostic tools in real-time design workflows?

Extended Essay Application

• Investigate the reliability of AI tools used for generating design concepts, user personas, or analyzing user feedback.
• Develop a framework to assess the confidence levels of AI-generated design recommendations.

Source

Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations · arXiv preprint · 2026