Transcript

Satiyabooshan Murugaboopathy: I’m going to talk about LUCID, which stands for Loop for Unbiased Calibration via Inference Time Diagnosis. What we did is introduce a framework that debiases LLM reasoning, especially LLM reasoning evaluation, using statistics and statistical methods. The aim is to debias LLM reasoning at a low cost without retraining at inference time.

Let’s start at the beginning. LLM agents reason via chain-of-thought traces, and I think most of us have experienced that. But should we trust those reasoning steps, and are we actually evaluating them enough, or are we just evaluating the final task completion? That is a question I asked myself. If we want to trust them, we need to have some kind of proper rating system. The gold labels would, of course, always be experts, but experts are very costly for full annotation. So we were thinking of how to reduce the costs. Maybe that could involve cheaper LLMs that could help us with rating those reasoning steps, whether they are good, bad, or somewhere in between.

Our question is: how could we diagnose the bias from a few expert labels so that we don’t need full annotation? And how could we use that to improve the LLM reasoning or the rating of the LLM reasoning steps? To do that, we used datasets that have been used to train process reward models that are labeled. There are datasets such as the PRM800K dataset, which has a lot of problems that are not only labeled by their task completion but also by how good the reasoning steps were, and whether the reasoning steps were actually right or wrong. That’s what we started with. We artificially limited ourselves to only using a thousand expert labels because we thought that could be a realistic scenario for an organization or research group wanting to evaluate their LLM’s reasoning manually. It was great to have the PRM800K dataset for evaluation. That is what we started with to evaluate our framework on.

As background information, we used DSL (Design-based Supervised Learning), which is a statistical method. It’s essentially our detection mechanism for bias in a statistical way. We cross-fit an honest forest model that predicts the true label from the bias prediction and the features. Then we build a pseudo-outcome—the model’s prediction on every item plus the residual on the labeled items. DSL gives us two things that we’ll use: a debiased estimate and a bias diagnosis, so that we can have per-feature gaps between biased and corrected rates. It is basically a pseudo-model on top of what the LLM actually predicted that gives the corrected version and helps us calculate the gap.

Connor Jerzak: What are the inputs and the outputs here?

Satiyabooshan Murugaboopathy: Right, so whether the step is correct or incorrect was not of highest importance for us; it was more about whether the rating of the step done by the LLM was actually right or wrong. We had those gold standard expert labels that we assume are right, and then we had the cheaper LLM—GPT-3.5 Mini in most cases, though we also compared it to another model for evaluation purposes. We wanted to see whether the rating from GPT-3.5 Mini was correct or wrong. It was not about whether the step yielded a correct task completion.

Connor Jerzak: So the idea is, we have all these reasoning steps, and we can generate those on the fly with a new model. If we could rate those accurately, we could then choose which reasoning outcome is the best based on the quality of the intermediary steps. But we don’t have the ability to apply human ground truth labels to that at will because that’s expensive. So the idea is we will generate all those reasoning steps on the fly, generate a bunch of ratings for those reasoning steps, and then adjust those ratings based on the DSL procedure to debias the LLM judgments of those reasoning steps.

Satiyabooshan Murugaboopathy: That’s exactly what we did. So we come to our contribution now, which is exactly what you just described. The agent—in our case, GPT-3.5 Mini—rates the step or many steps. Then we analyze with DSL to diagnose whether there is bias. If there is bias, we can feed that bias diagnosis back to the agent so it can re-rate itself. Then we’ll test if that new rating is biased or not. This is the LUCID loop that yielded a very good result; even after one loop, it oftentimes improved itself a lot, and it was very cheap. I guess that’s our success, and we did this with no weight updates, no fine-tuning, and no extra labels. That’s what makes it a bit special.

Connor Jerzak: That’s awesome. I’m just trying to think of failure modes now. The intuition here is we’re leveraging the human labels at inference time in a principled way. I think that’s the intuition about why it’s working. But it would be expected to fail if the reasoning task where the human labels were applied is very different than the reasoning task analyzed at inference time. I’m just curious to hear your thoughts: what are other potential limitations, as well as areas where you think this would really work well?

Satiyabooshan Murugaboopathy: Absolutely. You’re right that if the labels are not correlated enough with the task the agent is currently rating, it loses its value. Where this works well is always when there is a high systematic, structured bias. If we can diagnose a structured bias in the labels, then we can confidently say that structured bias might also exist in future ratings the agent does. That’s where this works well. But, of course, if there is randomized noise instead of bias, it makes things very tough for LUCID.

Connor Jerzak: In other words, and I think you’ll get to this in the results, we can be wrong for two reasons: either the model is systematically biased, or it’s essentially incompetent in each reasoning step. This LUCID procedure works well in the case where the bias is systematic; however, it struggles when the model is simply incompetent in each of its steps.

Satiyabooshan Murugaboopathy: Exactly. That’s also kind of our headline statement that we make at the end: we can correct bias, but we can’t correct incompetence. It’s intuitive that we cannot correct incompetence at inference time without adding complexity to the problem. It can be compared to humans: if a human has a skill but just has some bias, you can help them correct the bias with one statement. But if the person needs to learn the whole skill, that’s a different story.

Our key insight was actually that there is structured bias in LLM ratings, and we could find that. For example, when there are very long-horizon tasks, LLMs tend to rate rather negatively than positively. That’s a simple example, but such biases could be corrected very easily. We also noticed an issue: if there is a lot of underrating, we tend to overshoot in other categories because of our rather simple prompt feedback. That’s a downside of our framework we want to work on, so we don’t make things worse if they were actually better before our correction, even though the overall correction is still positive. We want to succeed in all categories.

Closer to zero is actually the best. That’s why when the original rating bias was very negative, we could correct it and bring it closer to zero. But if the original bias was already close to zero, we sometimes overcorrected and made it overshoot.

What we also did was check whether the debiasing actually mattered in the downstream task. We ranked candidate trajectories by their average step score and picked the best. With raw GPT-3.5 Mini scores, the top-one selection is at 15.2%. But with the DSL-calibrated scores, it’s 28.9%, an improvement of 13.7 points, and it actually beats the expert gold standard curve up to n=5 before closing the gap. Up to 5 is the tougher task, so it is quite a good result. It’s cool that a 30-cent statistical correction of a cheap rater can outperform costly expert annotation for this specific math task on the PRM800K dataset.

We also tested against another open-source model, GLM-5.2, to check whether these findings hold across different models. We saw similar results. We tested against other datasets and domains as well. On ProcessBench, which also has math problems, we still yielded an improvement of 5.2%. On EntailmentBank, a science dataset where there was a huge bias, we saw a massive gap reduction. Mind2Web features web agent traces where the rater was very conservative, and we came down from an 89% calibration gap to a 2.6-point calibration gap. These huge improvements only happen if there is a high initial bias.

So, we come to the question you already posed: is LUCID fixing inability or bias? LUCID can only fix biased models, not incompetent models, by fixing the calibration of systematic errors. This can lead to extreme benefits when there is extreme systematic bias, but smaller gains when the bias isn’t huge. Sometimes it can even mess things up if the model is already calibrated and we overcorrect.

To summarize the takeaways of this LUCID paper: LLM rating bias is structured, diagnosable, and actionable. We can fix it with just one round of statistical feedback, which is super cheap and doesn’t require costly hardware or retraining. DSL is a strong diagnostic tool to detect bias if you have a few expert labels. But the most important takeaway is that there is a difference between calibration and discrimination—or bias and incompetence. We can only fix the former; the latter requires retraining. Thank you.

Connor Jerzak: Thank you. A couple of questions. One thought is, you could imagine scaling this up for a new task by having a more expensive LLM potentially annotate some of these reasoning traces on the fly, and then applying a DSL-like procedure to cheaper models. What are your thoughts on generalizing this? Do you think we need a human in the loop, or can we generalize out the human? How could we scale this up across multiple tasks where we don’t already have those expensive reasoning traces?

Satiyabooshan Murugaboopathy: Absolutely. It is essentially the same concept. We could use costlier expert models instead of expert humans to help our smaller models improve their ratings or diagnose the bias. That’s a very viable option. We haven’t tried that yet, but in this first paper, we wanted to make sure we worked with real expert gold labels to prove it works before living entirely in the LLM world. Now that we’ve shown there is bias and the framework works with human expert labels, I’m sure we could use costlier LLMs to diagnose the bias first with DSL, and then improve the cheaper LLMs. Expert LLMs are cheaper than expert humans, so we could rate more data or rate specifically in clusters to always have the perfect DSL diagnosis for our specific task.

Connor Jerzak: Another thought would be, if it’s hard for the model to really get a rating or think through that scale, using pairwise comparisons can be a method to deploy. In contexts where there is no clear intrinsic scale, we can use pairwise trace outcomes to try to calibrate the model further.

Satiyabooshan Murugaboopathy: Yeah, absolutely. I was surprised to find such an amount of structured bias in some ratings. You assume there is bias, but it was surprising to see that much. What is even cooler is that we can correct it in such a cheap and easy way. That’s a win for us because there are so many low-hanging fruits for improving these ratings, which makes it very cool.

Related Work

@article{nicolas audinet de pieuchon 2025benchmarking,
  title={Benchmarking Debiasing Methods for LLM-based Parameter Estimates},
  author={Nicolas Audinet de Pieuchon and Adel Daoud and Connor T. Jerzak and Moa Johansson and Richard Johansson},
  journal={Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year={2025},
  volume={},
  pages={},
  publisher={},
  doi={}
}

–

@article{balter2026multiplication,
  title={Multiplication in Multimodal LLMs: Computation with Text, Image, and Audio Inputs},
  author={Balter, Samuel and Ethan Jerzak and Connor T. Jerzak},
  journal={Findings of the Association for Computational Linguistics (ACL)},
  year={2026},
  pages={},
  publisher={}
}

–

@article{pieuchon2024can,
  title={Can Large Language Models (or Humans) Disentangle Text?},
  author={Pieuchon, Nicolas Audinet de and Adel Daoud and Connor T. Jerzak and Moa Johansson and Richard Johansson},
  journal={Proceedings of the Sixth Workshop on Natural Language Processing and Computational Social Science (NLP+ CSS 2024)},
  year={2024},
  pages={57--67},
  publisher={}
}

–

@article{vaidya2026queryable,
  title={Queryable LoRA: Instruction-Regularized Routing Over Shared Low-Rank Update Atoms},
  author={Vaidya, Omatharv and Connor T. Jerzak and Nhat Ho and Chandrajit Bajaj},
  journal={ArXiv Preprint},
  year={2026},
  pages={},
  publisher={}
}

–

@article{libgober2024linking,
  title={Linking Datasets on Organizations Using Half-a-Billion Open-Collaborated Records},
  author={Libgober, Brian and Connor T. Jerzak},
  journal={Political Science Research and Methods},
  year={2024},
  pages={1-20},
  publisher={}
}

–

@article{jerzak2023improved,
  title={An Improved Method of Automated Nonparametric Content Analysis for Social Science},
  author={Jerzak, Connor T. and Gary King and Anton Strezhnev},
  journal={Political Analysis},
  year={2023},
  volume={31},
  number={1},
  pages={42-58},
  publisher={}
}

–

@article{murugaboopathy2026platonic,
  title={Platonic Representations for Poverty Mapping: Unified Vision-Language Codes or Agent-Induced Novelty?},
  author={Murugaboopathy, Satiyabooshan and Connor T. Jerzak and Adel Daoud},
  journal={Proceedings of the 7th International Conference on Social Computing (ICSC 2026), of Communications in Computer and Information Science},
  year={2026},
  volume={},
  pages={},
  publisher={Springer Nature}
}

–

@article{daoud2022conceptualizing,
  title={Conceptualizing Treatment Leakage in Text-based Causal Inference},
  author={Daoud, Adel and Connor T. Jerzak and Richard Johansson},
  journal={Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)},
  year={2022},
  volume={},
  pages={5638--5645},
  publisher={Association for Computational Linguistics},
  doi={10.18653/v1/2022.naacl-main.413}
}

–

Back to Home

• GitHub
• YouTube
• LinkedIn
• Bluesky
• Instagram
• TikTok
• Spotify
• X
• Mail