Speaker: Nicolas Audinet de Pieuchon

Paper: arxiv.org/abs/2506.09627

Transcript: This is a talk for the paper “Benchmarking Debiasing Methods for LLM‑Based Parameter Estimates,” to be presented at EMNLP 2025. My name is Nicolas Audinet de Pieuchon from Chalmers University of Technology. Thanks for joining.

Let’s begin with some motivation. Many domains run statistical analyses to examine correlations between variables, for example, by looking at the magnitude of coefficients in a logistic regression. However, in fields such as computational social science, the data often exists only as free‑form text. A standard solution is to manually annotate the text documents. These annotations are often high quality—especially when multiple annotators are used—but they are costly to produce at scale.

To reduce cost, researchers have started to use LLM‑based classifiers to generate annotations. This is much cheaper, but it risks introducing unwanted and poorly understood biases from the LLM into the downstream analysis. To counter this, researchers have developed debiasing methods that combine generated annotations with a small subset of expert annotations to produce unbiased downstream models while still benefiting from the scalability of language models.

In this paper, we consider two such methods: Prediction‑Powered Inference (PPI), introduced by Angelopoulos et al., and Design‑Based Supervised Learning (DSL). PPI first computes the parameter estimates from the generated annotations and then debiases them with a rectifier R(θ). The rectifier is computed from the difference between the gradients of the loss function using the expert annotations y[i] and the generated annotations ŷ[i]. DSL, by contrast, assumes that we know the sampling probability p[i] for when a sample is selected for expert annotation. It then constructs a debiased outcome y[i]‑tilde that is, in expectation, equivalent to the expert annotations, and adjusts the statistical analysis using y[i]‑tilde and its moment function.

These advances raise two questions. First, since we must produce some expert annotations anyway, when is it preferable to use scalable debiasing methods rather than running the statistical analyses using only the expert annotations? Second, what are the performance differences between the debiasing methods?

To answer these questions, we conduct two experiments. In our first experiment, we vary the proportion of samples that are expert‑annotated. The goal is to determine, given a fixed number of samples, how many should be annotated by hand to observe significant benefits from the debiasing methods. In the second experiment, we keep the number of expert‑annotated samples fixed and vary the number of samples with generated annotations. This setup targets scenarios where we have limited resources for expert annotation but abundant data. How much additional data should be annotated with language models, and are there diminishing returns?

For both experiments, we use a logistic regression with four discrete input features and a binary outcome. We average results across four datasets and four annotation methods, and we use a standardized RMSSE score for evaluation relative to a reference model in which all samples have expert annotations.

Here are the results for our first experiment. Both PPI (the green line) and DSL (the orange line) outperform using only expert annotations (the blue line) for almost all data points. Additionally, DSL seems to be more efficient than PPI at using expert samples, especially at lower proportions. However, we also observe a crossover effect at higher proportions of expert samples, where DSL performs worse than both PPI and the baseline. When we split the results by dataset, PPI remains relatively consistent, whereas DSL’s performance varies substantially across datasets.

Our second experiment shows similar trends. Both PPI and DSL outperform using only expert annotations. We also observe diminishing returns—approximately exponential—for both methods as we increase the number of samples with generated annotations.

In conclusion, we find that both DSL and PPI outperform using only expert annotations, and we encourage researchers to include generated annotations in their analyses. We also observe that DSL tends to outperform PPI but that its performance is more variable. We therefore recommend reporting results from both methods until DSL’s variability is better understood. Thank you for listening.

References

@article{nicolas audinet de pieuchon 2025benchmarking,
  title={Benchmarking Debiasing Methods for LLM-based Parameter Estimates},
  author={Nicolas Audinet de Pieuchon and Adel Daoud and Connor T. Jerzak and Moa Johansson and Richard Johansson},
  journal={Forthcoming in: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year={2025},
  volume={},
  pages={},
  publisher={},
  doi={}
}

Related Work

@article{pieuchon2024can,
  title={Can Large Language Models (or Humans) Disentangle Text?},
  author={Pieuchon, Nicolas Audinet de and Adel Daoud and Connor T. Jerzak and Moa Johansson and Richard Johansson},
  journal={Proceedings of the Sixth Workshop on Natural Language Processing and Computational Social Science (NLP+ CSS 2024)},
  year={2024},
  pages={57--67},
  publisher={}
}

@article{daoud2022conceptualizing,
  title={Conceptualizing Treatment Leakage in Text-based Causal Inference},
  author={Daoud, Adel and Connor T. Jerzak and Richard Johansson},
  journal={Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)},
  year={2022},
  volume={},
  pages={5638--5645},
  publisher={Association for Computational Linguistics},
  doi={10.18653/v1/2022.naacl-main.413}
}