Speaker: Richard Johansson

Paper: aclanthology.org/2022.naacl-main.413 // [.bib]

Transcript: This paper is about a methodological issue in text-based causal inference. In causal inference settings, we typically want to consider the effect of some treatment on an outcome variable Y. And to give an example, this could, for instance, be the effect of deregulation on child poverty.

And we typically quantify this using the average treatment effect. Since we typically cannot conduct randomized controlled experiments, we must work with observational data and control for various types of confounding variables. This could include both observed and unobserved confounders. What can we do if we have unobserved confounders?

Recently, a line of work has emerged that attempts to use text as a proxy for unobserved confounders. The recent paper by Keith and colleagues provides an introduction to this line of research. Therefore, the properties of the text will obviously be very important. In this paper, we are interested in the problem where the treatment variable affects the text itself.

This could, for instance, mean that the treatment is directly mentioned. Why does this matter? Well, in this case, if the treatment affects the text, then, as we say, the text is a post-treatment variable.

And as Pearl and others have shown, conditioning on post-treatment variable in causal inference will lead to a wide range of theoretical problems with identifiability and practical problems with estimations. If we use propensity scores to compute the causal estimates, then the propensity scores will be incorrectly estimated because the treatment is directly visible in the text.

Therefore, to proceed, we need a method to eliminate the effect of the treatment variable on the text itself. And in the simplest case, we have a property that we call separability, which simply means that the treatment-affected parts of the text are localized to some individual passages. Hence, we can simply attempt to remove those passages.

In this paper, we simply assume that this can be done. For instance, this assumption would hold if we had a domain expert who could manually review all the passages and identify the treatment-related passages. Alternatively, that we have a perfect automatic procedure that can take away the related passages automatically.

This, in practice, means then that we remove this edge from the causal graph that goes between the treatment variable and the text or its representation. To illustrate this problem, we conducted a simulation using the framework established by Wood-Doughty and colleagues. A key takeaway from the experiments is that if we are not careful and apply text-based causal inference methods naively, the estimates will be significantly off the mark.

On the other hand, if we are careful and get rid of the treatment-related passages of the text, then the estimated causal effects will be much closer to the true effects.

This concludes our presentation. The main takeaway is a recommendation for the community that we need to be careful when applying text-based causal inference so that we don’t get the leakage from the treatment variable into the text. This problem is quite likely to occur in practice that in many use cases the text might contain some information about the treatment. In order to proceed we need to try to get rid of this effect of the treatment on the text.

In the simplest case, we simply want to remove the passages that are affected by the treatment. As we have seen in our simulation, if this is done carefully, we can carry out the causal estimates properly. But if we are not careful, and we leave these treatment-related passages in the text, then the estimated causal effects will be way off the mark. In this work, we have simply assumed that we can carry out this distillation process, as we call it, but in future work, we want to examine the extent to which this can be done automatically.

References

@article{daoud2022conceptualizing,
  title={Conceptualizing Treatment Leakage in Text-based Causal Inference},
  author={Daoud, Adel and Connor T. Jerzak and Richard Johansson},
  journal={Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)},
  year={2022},
  volume={},
  pages={5638-5645},
  publisher={}
}

–

Related Work

@article{pieuchon2024can,
  title={Can Large Language Models (or Humans) Disentangle Text?},
  author={Pieuchon, Nicolas Audinet de and Adel Daoud and Connor T. Jerzak and Moa Johansson and Richard Johansson},
  journal={Proceedings of the Sixth Workshop on Natural Language Processing and Computational Social Science (NLP+ CSS 2024)},
  year={2024},
  pages={57--67},
  publisher={}
}

@article{libgober2024linking,
  title={Linking Datasets on Organizations Using Half-a-Billion Open-Collaborated Records},
  author={Libgober, Brian and Connor T. Jerzak},
  journal={Political Science Methods and Research},
  year={2024},
  pages={1-20},
  publisher={}
}

@article{jerzak2023improved,
  title={An Improved Method of Automated Nonparametric Content Analysis for Social Science},
  author={Jerzak, Connor T. and Gary King and Anton Strezhnev},
  journal={Political Analysis},
  year={2023},
  volume={31},
  number={1},
  pages={42-58},
  publisher={}
}

@article{daoud2022conceptualizing,
  title={Conceptualizing Treatment Leakage in Text-based Causal Inference},
  author={Daoud, Adel and Connor T. Jerzak and Richard Johansson},
  journal={Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)},
  year={2022},
  volume={},
  pages={5638--5645},
  publisher={Association for Computational Linguistics},
  doi={10.18653/v1/2022.naacl-main.413}
}