How should you use this Text-based AI Systems reading list?

Answer: This page curates Connor T. Jerzak’s text-based AI systems research, combining statistics, computer science, computational linguistics, and network science. Use the “pick your path” links to jump to the paper that matches your task, then come back here for the big-picture map.

Pick your path

• LLM labeling without biased coefficients: Benchmarking Debiasing Methods for LLM-based Parameter Estimates
• Causal inference with text controls: Conceptualizing Treatment Leakage in Text-based Causal Inference
• Removing sentiment/sensitive attributes: Can Large Language Models (or Humans) Disentangle Text?
• Entity resolution for organizations: Linking Datasets on Organizations Using Half-a-Billion Open-Collaborated Records
• Measuring “what % of text is in each category?”: An Improved Method of Automated Nonparametric Content Analysis (“Readme2”)

If you want datasets and code, the most direct entry points are Data Assets and Models & Code.

What are text-based AI systems in computational social science?

Answer: Text-based AI systems are pipelines that turn unstructured language (documents, posts, transcripts) into measurable variables you can analyze at scale. In practice, they combine NLP representations (features, embeddings, prompts) with statistical goals like measurement, prediction, record linkage, or causal inference.

• Inputs: raw text + metadata (time, source, author, unit of analysis).
• Outputs: labels, prevalence estimates, linked entities, or debiased parameters.
• Core constraint: you need validity checks—not just “the model seems accurate.”

Text classification vs. text quantification vs. text-based causal inference—what’s the difference?

Answer: Classification predicts a label for each document, quantification estimates category shares in a corpus, and causal inference estimates the effect of a treatment when text is part of the identification strategy. They use similar NLP tools, but the evaluation target (and the failure modes) are different.

Rule of thumb: If your estimand is a corpus-level proportion, “classify-and-count” is often the wrong default. If your estimand is causal, you must worry about whether your text is accidentally post-treatment.

Which paper should you read first (based on your project)?

Answer: Start with the paper that matches your bottleneck: validity of LLM labels, causal identification with text, removing a sensitive signal, linking entities, or measuring prevalence. Each overview is written as a checklist you can reuse in your own workflow.

1. If you use LLMs to label text variables: read Benchmarking Debiasing Methods for LLM-based Parameter Estimates for practical guidance on combining LLM labels with a smaller expert set.
2. If you adjust for text in a causal design: read Conceptualizing Treatment Leakage to avoid conditioning on post-treatment information.
3. If you want “text without X” (e.g., without sentiment): read Disentangling Text for an empirical stress test of LLM and human rewrites.
4. If you need to merge datasets without shared IDs: read Linking Organizations for ML + network approaches that outperform fuzzy matching in hard cases.
5. If you care about population shares over time: read Readme2 for direct quantification under distribution shift.

How do you debias LLM labels so your regression or causal estimates are credible?

Answer: Treat LLM annotations as cheap-but-biased measurements, then use a smaller set of trusted human labels to correct downstream parameter estimates. The key is to debias the estimand (coefficients, treatment effects), not just chase classification accuracy.

Debiasing vs. “just label more”: what’s the trade-off?

• Just label more (human-only): simple and reliable, but expensive and slow at scale.
• LLM + debiasing: cheaper scaling, but you must model and correct systematic error with an expert subset.

A practical workflow (copy/paste)

1. Define the parameter you care about (e.g., a regression coefficient or treatment effect).
2. Sample a high-quality expert-labeled subset (hundreds is often a useful starting point).
3. Label a much larger corpus with an LLM using consistent prompts.
4. Run at least one debiasing estimator (and, when feasible, two different ones as a robustness check).
5. Report both point estimates and uncertainty, and document the sampling design for expert labels.

Best for: projects where you need scalable annotation but still want valid inference (not just “pretty labels”). Watch out for: systematic error that correlates with style, topic, or demographic cues—those biases do not cancel out in regressions.

Read the debiasing benchmark overview →

What is treatment leakage in text-based causal inference, and how do you detect it?

Answer: Treatment leakage happens when your text contains information caused by the treatment, not just pre-treatment confounding information. Conditioning on leaked text can create post-treatment (collider) bias and can break overlap by making treatment nearly predictable from the text.

Fast leakage diagnostics

• Timestamp check: Was the text produced after treatment assignment? If yes, leakage risk is high.
• Predictability test: Can a simple Text → Treatment model predict treatment unusually well?
• Keyword scan: Does the document explicitly mention the intervention, program name, or downstream consequences?

What to do instead (text distillation)

Answer: Distill the text to remove treatment-related signal before using it for adjustment, then re-test predictability. If you cannot reduce treatment predictability without destroying the confounder signal you need, abstaining from text adjustment may be safer than “controlling for everything.”

Read the treatment leakage overview →

Can LLMs (or humans) actually remove sentiment or sensitive attributes from raw text?

Answer: Not reliably, if your standard is “a model can no longer predict the forbidden variable from the rewritten text.” Even when a rewrite sounds neutral to humans, residual statistical signal often remains—especially for concepts woven throughout the text (like sentiment in reviews).

Text-level rewrites vs. representation-level removal: which is better?

• Text-level rewriting: readable and auditable, best for localized deletions (names, addresses), but often leaves detectable traces.
• Representation-level methods: stronger at reducing detectability, but less interpretable because the “removal” happens in embedding space.

Practical guardrail: Don’t treat a “neutral rewrite” as a privacy or fairness guarantee. Always run a detection test (forbidden-variable classifier/adversary) on the transformed output.

Read the disentanglement overview →

How can you link organization names across datasets when there’s no shared ID?

Answer: Organizational record linkage (entity resolution) matches the same real-world organization across datasets even when names are inconsistent and IDs don’t exist. The most robust approach combines learned name similarity (ML) with network/community structure when alias relationships are “hidden” from string metrics.

Fuzzy matching vs. ML vs. network methods: what’s best?

• Fuzzy matching: fast baseline; fails on acronyms and brand families (low character overlap).
• ML similarity: strong general-purpose upgrade; learns patterns beyond edit distance.
• Network/community detection: best for “you just have to know” relationships when coverage is good.
• Ensembles: highest-quality merges when you can afford more compute.

Read the organization linkage overview →

How can you measure trends from text without labeling every document?

Answer: If your goal is a corpus-level answer (“what percent of documents fall in each category?”), direct quantification can outperform classify-and-count—especially when language shifts over time. Readme2 is designed for exactly this setting: prevalence estimation under distribution shift.

Best for / not ideal for

• Best for: time series monitoring, public opinion measurement, category shares in evolving discourse.
• Not ideal for: cases with no overlap between labeled and unlabeled language (extreme drift) unless you can refresh labels.

Read the Readme2 overview →

What are the most practical best practices for text-based AI research?

Answer: The most common failures in text-based AI systems come from mismatched goals (optimize accuracy when you need valid inference), untested leakage (post-treatment text), and overconfidence in “neutral” rewrites. A short checklist catches most issues before they get baked into results.

1. State the estimand first: prediction, prevalence, linkage, or causal effect?
2. Audit distribution shift: time, domain, platform, and population changes are the default.
3. Measure leakage, don’t assume: run predictability tests for treatment or forbidden variables.
4. Keep human ground truth: even a modest expert subset enables calibration and debiasing.
5. Save uncertainty: keep probabilities, multiple candidates, and sensitivity checks.
6. Publish artifacts: code + data + prompts + versioning improve reproducibility and reuse.

Next steps: Explore Models & Code and Data Assets for implementation details and released datasets.

What are the most common questions about text-based AI systems (and these papers)?

Do I still need human labels if I use LLMs for annotation?

Yes. Human labels provide the trusted reference you need to detect and correct systematic LLM bias, especially when your goal is valid inference rather than prediction-only performance.

When should I avoid controlling for text in causal inference?

If the text is produced after treatment (or clearly reflects treatment consequences) and you can’t distill out that signal, controlling for it can introduce post-treatment bias and destroy overlap. In that case, redesigning the identification strategy may be safer than “adding more controls.”

Is “neutral rewriting” a privacy or fairness guarantee?

No. A rewrite can look neutral while still leaving enough statistical signal for a model to recover the sensitive attribute. The only reliable check is a detection test on the transformed text.

Where can I find supporting code and data?

Start at Models & Code and Data Assets, then follow each paper’s overview page for replication links and downloads.