Reading the whole population
Four populations, each read in full rather than sampled: 149,364 papers, 131.7 million job postings, 144,101 dissertations, and fifteen years of science hiring. The question underneath all of them is what changes about empirical work when reading the whole population stops being expensive.
A literature review picks keywords and reads what comes back. A labour statistic picks a sample and asks it questions. Both designs exist because reading everything was impossible, and both quietly inherit whatever the selection step happened to miss.
That constraint has moved. Annotating a million documents with a language model is now a matter of days on hardware that fits in a room, and the interesting consequence is not that the old studies get cheaper. It is that the selection step can be removed altogether, and once it is removed you can measure how much it was costing you.
Each of the four corpora below is a population, not a sample. Each was labelled document by document by a model. What follows is what that bought, what it cost, and the one place the cost turned out to sit.
Why keywords fail, measured
The clearest result so far comes from the self-driving-laboratory corpus, because there the two designs can be run against each other on the same pool.
Papers were classified against a definition (a two-axis test on how much of the experimental loop is software-controlled and how much is physically automated) rather than against the phrase “self-driving laboratory”. Of the 845 papers the classifier accepted as relevant:
820 of 845. The phrase list was not hand-picked to make the point; it is the 42 surface variants generated mechanically from the 34 canonical queries the round-1 keyword gate had already produced. Tighten the list and the number goes up, not down.
The papers that go missing are not marginal. They are run-to-run control in semiconductor fabrication, reinforcement learning for single-molecule manipulation, nonlinear model-predictive control of fed-batch fermentation, automated stem-cell production, evolutionary algorithms in photonics. The method is spreading across fields under other names, and a keyword survey cannot see any of it.
Two recall channels found those papers: citation expansion alone found 399, generalised (non-brand) keywords alone found 337, and 84 were found by both. Neither channel is redundant, and neither is what a conventional review does.
The four populations
| Corpus | Population | Annotated | Annotator | Question |
|---|---|---|---|---|
| Self-driving laboratories | 149,364 papers | 125,757 | gpt-5-mini, batch | Which experiments are already automated, at the level of the unit operation |
| US job postings | 131,706,672 postings | pilot cut | Qwen3.6 35B-A3B, local | What tasks employers actually ask for, and what AI exposure looks like task by task |
| Computing dissertations | 144,101 theses | 144,101 | local model, on cluster | What CS graduates worked on and where they went |
| Science job postings (not yet public) | 2010–2025 | 4,717 task nodes | GPT-4o-mini | A poly-hierarchy of what scientists are hired to do |
They are deliberately different shapes. Two are literatures and two are labour markets; two were annotated through a hosted batch API and two on local GPUs. The design question is the same in all four: what does the label have to be, for the population to be worth reading in full?
What full coverage costs
The job-posting corpus is the honest test, because at 131.7 million documents nothing is free. The throughput below is measured, not estimated: ten independent local workers (eight V100s, one Spark, one consumer card), each running one llama.cpp instance of a 35B mixture-of-experts model with roughly 3B active parameters, quantised to 4 bits, decoding at about 100 tokens per second with speculative decoding on.
The input side barely matters here. Every posting is about 960 tokens of body text and prefill runs at roughly 2,000 tokens per second per card on a model with 3B active parameters, so reading is cheap and reading more is cheap. Writing is what you pay for, and in a low-concurrency setup you pay for it linearly, with none of the batching relief a high-throughput serving stack would give you.
The practical rule that falls out is unglamorous and worth stating plainly: make the model think and then emit almost nothing. A study that wants a chain of reasoning attached to every document is not a study of a population; it is a study of a sample, and it should be designed as one from the start.
Where the cost actually goes
The throughput chart is the visible cost. The one that surprised me sits in the prompt.
In the task-matching run, each posting is matched against the O*NET intermediate work activities: 332 candidate items, presented in full so the model can choose among them. Measured against a hosted model on real sampled bodies:
Eighty-nine percent of the prompt is the answer sheet, not the question. That is not a modelling choice anyone made on purpose. It is what happens when a classification schema designed for human coders gets handed to a model verbatim, once per document, for a hundred million documents.
It can be recovered here, because a stable candidate list sits at the front of the prompt and caches. But the general shape of it does not go away, and it is the reason I think the binding constraint on this kind of work is no longer the price of the model. Two studies with identical scientific content can differ by an order of magnitude in cost depending on how the schema is shaped, and nothing in the current methods literature tells you which shape you are choosing.
The open question
All four corpora were built to answer substantive questions, and those questions have their own papers. What none of them has yet is a home for the method itself.
A full-population annotation is not a survey in the statistical sense and not a review in the bibliographic sense. It has the coverage of a census and the label quality of a coder, and the things it gets judged on are unlike either: not sampling error but definitional drift, not inter-coder reliability but prompt stability, not response rate but what fraction of the population the model refused to commit on. The self-driving-laboratory corpus, for instance, reports 75.4% of 98,298 classifications as noise, which is a meaningful number in this design and a meaningless one in the frameworks it would otherwise be evaluated against.
I do not think that gap is permanent, and I do not think it is only my problem. It is the question I am currently most interested in talking to people about.