Part 04

How two datasets that share no key were joined

Names, institutions, degree levels and years, plus a local model reading 144,101 abstracts. Short enough to reuse: this is what it takes to put a subject on a career record.

70,624 one-to-one matches · 99.7% institutional coverage · all model inference run locally

Two datasets that share no identifier were joined on people’s names. This page is short on purpose: it is the recipe, and the two checks that say it worked.

The key

Normalized name × institution × degree level × graduation year. Names reduce to first-plus-last, lowercased and accent-stripped. Middle names are discarded, because the two sources record them inconsistently, and that inconsistency was the single largest source of missed matches in early passes.

Institutions are not one-to-one, and assuming they are destroys the data

The hardest part was not names. It was that an institution on one side does not correspond to a single institution on the other, and it fails in both directions:

The entire University of Minnesota system’s records, 483,971 of them, sit under one identifier whose name is the Duluth branch campus. There is no Twin Cities record at all. A crosswalk insisting on exact institutional identity drops the flagship.
Rutgers is the opposite: split across at least six identifiers, none of them named simply “Rutgers University”.

So the crosswalk is one-to-many. Each degree-granting institution maps to the set of career-panel identifiers that plausibly denote it, and matching blocks against that whole set. Candidates are accepted only when the shorter cleaned name is a token prefix of the longer one, in either direction.

That rule replaced a looser similarity score which silently absorbed Washington State University into University of Washington, and California State University, Los Angeles into UCLA. Both score a perfect token-set match, because the distinguishing word sits in the middle. Prefix matching rejects both, and still accepts “University of Minnesota” → “University of Minnesota Duluth”.

Three further defects each destroyed whole institutions before they were caught: HTML entities left unescaped in the panel’s own pre-cleaned name column, so Texas A&M carried a literal token “amp”; mojibake in non-ASCII names; and a US-state field that is simply wrong for some institutions (Tufts is recorded in Oregon), which forced a fallback from state-restricted to nationwide search. Final coverage: 99.7% of dissertations, averaging 1.9 identifiers per institution.

One-to-one or nothing

A match counts only if it is unique in both directions. If a dissertation matches more than one profile in its block, or a profile matches more than one dissertation, it is recorded as ambiguous and excluded rather than resolved by a tiebreak. That throws away 7,343 candidate pairs. The alternative would be manufacturing precision by decree.

70,624 clean one-to-one matches out of 144,101 dissertations in the 2000–2025 window, 49% of them. Graduation years agree exactly in 88%.

Does it work

Two checks. The placebo: hand every dissertation’s institution set to a different dissertation and rerun the strictest tier, leaving name, degree level and year untouched. What survives is the rate at which two different people happen to share all three. The placebo matches 0.66% against a real rate of 45.7%, about seventy times chance.

A variable held out of the join: field of study was deliberately kept out of the key, which leaves it free as an independent test. Among matched profiles, two thirds record their field as Engineering, the panel’s catch-all for computing. In the graduate population at large it is one tenth. Business runs the other way. If the matches were noise the two distributions would be identical.

Field of study recorded by matched profiles. This variable was excluded from the matching key, so it is an independent test.

The same variable across all US graduate-degree records in the panel. Engineering runs 10.8% here against 67.1% above, a 6.2-fold enrichment on a field that never entered the matching key.

The match rate is a measurement, not a score

It climbs monotonically from about a third for the 2000 cohort to over half for recent ones, which is what growing coverage in resume data looks like. Read it as an estimate of how much of a computing cohort is observable in the US labor market at all.

Share of each cohort matched to a career profile, by degree level.

Reading the dissertations

Subject and method do not come from a keyword list. Every abstract in the corpus, 144,101 of them, was read by a local open-weight model and asked for a subject area out of a bounded taxonomy plus a set of method flags: does this work use machine learning, deep learning, a large language model, and is it core computer science or adjacent work filed under a computing department.

How far the labels carry was measured by relabelling samples with an architecturally different model on the identical prompt. On a sample of 360 dissertations drawn so that it actually contains work using each technique, the second model agrees on the flags this one assigned: machine learning 98.6% of 279, deep learning 94.5% of 238, large language model 92.5% of 120. The subject taxonomy reproduces at 86.5%, and 89% once collapsed to eight coarse families. So the method flags carry levels, and the subject labels carry trends and large gaps.

Provenance

Dissertation metadata and the career panel are both licensed subscription data, accessed through a university subscription and processed on a shared academic cluster. All model inference ran on local hardware, because neither source may be sent to a third-party service. The career extract runs through July 2026 and is frozen at that point.