A lot of analyses have been run on the Human Phenotype Project, and very few of them can be set beside one another. Each picks its own population, its own training and test participants, and its own baseline, so a result that looks stronger than another may only be answering an easier question. The cohort kept producing evidence that did not accumulate.
What it is
PhenoBench turns the Human Phenotype Project into a common test bed for any model, from classical baselines to large language models. It came out of my team’s evaluation workstream, and it is public so that other groups can put their own models through it. Gal Sapir led the project and first-authored the paper; I set its direction and worked hands-on inside it.
90 tasks across 15 clinical domains and 26 input modalities, over horizons from seconds to six years. Each question fixes its target, the eligible population, the timing, and the information a model may use, and its evaluation contract fixes the metric, the baseline, and how far the result may be read. We split participants into training, validation and test sets once, for the whole cohort rather than per study, so no analysis can pick a split that suits it. A new model can then be evaluated without redefining the question, so results from different groups and different years can be read against each other.
It also informs how we train and evaluate models internally, including the multimodal foundation models running alongside it. The plan is for every model we build to be measured on it, so that an improvement is a gain against a declared baseline rather than against whatever the previous project happened to use.
What it showed
We ran three pretrained tabular foundation models against ridge regression, gradient-boosted trees and a tuned neural baseline, over 160 matched comparisons spanning 52 tasks.
The pretrained models ranked highest overall. In 121 of 160 cells they came out ahead of the average of ridge and the neural baseline, and their improvement over ridge had a median of only 0.004 R2, with a 95% confidence interval of 0.002 to 0.006.
Mean rank within each of 160 matched comparison cells, best at the top, and the median gain over ridge.
Frozen zero-shot Chronos-2 embeddings underperformed features engineered from the same signal, in 25 of 31 glucose comparisons and 30 of 32 sleep comparisons. The caveat travels with that result: both routes were read by the same linear ridge head, so it mixes representation quality with the sample efficiency of fitting ridge to embeddings of 770 to 3,076 dimensions. It does not test whether a tabular foundation model could use those embeddings differently.
What the contracts catch
Every language model we tested predicted retinal artery width on a micrometre-like scale. The target is in AutoMorph pixels, and the answers stayed off-scale even after the Task Card’s reference values were supplied, under both versions of the card.
It shows up because each question states what is being predicted, in which units, and against which baseline.
Where the claim weakens
A matched comparison assumes every model was held out from the same participants, and not every model is. Some predate that shared split. Some were trained by collaborators who used their own.
The evaluation strategy still holds in those cases: the same question, the same metric, the same baseline. What changes is how far the result can be read, from held-out evidence to a description of what a measurement route can do, and the paper reports them that way.
Where it goes next
So far the language-model work reuses tasks written for fitted models, under a single prompt and evidence contract: 14 models across 40 of the 90. The next step is tasks and evaluation strategies designed for language models rather than adapted to them.
The metabolic health agent predates PhenoBench. That project built its own evaluation harness, because there was no shared place to define what an agent should be measured on. Those questions would be PhenoBench tasks now.
Agentic auto-research loops are the further version of that. An agent that proposes and runs its own analysis needs somewhere to be checked against a fixed question, held-out data and a declared comparator, which is what an evaluation contract already is.
Read it
The public benchmark has the task catalog, the Task Cards, per-track results and a submission path.