Susmit Das
← back home
Under review at ICLR 2027

Ephemeral Models,
Durable Claims

short public explainer · manuscript not hosted here during review
OpenReview ↗

A paper can outlive the model it used. I wanted to know how often modern AI research already depends on closed computational artifacts that researchers themselves cannot preserve.

The thing that bothered me

If you can keep the model

A locally retainable model can usually live with the rest of the research artifacts: code, prompts, data, scripts, environment details, and so on.

If you cannot

With a provider-controlled model, you can record what you used, but you still do not control whether that exact computational object exists later.

What I actually counted

I froze a census of 53,538 final archival papers from ACL, EMNLP, ICLR, ICML and NeurIPS from 2023 to 2026 where available, then measured whether covered closed models played one of four scientific roles. Writing help or coding assistance was tracked separately and excluded from the main measure.

Directthe model is part of the experimental system or object of study
Baselinethe model provides a comparison condition
Judgethe model scores or evaluates other systems
Data generationthe model creates or annotates study material

How I measured it

Archive census53,538 papers
High-recall retrievalfind plausible model mentions
Deterministic triageresolve clear cases
LLM adjudicationambiguous entities only
Freeze + validateblind human sample
Lifecycle joinfirst-party status evidence
0.967end-to-end F1 for Scientific Use
0.888four-role micro-F1
200papers in blinded validation
302implementation tests

What showed up

33.32%17,841 of 53,538 papers had measured Scientific Use
15 → 38%common five-venue prevalence, 2023 to 2025
1.47 → 11.50%Judge prevalence, 2023 to 2025
7 / 20most-used normalized models already retired at cutoff
This is not an irreproducibility rate. It is a map of where the research record depends on computational artifacts whose identity or future availability sits outside the scholarly archive.

Two trends that are easier to see as graphs

Closed-model Scientific Use became common fast

2023
15%
2025
38%

Shown here for the common five-venue slice discussed in the paper.

The judge role grew especially quickly

2023
1.47%
2025
11.50%

This was not just about models being used as systems. They were also being used to evaluate other systems.

Why I care

I dislike the idea that a durable scientific claim can outlive the very model behavior it depended on, with no way for a later researcher to interrogate the same artifact again. The practical fixes are not glamorous, but they matter: exact identifiers, preservation-minded model choices, archival traces, and some form of post-commercial research access for high-footprint models.