TMLR Editor Calls Authors About Their Own Papers—Most Can't Answer Basic Questions
Asking Authors About Their Own Papers
Nihar B. Shah, an Editor-in-Chief of TMLR, personally interviewed authors of 10 papers slated for desk rejection. Three couldn't answer basic questions about their own work, three more struggled with technical details, and only one answered everything—though Shah found a major flaw. Two later sent answers that Pangram flagged as 100% AI. The experiment suggests desk rejection is working, but raises concerns about authorship credit and AI-assisted submissions.
In a separate meeting, the author of one paper tried to describe the methods they used for analysis, but inadvertently ended up describing an entire p-hacking workflow.
- fn-mote
The Medium comments on this post are also on point. Running the same experiment with accepted papers is a good control. Running a similar experiment with reviewers would be interesting, but more obnoxious because they are not being paid.
I would keep a private blacklist (shadow ban) the authors who wasted several hours of a reviewer's time to prove they were not legitimate. The existence of such a list would be problematic, though.
Could the same system we use here be applied? Accepted authors could "vouch" for "dead" papers in case they were "auto-killed"?
This system is broken and providing more evidence that it is broken isn't much of a step towards fixing it.
- greenflag
One larger problem here is the value of a research paper is rarely the specific knowledge it adds but in the process of researching that adds to the collective knowledge+experience of those involved, especially training graduate students. AI papers shortcut this entirely. Academia has a lot to answer for this too by making papers the currency of success. AI generated papers are almost shortcut learning at a full system level.
- syntamono
In the same vein, the Symposium on Theory of Computing (STOC) for 2027 has made some interesting changes to its Call For Papers (https://acm-stoc.org/stoc2027/stoc2027-cfp.html), notably requiring that papers be submitted beforehand to a preprint repository and that authors submit a 20-30 minutes video presentation explaining their results.
- danieltanfh95
Academia is in a time for reckoning.
Actors who already went through the process (or otherwise) gained sufficient reputation or credentials to self-market their own paper can skip journals entirely. That was what OpenAI did. With a sufficiently powerful AI model and correctional pipelines, generating a paper is trivial given some insight.
I think we should be reminded that papers are a channel to distribute papers. Editors are unpaid now, but the economics of a journals are such that the editors are incentivized to curate or distribute papers to schools that pay for the paper. Some perceive quality as a core metric for this. However, in my experience of dealing with computational biology, a paper in so and so journal hardly means a stamp of quality as compared to a paper in some github repo with code to reproduce the paper. This, simply, is broken, because journals and peer reviewers cannot guarantee that data and results in the paper is correct (assuming that it is not maths or theoretical) without reproducing the results in the paper.
None of these are helpful towards students who are already struggling to keep up with the cadence of producing papers.
- c7b
> Separately, our group has been exploring approaches along these lines to make such evaluations more scalable
Actually, that sounds like an interesting idea for peer review in general, to include an interview between referees and authors. If it saves one round of rebuttals/reactions, it needn't even consume a lot more of everyone's time if you're doing those things properly. What it would undermine would be blindness, but something's gotta give, and it was already on its way out.
- mlmonkey
IMHO (not a paper writer, but read a lot during my grad school years), the Genie is out of the bottle. The only way forward, as I see it, is using LLMs for reviews also. Basically, filter all submitted papers with an LLM and ask it to summarize it, find the biggest weaknesses and main strong points, etc. that a human can then use to review the paper. Basically, LLM-as-a-reviewer .
Personally, I would love to see a conference where people are explicitly encouraged to use LLMs for doing the work and writing the papers, and LLMs are used to review them too.
- figassis
> When authors could not answer questions about the technical parts, and sometimes even basic questions about the paper, it is difficult to see how they could have verified the paper's contents
If you cannot answer, you did not author the paper, meaning you are misrepresenting your contribution, and there is already a process for this. And this is actually a really good test for any field. Use AI as much as you want, but you need to be able to explain your work. Applies to SWE as well, you need to understadn what you built, at the code level and system level.
- JasonCEC
Peer review has its historical issues, but the landscape of science and science-publishing has changed. New problems of authorship and authorial-understanding are now challenged by LLMs writing (at least) good sounding papers - some of which might be of acceptable quality in subject (I am not against AI in the sciences; some of the math work has been great). On the other hand: I am against authors not understanding their own work. High repute journals may need to add "oral exams" to the paper acceptance process...