One Night in Uzbekistan: How a Single Country's Data Derailed a Study
One night in Uzbekistan: Why was this one data point so influential?
A research team's findings couldn't be reproduced, and the culprit turned out to be anomalous data from Uzbekistan. This post from the Columbia University statistics blog recounts how a single country's data skewed the results, highlighting the importance of checking for outliers and the dangers of blindly trusting aggregate statistics.
We couldn't reproduce their findings and realized that it was all driven by weird data from Uzbekistan.
- Aurornis
> We wanted to see why Uzbekistan didn’t jump out, so we reproduced it in our comment (Extended Data Fig 1). It turns out that Uzbekistan wasn’t even the biggest outlier, but that the version they had published had the axes cropped so you couldn’t see the outliers (see red boxes in our version). This seemed indicative of a different issue, which is why we documented it in the comment.
Cropping the chart to hide the outliers is so bad that I can't tell if they're incompetent or malicious. I wouldn't be surprised if this is the kind of thing an LLM would produce in the hands of an operator not paying too much attention, but the paper was published in the time period before LLMs were everywhere in publishing.
- ktoyame
Makes me wonder how many more papers out there have hard-to-pin-down errors like that
And how useful potentially AI could be to spot those (even if retrospectively)
- MarkusQ
We need something akin to the international geophysical year, but for data integrity. Make it an interdisciplinary priority to clean house and root out papers that are hanging by a thread of included / excluded outliers, biased samples, and outright fraud. It would be humbling, but we'd be in much better shape afterwards.