Two of three AI paper explainers made claims unsupported by their sources

Flashpapers found errors in explainers for Cross-Teacher Guarding and LatentLoop, including estimated compute savings presented as measured results. The company has added PDF checks and human approval.

Flashpapers, an AI-driven site that turns arXiv papers into interactive explainers, says its own experiment with three recent research papers exposed how easily a polished summary can drift away from the source text. In October, the publisher generated explainers for LatentLoop, Cross-Teacher Guarding, and Cortex, then checked each number, table reference and claim against the underlying PDFs before publication. According to the company, two of the explainers contained material that the papers themselves did not support, despite the pages looking authoritative to readers.

The most serious problems appeared in the Cross-Teacher explainer. Flashpapers says the page attributed results to a 1.7-billion-parameter model in settings where the paper only reported figures for the 4-billion-parameter version, then framed those numbers as evidence of consistent gains across model sizes. It also attached conflict-rate statistics to the wrong figure and described an additional scoring step as though it introduced no extra work at test time, even though the paper places that cost in training. The underlying paper is about Cross-Teacher Guarding, a method designed to reduce interference between multiple teachers by using cross-domain critics, agreement-based feedback and gradient projection.

LatentLoop had a different kind of error. The paper studies asymmetric compute allocation in looped mixture-of-experts systems and reports improved validation loss across fifteen configurations, but the savings headline is not a direct measurement: it comes from a fitted scaling-law estimate. Flashpapers says its explainer incorrectly presented that estimate as empirical and transformed a comparison of GPU hours into a claim of wall-clock speed-up. The paper’s own abstract makes clear that the 19.0 per cent compute saving is a prediction from the fit, not a measured runtime result.

The Cortex explainer, by contrast, was not singled out for invented numbers in the same way, but it still sat within the broader problem Flashpapers is warning about: generated summaries can sound precise while quietly overstating what a paper has actually shown. Cortex focuses on preserving the gains an LLM gets from context after that context is removed, using prospective query generation and distillation into low-rank or structural updates. The paper reports results across five task families and models from 4 billion to 120 billion parameters, including a sequential-edit experiment in which Cortex-Structural recovers most of the context-induced gain while retaining nearly all of the original non-target performance.

Flashpapers says the good news is that the headline numbers, formulas, most tables and author details were largely correct; the mistakes clustered in the very features readers are most likely to trust, such as interactive widgets and takeaway boxes. That concern is backed by broader research showing that multimodal systems often struggle more with charts than tables, and by work suggesting that domain-specific fine-tuning does not guarantee factual reliability. Flashpapers says it has now corrected the pages, added PDF checks for seeded explainers and introduced human approval before publication, a modest but telling reminder that fluent machine-written research summaries still need the same kind of verification as any other technical reporting.

Inspired by headline at: ^[1]

Sources by paragraph: - Paragraph 1: ^[2], ^[3] - Paragraph 2: ^[2], ^[4] - Paragraph 3: ^[2], ^[3] - Paragraph 4: ^[2], ^[5] - Paragraph 5: ^[2], ^[6], ^[7]

Source: Noah Wire Services · Original