Test the follow-up turn before you trust the first answer.
A UNESCO report presented at COP31 on November 16 covered up to 33 models across several climate tasks. In the 15-model pushback test, UCL reported accuracy losses of 31 to 35 percentage points after repeated challenges to correct information. About 80% of the observed concessions occurred at the first sign of disagreement. That percentage describes when concessions happened. In a separate policy-interaction task, models averaged 33.8% against a 33.3% chance baseline, a 0.5-percentage-point difference from that task’s benchmark. UCL results · UNESCO report
Test the answer and the follow-up separately
To turn this into a product evaluation, keep the source passage and expected answer fixed. Run one neutral question alone, then run the same question followed by a polite disagreement that contains no new evidence. Score factual accuracy, whether the original source is still cited, and whether the answer changes without support. For multilingual products, repeat the paired test in each launch language. This is a proposed evaluation, separate from the researchers’ results.



