Skip to content
MustHave.ai Musthave.ai
Models

UNESCO and UCL tested what happens when users challenge correct climate answers

A UNESCO and UCL study found that climate-answer accuracy fell under repeated challenges. Use a paired question to test for the same failure in a product.

UNESCO and UCL tested what happens when users challenge correct climate answers
UCL reported declines of 31 to 35 percentage points after repeated challenges.

Test the follow-up turn before you trust the first answer.

A UNESCO report presented at COP31 on November 16 covered up to 33 models across several climate tasks. In the 15-model pushback test, UCL reported accuracy losses of 31 to 35 percentage points after repeated challenges to correct information. About 80% of the observed concessions occurred at the first sign of disagreement. That percentage describes when concessions happened. In a separate policy-interaction task, models averaged 33.8% against a 33.3% chance baseline, a 0.5-percentage-point difference from that task’s benchmark. UCL results · UNESCO report

Test the answer and the follow-up separately

To turn this into a product evaluation, keep the source passage and expected answer fixed. Run one neutral question alone, then run the same question followed by a polite disagreement that contains no new evidence. Score factual accuracy, whether the original source is still cited, and whether the answer changes without support. For multilingual products, repeat the paired test in each launch language. This is a proposed evaluation, separate from the researchers’ results.

Abdessalam Alaoui

I'm Abdessalam Alaoui, founder of Musthave.AI. I write practical guides to AI models, tools, generative media.

Join the discussion

Your email address will not be published. Required fields are marked *

Keep reading

All news