Skip to content
MustHave.ai Musthave.ai
Models

Sakana AI releases JFinBench for Japanese finance

Sakana AI’s JFinBench covers six Japanese finance exams. The public download includes 264 of 321 questions. See what a full rerun needs.

Sakana AI releases JFinBench for Japanese finance
A complete JFinBench run uses the hosted questions and examples rebuilt from publisher materials.

Downloading the public dataset gets you 264 questions. The full benchmark has 321.

Sakana AI introduced JFinBench on November 17 to test how language models handle Japanese finance. The benchmark draws from the 2026 sittings of six Japanese professional certification exams. Its questions cover accounting, tax, financial planning, and banking. Some ask the model to read an image or supply a written answer. The preprint was submitted on November 13.

The public download covers about 82% of the questions

Sakana hosts 264 examples and provides code to reconstruct the full benchmark from publisher PDFs. Some questions and answer keys remain with their respective publishers. Subtracting 264 from 321 leaves 57 examples that require reconstruction. That is 17.8% of the full set. Use the benchmark’s data and code links to obtain the appropriate artifacts under their stated terms.

That missing share matters when comparing a new run with a full-set result. A quick run on the hosted rows has its own denominator and may have a different mix of exam topics or answer formats. Label that result as a 264-question run. If you later recreate the 321-question set, keep the subset result so you can see how the extra material affects the comparison.

One leaderboard can hide two different test conditions

The authors evaluated 12 models and compared runs with and without access to a terminal and the web. In their tests, every model improved with those tools. The authors report that the gap between Claude Fable 5 and the strongest open-weight model in their sample, Kimi K3.1 Pro, narrowed from about 12 percentage points to about four. These figures are for the authors’ specific setup. Record tool access with a score so someone can reproduce the same comparison.

The Japanese announcement also describes different grading procedures. Multiple-choice answers can be extracted with patterns. More open-ended responses use model-based grading with rules for equivalence, partial credit, or rubric coverage. Changing the grader can move scores even when the test-taker model gives the same answer.

Give a rerun enough detail to compare

Record the commit of the evaluation code, the exact set of question IDs, the model version, the allowed tools, and the grader. Publish scores by exam category before choosing a single aggregate. If you change any of those pieces, rerun a fixed baseline through the same setup. That lets you separate changes in model performance from changes in how it was evaluated.

Sakana’s refresh process gives teams a way to bring in the next exam cycle. Version the question set when doing so. A higher score on a new edition describes performance on that edition and needs a matched comparison before it supports a claim about progress.

Abdessalam Alaoui

I'm Abdessalam Alaoui, founder of Musthave.AI. I write practical guides to AI models, tools, generative media.

Join the discussion

Your email address will not be published. Required fields are marked *

Keep reading

All news