A financial answer can be numerically right and still be grounded in the wrong company, filing period, or disclosure. That is an evidence failure, not a rounding error.
FinRank evidence errors are the point of a new benchmark built from U.S. public-company filings. Instead of asking only whether a system returns a plausible answer, FinRank separates retrieval, reranking, and hard-negative discrimination. The practical question is whether the model can find the passage that supports the answer for the intended company and reporting period while rejecting passages that look almost right.
The August 7 preprint reports 1,185 manually authored question-answer records over 10-K and 10-Q filings from 22 companies in pharmaceuticals, oil and gas, and automotive. The filings cover 2024 and 2025. The authors also release the dataset, taxonomy, repair log, validation tooling, and baseline harness in the FinRank repository.
The benchmark measures provenance, not just answer fluency
SEC filings repeat familiar language. Comparable companies discuss the same risks. One issuer may report similar figures across adjacent quarters. A retriever can therefore return a passage that is topically relevant but belongs to the wrong entity, date, or disclosure context.
The answer is not grounded until the cited evidence belongs to the exact claim being made.
That distinction matters anywhere a generated answer may be audited. Our review of the Lattice static retriever made a related point: retrieval speed and compact deployment do not remove the need to test what the system misses. FinRank adds a useful adversarial test for passages that a weak evaluator might accept.
What is actually inside FinRank
| Asset | Reported size | Why it matters |
|---|---|---|
| Question-answer records | 1,185 | Each record pairs a reference answer with gold supporting passages and metadata. |
| Companies | 22 | Three sectors create realistic same-topic and comparable-firm confusion. |
| Curated hard negatives | 6,021 | Plausible but incorrect passages test whether a model can reject near-matches. |
| Unique pooled passages | 5,230 | The retrieval task ranks evidence across a shared corpus rather than a tiny per-question set. |
| Hard negatives per record | 5.08 average | Musthave.ai calculation: 6,021 divided by 1,185. |
The hard negatives are selected from confusable evidence within filings, across reporting periods, and across comparable firms. This is more demanding than drawing random passages that are obviously unrelated. The benchmark also includes question-level metadata for document type, reasoning type, evidence scope, and difficulty, plus decomposed query rewrites for 69% of records.
The reported baseline gap is the warning
The authors report that the strongest tested 7B instruction-tuned embedder reaches 44.8% Recall@10 on the pooled evidence corpus. They also report that pairwise accuracy falls by 13.0 to 20.5 percentage points when curated hard negatives replace random negatives.
Those are author-reported benchmark results, not independent production measurements. Still, the comparison identifies a concrete evaluation mistake: random negatives can make a ranking system look safer than it is. A model that easily rejects an unrelated filing passage may still confuse two companies discussing the same metric.
The same lesson applies to model comparisons. Our Qwen benchmark rerank analysis explains why aggregate ranks can move when task design changes. FinRank shows the retrieval version of that problem: difficulty depends on the alternatives the system must distinguish.
The repair log is as important as the leaderboard
The repository ships a machine-readable repair log and deterministic validation tooling. The paper says 56 stored-label and question-ID disagreements are preserved in that audit trail. That is not a reason to dismiss the benchmark. It is a reason to treat dataset repair as part of the evidence.
Builders should copy this behavior. When an evaluation label changes, record the old value, the new value, the reason, and the records affected. Silent cleanup can make later score changes impossible to explain. This is the same evidence discipline we applied when reviewing OpenAI Astra’s published mathematics package: inspect the artifact and its audit boundary, not only the headline result.
What FinRank does not establish
- Each record had one annotator, and the paper reports no formal inter-annotator agreement statistic.
- The pooled corpus is a curated passage collection, not the complete universe of SEC filings a production search system would face.
- The benchmark concentrates on three sectors and two filing years.
- The released license is CC BY-NC 4.0, so commercial reuse needs a separate review.
- Retrieval and reranking scores do not prove that a downstream generator will quote, calculate, or qualify the evidence correctly.
These limits make FinRank an evaluation component, not a certification. A production financial assistant still needs filing ingestion checks, table extraction tests, calculation verification, source-date controls, and human review for consequential decisions.
A builder test: make the distractor look credible
- Collect questions where two companies or reporting periods use similar language.
- Require the retriever to return the exact filing, period, section, and passage identifier.
- Mix gold evidence with same-company prior-period passages and comparable-company disclosures.
- Score retrieval recall and pairwise discrimination separately.
- Run the generator only after retrieval passes, then verify every number and citation against the selected passage.
- Log false positives by confusion type: wrong entity, wrong period, wrong metric, or wrong disclosure context.
- Repeat the test whenever the filing corpus, embedding model, reranker, or chunking policy changes.
My verdict: test the evidence the model almost chose
FinRank is valuable because it makes a common financial QA failure measurable. The difficult mistake is not retrieving a random irrelevant paragraph. It is choosing a credible passage that belongs to the wrong company or period.
Use the benchmark as a template for your own domain. Build hard negatives from the errors a reviewer could plausibly miss, preserve a repair log, and keep retrieval evidence separate from generated-answer quality. A correct answer without correct provenance is still an unsafe answer.
Read the primary sources
- Read the FinRank preprint on arXiv.
- Inspect the released FinRank dataset and evaluation files.
Which near-match would be most dangerous for your retrieval system to accept?
Checked August 10, 2026. Dataset counts and benchmark scores are reported by the FinRank authors. Repository contents were inspected directly. The 5.08 hard-negatives-per-record figure is Musthave.ai’s calculation.
