AI agent testing has a familiar problem: the most realistic inputs often exist in production, where discovering a regression is already too late. Raindrop wants to move that evidence into the pull-request loop.
Raindrop Simulations is an early-access product that the company says replays production traffic and existing test cases against every pull request, then looks for anomalous behavior before deployment. Raindrop announced the product alongside a Series A led by CRV, bringing total funding to a company-reported $50 million.
The product thesis in one sentence
Instead of asking teams to predict every correct agent answer, replay representative real interactions and flag behavior that changed. This is attractive for agents because valid outputs can vary while still needing stable permissions, costs and task results.
How simulations differ from a fixed benchmark
| Method | Good at | Blind spot |
|---|---|---|
| Golden tests | Checking known inputs against expected outputs. | Misses new production patterns and acceptable variation. |
| Model benchmark | Comparing a capability across systems. | May not represent one company’s tools, data or users. |
| Production monitoring | Finding incidents in live traffic. | Detection occurs after exposure. |
| Pull-request replay | Testing a candidate change against past traffic. | Only as representative and safe as the replay set. |
What Raindrop claims
Raindrop says Simulations can work across agent harnesses and create a world around the agent rather than requiring one rigid evaluation format. It also says the system can replay production traffic and test cases on each pull request and detect anomalies before the change ships. These are company descriptions; independent comparative results are not yet public.
The useful question is what counts as an anomaly. A changed answer may be an improvement, a harmless wording shift or a serious loss of tool discipline. Buyers need to see the feature representation, thresholds, review queue and false-positive rate before making it a deployment gate.
Production replay creates a data-governance job
Real traffic can contain personal data, secrets, customer content and time-sensitive permissions. Copying it into an evaluation environment can multiply exposure. A safe design should minimize fields, redact secrets, preserve tenant boundaries and expire replay data on a defined schedule.
- Record whether the replay uses raw, redacted, synthetic or sampled events.
- Prevent simulated tool calls from reaching real systems or sending messages.
- Keep the authorization state that mattered, without reusing live credentials.
- Separate model regressions from tool, prompt, retrieval and policy changes.
Five questions for an evaluation buyer
- What is the unit of comparison? A final answer, a trajectory, a tool call or a business outcome?
- How is drift scored? Ask for thresholds, reviewer workflow and false-positive data.
- Can a replay cause side effects? Require inert tools or a controlled sandbox.
- How are sensitive events handled? Inspect redaction, retention and tenant isolation.
- What blocks deployment? Define which failures stop a pull request and who can override them.
Funding supports a larger category bet
The Series A is not proof that the product works, but it signals that investors expect agent reliability to become its own infrastructure category. Raindrop names CRV as lead and says researchers and executives from major AI labs participated. The company expects broader availability over the coming month after early access.
The category is moving from model evaluation to system evaluation. Our guide to held-out agent safety tests explains why an agent should not see or rewrite its own final test. The Google agent anomaly detection preview shows a parallel enterprise approach.
What to watch during general availability
The decisive evidence will be operational: supported harnesses, replay isolation, review burden, false alarms, time to diagnose and defects caught before release. A useful simulation system should make a failure reproducible and attributable, not merely label a pull request risky.
Teams should also measure escaped regressions. A tool that catches many cosmetic changes but misses permission or side-effect failures can create false confidence. Keep a small held-out incident set outside the tuning loop, then compare pre-deployment findings with what still reaches production.
The bottom line
Raindrop is aiming at a real gap between offline benchmarks and live monitoring. Replaying representative traffic before production can expose regressions that synthetic tests miss. The product should be judged on safe replay, explainable anomalies and whether teams act on its findings, not on the funding headline.
Read the primary record
Checked September 18, 2026. Product details and reported results come from the linked first-party sources. Interpretation, limitations and implementation advice are MustHave.ai analysis.
