Evals
Suites run against pinned datasets on every agent version. The column that matters is the delta, not the score — a 94% pass rate means nothing until you know it was 97% last week.
Suites
4
Cases
5,690
Regressions
1
| Suite | Agent version | Cases | Pass rate | Δ vs last | Last run |
|---|---|---|---|---|---|
invoice-extraction invoices-2026-q2 · 1,240 docs | v4.3.0-rc1 DataAgent | 1,240 | 94.1% | -2.8 pt | 40m ago |
invoice-extraction invoices-2026-q2 · 1,240 docs | v4.2.0 DataAgent | 1,240 | 96.9% | +0.4 pt | 3d ago |
earnings-summary-faithfulness sec-10q-sample · 320 filings | v2.8.1 FinanceBot | 320 | 91.2% | +1.7 pt | 11h ago |
incident-triage-precision pagerduty-replay · 890 incidents | v2.0.0-rc3 OpsAgent | 890 | 88.3% | +6.1 pt | 6h ago |
tool-schema-conformance synthetic · 2,000 calls | — all | 2,000 | 99.8% | — | 1h ago |
What regressed in invoice-extraction
-2.8 ptv4.3.0-rc1 against DataAgent’s previously deployed build, same pinned dataset.
multi-page invoices with rotated scans
Was
pdf.parse fallback recovered the line items
Now
retries OCR twice, then times out before reaching the fallback
invoices with handwritten totals
Was
flagged for review
Now
returns the printed subtotal without flagging