Evaluations
Agents even more accurate than your best people
Varick builds a custom evaluation suite for every agent from your own past cases, graded by your experts and Varick's expert annotators.Custom test suites from your own cases, graded by experts.
Release 14
Running suiteAP invoice matching, suite v9 · 1,860 cases × 5 runs · Nov 18, 9:12 AM · 38 min · graded in your cloud
Pass rate
…
Bar 98.0%
Cases graded
0
of 1,860, 5 runs each
Critical errors
…
Rule: zero
Regressions
…
Against release 13
Custom suites
Every agent gets its own test suite
Built from your own past cases, including the rare ones that cost the most.
Cases
Of history
Source system
Your team today, against the agent's bar
Set by your AP manager and controllerHard cases it covers
- Freight billed separately
- Remit-to address changed
- Credit memo, partial receipt
- Euro invoice, dollar PO
Graded on, every criterion must pass
- PO and receipt match
- Variance within tolerance
- GL account and cost center
- Right approver
Expert annotators
Graded by people who have done the work
Controllers, CPAs, procurement and payroll leads, contract lawyers, and IT service managers label every case. Two label it on their own, and a senior reviewer rules where they disagree.
Invoice INV-19877
Halden Freight
Billed Nov 3. Goods received Oct 15.
Supply agreement, section 4.2
Freight billed separately is payable up to $1,500 per shipment for goods received against a valid PO.
Freight is not on the PO.
The contract covers freight up to $1,500.
Section 4.2 allows freight billed separately.
Rubric v8 to v9
Freight not on the PO: hold.
Freight not on the PO: check the supply contract and approve up to its limit.
Grading
Three layers grade every answer
Here is one invoice the agent held for a price 4.1% over the PO, graded three ways. Each case runs five times, and all five runs must pass.
Rules
Exact checks for everything that can be checked exactly.
- PO 4500-2231 matched to receipt GR-88120Pass
- $1.85 over on 2,400 units is $4,440.00Pass
- 4.1% is over the 2% tolerance, so holdPass
- Sent to the buyer who owns the POPass
Checked in all five runs
Model graders
Judgement calls, scored against the experts' rubric.
- Names the right line and causeCites line 3 and the unit price.5 of 5
- Clear for the buyer to act onStates the amount and next step. Passes at 4.4 of 5
- Every figure traces to a documentPrice, units, and tolerance match the PO.5 of 5
97.4% agreement with experts on 400 held-out cases
Experts
Every disagreement, every critical error, and a weekly sample.
- Drawn into this week's sampleSampled
- Former AP manager agrees with the agentAgrees
- No critical error to reviewClear
Any expert ruling updates the rubric
INV-20420 · Corvo Industrial · $112,480.00
Hold for the buyer. Line 3 is billed at $46.85 a unit against $45.00 on the PO, 4.1% over your 2% tolerance.
- Run 1Pass
- Run 2Pass
- Run 3Pass
- Run 4Pass
- Run 5Pass
From pilot to production
The suite runs the whole engagement
Cases from every path process mining finds, weighted to the rare ones, with a rubric your team signs off.
Path, from process mining
Share of the work
Share of the suite
Cases
- Standard 3-way match71%33%620
- Partial receipt11%14%260
- Price variance over tolerance8%16%300
- Freight on its own invoice4%13%240
- Credit memo applied3%12%220
- Foreign currency3%12%220
Rare paths get more cases than their share, since they cost the most when they go wrong.
Model selection
Every model, tested on your cases
Each step runs on the cheapest model that clears its bar. When a new model ships, the suite runs again before anything changes.
Step
93%Accuracy on your cases100%
Chosen
- Read invoicesVision99.2%2.1s · $1.80 per 1,000
- Code GL linesFast98.4%0.4s · $0.12 per 1,000
- Explain price variancesReasoning96.9%9.8s · $6.40 per 1,000
- Find vendor historyEmbeddings97.6%0.2s · $0.04 per 1,000