Skip to content
Varick Agents

Evaluations

Agents even more accurate than your best people

Varick builds a custom evaluation suite for every agent from your own past cases, graded by your experts and Varick's expert annotators.Custom test suites from your own cases, graded by experts.

Custom suites

Every agent gets its own test suite

Built from your own past cases, including the rare ones that cost the most.

Expert annotators

Graded by people who have done the work

Controllers, CPAs, procurement and payroll leads, contract lawyers, and IT service managers label every case. Two label it on their own, and a senior reviewer rules where they disagree.

Grading

Three layers grade every answer

Here is one invoice the agent held for a price 4.1% over the PO, graded three ways. Each case runs five times, and all five runs must pass.

Rules

Exact checks for everything that can be checked exactly.

  • PO 4500-2231 matched to receipt GR-88120Pass
  • $1.85 over on 2,400 units is $4,440.00Pass
  • 4.1% is over the 2% tolerance, so holdPass
  • Sent to the buyer who owns the POPass

Checked in all five runs

Model graders

Judgement calls, scored against the experts' rubric.

  • Names the right line and causeCites line 3 and the unit price.5 of 5
  • Clear for the buyer to act onStates the amount and next step. Passes at 4.4 of 5
  • Every figure traces to a documentPrice, units, and tolerance match the PO.5 of 5

97.4% agreement with experts on 400 held-out cases

Experts

Every disagreement, every critical error, and a weekly sample.

  • Drawn into this week's sampleSampled
  • Former AP manager agrees with the agentAgrees
  • No critical error to reviewClear

Any expert ruling updates the rubric

From pilot to production

The suite runs the whole engagement

Cases from every path process mining finds, weighted to the rare ones, with a rubric your team signs off.

Model selection

Every model, tested on your cases

Each step runs on the cheapest model that clears its bar. When a new model ships, the suite runs again before anything changes.

  • OpenAI
  • Anthropic Claude
  • Google Gemini
  • xAI Grok
  • Meta
  • Moonshot Kimi

Frequently asked questions

Book a call