Fixing Enterprise AI Pilots

7 Common Cases of AI Pilot Failure

If you're a CFO, COO, or even CIO, you've likely gone through 3-4 AI pilots in the last 18 months. One or two of them are probably still running, and for most companies none of them are reflected in your P&L. The reasons for failure are always attributed to either data, model, or some vague “change management” related issue.

Most of the time, none of these are the root cause. We've been called in after a lot of failed pilots, and have narrowed it down to several reasons why they commonly fail. Here's the longer form of the top 7 most common reasons why pilots fail, in order of frequency. (Note: model selection is not on the list).

No process map

Let's use a single process - vendor onboarding in procurement - as an example (and before you make fun of it, recognize your own company almost certainly has a version of this that takes 3 weeks).

There’s probably around six overarching steps for this on paper: vendor onboarding requests, procurement checks, W9/banking details, bank verification, sanctions checks, and logs in SAP/NetSuite. If you follow the SOP, it takes 5 business days. In reality it takes 22 days. The form is often filled incompletely. Bank details come in a variety of formats (including letterhead + Gmail). Treasury calls to confirm details. Compliance has a lot of false positives to weed through. If the vendor is software, there's a separate infosec questionnaire. 1/10 vendors are already in your vendor master under a slightly different name. The list goes on.

Tap to enlarge (opens the full-size image)

But your pilot was scoped based on the SOP - and so when it gets to a real world environment, where exceptions are the norm (>40% often), it all falls over. The fix is to do process mapping. Talk to the team who does this process in their regular jobs and mine the event logs (e.g. ERP, ticketing system, email inbox) to understand what actually happens. From there, you can quickly find out that it's different from what you think should happen.

We work with extremely large companies ($1B+ in revenue, 1k to 100k employees), helping them adopt AI from general strategy, to process re-engineering, to building, deploying, and managing their agents.

One step made faster

In the absence of a proper process map (see above), the choice of which step to automate is usually done based on which part of the process can be demonstrated the best with AI. In the case of vendor onboarding, it's usually using an LLM to read a W9, extract name, TIN, address, etc and post to the vendor master. This can save around 8 minutes for the person doing that work (in procurement).

Unfortunately, the 8 minutes is immaterial. 22 days is 31,000 minutes, meaning you've shaved 0.03% of the cycle time. In the context of onboarding maybe 1,500 vendors a year, that's 200 hours (or about 10% of one person's time for a whole year). From a business perspective, it's immaterial. The bigger issue is the vendors are still being onboarded in 22 days. So projects are being slowed down because we can't get a vendor onboarded, and the business is just charging it to a corporate card to avoid going through procurement. Saving 8 minutes in a 22-day process still leaves you with a 22-day process.

Tap to enlarge (opens the full-size image)

And nearly 100% of the time is spent in handoffs: sitting in a queue for work to start, sending an email to the vendor, waiting 4 days for a response and calling them back because they didn't respond, and finally calling back treasury to get a question answered but since they're busy you have to come back next week. Process re-engineering needs to happen first, and then you decide where AI goes. When we do this, the process usually goes from 15 steps to 5 steps.. But you have to look at the whole process to do it.

No baseline

Then, about 4 months into your pilot, somebody from finance is going to ask what the ROI is of this pilot. And nobody is going to have any idea, because nobody took the time to measure the process before it started. Nobody will be able to tell them how long it took to onboard a vendor before the pilot. Or how much it cost to onboard a vendor. Or how often vendors were onboarded incorrectly before the pilot. Thus, the team is probably going to be left trying to prove value with a user survey (“users say they feel 30% more productive”) or by showing how many people are logging into the AI app (“it's been adopted by many people!”).

Finance is not going to care.

You need to measure the process before you start building. Cycle time, cost per transaction, error rate, rework rate, etc. It shouldn't take longer than maybe 2-4 weeks, and you can get most of the data you need from the same logs you pulled to map out the process. Furthermore, you need to make sure you are measuring the right thing. Is it cycle time or cost you are looking to reduce? Or is it revenue uplift, after what you spend on tokens? Or is it risk reduction? Agree up front which one, and what number counts as success. In the case of vendor onboarding, a big part of it is reducing risk - payment fraud occurs when duplicate vendors are onboarded or when vendor bank account information is changed.

Built a sidekick

Most pilots are either chatbots or copilots (usually a pane on the right side of the screen). It's valuable for an analyst to be able to ask the copilot if a vendor already exists, for example. But at the end of the day, it's still a sidekick - something an analyst uses to make them more efficient, but contingent on a person to prompt it and use it. Only about 10% of people (the "tinkerers") will actually use a chatbot, and it'll only make them maybe 10-20% more efficient.

You need a background agent - something that does the work without people prompting it. In the case of vendor onboarding, the background agent would take the request off the queue, follow up with the requester for missing information, send an email to the vendor to get the vendor to submit a W9 and a bank letter, verify the bank information matches the vendor name on the W9, run a sanctions check, check for duplicates in the vendor master, create the vendor in SAP, and only send work to people when there are exceptions (e.g. bank account name doesn't match vendor legal entity name, potential sanctions match, requester doesn't want to complete infosec questionnaire). The workflows we take on with background agents see 60-90% gains, or about a 1:10 ratio in enterprise value when compared to sidekicks like chatbots or copilots. There are likely hundreds of chatbots and copilots in your enterprise, but I'd bet you can't think of even one background agent. That's because the background agent is the one that's hard to build.

Tap to enlarge (opens the full-size image)

Outside the systems of record

The pilot is likely in its own application that people have to go to in order to use. In other words, it has its own URL and users have to log in to use it. Are you kidding? Procurement probably uses Coupa or SAP Ariba. The people making the vendor requests probably use ServiceNow and Outlook. The people in treasury probably use the bank's portal and Excel. Ask people to log into a different application and you won't get adoption.

You have to build it on top of your existing tech stack. The agent can read the request in ServiceNow, then create the vendor record in SAP, post the status in the ServiceNow ticket, and send an exception to the analyst in Teams or Outlook. Avoiding the hassle of migrating data to a new system, adding yet another tool that your team has to log into, and retraining your team on how to use the new tool. A positive side effect is that you'll be able to clear it much quicker with infosec, legal, and IT because the agent is running in your cloud on your data. We've seen lots of pilots fail because they didn't realize they'd need to go through compliance until month 3, where the project gets killed on the spot.

No owner

We've seen a lot of agents that were developed in IT, then launched to the business, and put in a quarterly update by the innovation team. Then, when the agent went haywire on a Thursday afternoon at the end of the quarter (most likely due to an IT change), there was no one on call to manage the exceptions, so the process went back to being done manually and the agent never got turned back on.

The agent needs to be owned by the function that is responsible for the process that you're automating, and more specifically the person who is responsible for the metrics that come out of the process. For example, the head of procurement operations may own the vendor onboarding process. You also want someone who is at least a little bit technical (this is the internal FDE role, a forward deployed engineer who sits with the function) so that someone can fix the agent when something goes wrong. And of course, the people who are managing the exceptions queue will ultimately be making the agent better with their feedback.

No plan for month 7

Pilots often get funded through an innovation budget for 6 months. But, then in month 7, who is going to pay for the tokens? Who is even going to own the agent? What happens when the model gets deprecated? What happens when your AI provider raises their prices? What happens when the consultants roll off of the engagement? What happens when the one person in your function that knows how to manage the exceptions queue changes roles? Oh, and if the agent was built using a single AI provider, be prepared for the pilot to turn into a migration project when they deprecate their model or raise their prices.

So, plan for month 7 when you start the pilot. Make sure that the agent has an owner in the function and is funded through the function's operating budget and not the innovation budget. Know how much each token costs so that you know how much each transaction costs. Make sure that the agent is running in your environment, using your data, and that you own all of the process maps, evals, logs, and code. And build it agnostic to the model that you are using so that when a better (or more cost-effective) model comes along (which it will, every few months at this point), you can swap it out quickly and easily. Finally, start with the agent running in shadow mode and make sure that its outputs match the outputs of the people who are doing the process.

Executing the same process properly

To automate vendor onboarding the right way, you should spend 2-4 weeks mapping the process by interviewing the people who do the process and looking at their process logs. In doing so, you'll realize that the process takes 22 days, 50% of the requests come in incomplete, and 1 in 10 vendors are duplicates. You'll want to measure the cycle time, the cost per vendor onboarded, the % of duplicate vendors, and the # of changes to vendor bank details that weren't validated. Then, you'll re-engineer the process, making sure that the intake form is smart enough to prevent incomplete requests, and that treasury only calls back the vendor when there's a mismatch between the bank letter and the W9. Then, you'd build the agent to run in the background as people interact with ServiceNow and SAP. The agent would be owned by the head of procurement operations and it would first run in shadow mode for a few weeks.

Tap to enlarge (opens the full-size image)

We've seen vendor onboarding go from 22 days to 4 or 5 this way, with the analysts only touching the 15% or so of vendors that actually need a judgement call. And because there was a baseline, the CFO can see it, which is what gets you the budget for the next 10 processes.

Why don't more companies do this?

There’s a severe incentives mismatch with most parties involved: vendors selling AI to large companies are incented to sell you something that won't work. Software companies want to sell you seats, so they'll sell you sidekicks and measure success by # of logins; big consultancies want to sell strategy and a roadmap of pilots, so 6 months later you'll have a 300 page slide deck, but no solution in production; internal teams want to show they're 'doing something' with AI, so you end up with lots of pilots that are small enough to launch in a quarter and easy to demo to leadership (e.g. one step of a process).

The reality is that all the parts we identified above (e.g. mapping the process; figuring out exception volume; establishing a baseline; integrating with SAP/ServiceNow/etc; ensuring there's an internal owner; figuring out who's paying for the project in month 7) are slow, unglamorous, and hard to demo. So does this mean all the AI pilots your company has run are a waste? Not at all. You have an internal team that is getting comfortable with the tools and you likely have a list of processes that would benefit from AI. You just have to plan your next pilot a bit differently.

Work with us

This is exactly what we do at Varick. We work with companies between $1b+ in revenue to automate processes in departments like Finance, Sales, HR, Procurement, Operations, and IT. We start by helping you map a process in your department. We then re-engineer the process and work with your team to build background agents (that use your existing tech stack) to automate the process in a way that your team can own in month 7. 100% of our clients have taken their project to production and 100% have engaged us on project #2. If your company has run AI pilots that you were hoping to take to your P&L, you should visit our website.

TLDR

Most AI pilots fail for one (or more) of 7 reasons, and it has nothing to do with the underlying AI model. They're based on a standard operating procedure (SOP), not a process map, so they can't handle exceptions, which are often 40%+ of process volume.

They only speed up one step in a process (e.g. they speed up W-9 reading in vendor onboarding, but vendor onboarding still takes 22 days). And since no one established a baseline, it's hard to quantify the value of the AI solution in month 4.

They build sidekicks (which are typically used by ~10% of users and result in 10-20% improvements) instead of background agents (where we've seen 60-90% improvements). And they require a separate app and login outside of existing platforms like SAP, ServiceNow, and Outlook.

There's no owner on the business side, so when the solution breaks, it stays broken. And no one thought about who is paying for the project in month 7 (to cover token costs, deal with model deprecations, replace consultants, etc).

When all 7 are solved, we've seen vendor onboarding go from 22 days to 4-5 days, with humans only touching ~15% of vendors to make judgement calls.