The client
A global cloud software company with about 1,400 engineers in 60 product teams. Code lives in GitHub, work is tracked in Jira, and the teams work in Slack.
The engagement
The chief technology officer sponsored the engagement, and the head of the developer platform team owned it. Four workflows were in scope:
- Failed builds: about 38,000 builds a week, 14% of them failing.
- Code review: about 6,500 pull requests a week.
- Backlog triage: about 4,800 new bugs and requests a month.
- Security upgrades: about 1,300 open findings, 140 of them critical or high.
What the pilot found
Two forward-deployed engineers ran a 4-week pilot with the developer platform team. They studied a year of builds, pull requests, and tickets, and read the team's guidelines and policies. They also interviewed engineers, team leads, product managers, and the security team.
- Failed builds were rerun before anyone read them. 31% of failures were flaky and passed on a rerun. A real failure reached the engineer who caused it 3.5 hours after it broke.
- Reviews started with detective work. A pull request waited 18 hours for its first review, and 44% linked no ticket.
- The backlog hid what mattered. 40% of tickets were untriaged after a week, and critical security findings stayed open for 90 days.
The dashboard counted a failed build as the 9 minutes a rerun takes, or 700 hours a week. Counting reruns, lost focus, and blocked merges, a failed build cost about 26 minutes. That is 2,300 hours a week, more than 3 times the dashboard's figure.
The redesign
Failed builds were the focus. The process now runs on 3 agents with 1 decision left to people. When a build fails, an agent reruns only the failed step and checks each failing test against its own history to spot flaky ones. A real failure goes straight to the engineer whose change caused it, with the error and the change attached. A flaky test goes to its team as a ticket, and the team lead decides what to do with it.
Before11 steps
3.5 hours
After4 steps, 1 human decision
9 minutes
- Manual step
- Agent step
- Person decides
Process steps
- Before
- 11
- After
- 4
- Change
- 64% fewer
Red build to owner notified, median
- Before
- 3.5 hours
- After
- 9 minutes
- Change
- 96% shorter
Manual reruns per red build
- Before
- 2.1
- After
- 0.2
- Change
- 90% fewer
Engineer time per red build
- Before
- 26 minutes
- After
- 5 minutes
- Change
- 81% less
Hours lost to red builds a week
- Before
- 2,300
- After
- 420
- Change
- 82% fewer
Working hours with main branch red
- Before
- 11%
- After
- 2%
- Change
- -9 pts
| Before | After | Change | |
|---|---|---|---|
| Process steps | 11 | 4 | 64% fewer |
| Red build to owner notified, median | 3.5 hours | 9 minutes | 96% shorter |
| Manual reruns per red build | 2.1 | 0.2 | 90% fewer |
| Engineer time per red build | 26 minutes | 5 minutes | 81% less |
| Hours lost to red builds a week | 2,300 | 420 | 82% fewer |
| Working hours with main branch red | 11% | 2% | -9 pts |
4 weeks of discovery, then 4 weeks of deployment
- Weeks 1 to 4, discovery: the pilot above maps the process, interviews the team, and sets the baseline for every figure on this page.
- Week 5, build: the agents are built on the redesigned process and the pilot's prototypes.
- Week 6, shadow: the agents work live builds with read-only access. Nothing is posted, and their findings are compared with the engineers'.
- Week 7, validate: the agents are tested on 3 months of past failed builds and pull requests, and the platform team and 3 team leads sign off.
- Week 8, deploy: live in GitHub, Jenkins, Jira, and Slack, with every action logged.
- From week 8, train: the engineers stay on with the teams whose work changed, and tune the agents.
What the agent does
- When a build fails on the main branch, the agent reruns the failed tests once.
- A test that fails at random on unrelated changes is marked flaky, and its team gets a ticket.
- A test that fails for real is traced to the change that broke it.
- Within minutes the agent messages the author in Slack, with the error and the change attached. People decide every fix.
Results
Time to first review fell from 18 hours to 4, because every pull request now arrives with a summary, its ticket, and the routine checks done. Untriaged tickets after a week fell from 40% to 5%. Critical security findings now close in 12 days instead of 90. Reverts and hotfixes fell from 140 a quarter to 60. Of the 164,000 hours released, 90,200 are in failed builds, 41,600 in review, 21,600 in triage, and 10,600 in upgrades.
| Source | Annual value | How |
|---|---|---|
| Capacity released | $15.1M | 164,000 hours across the 4 workflows, at the function's $92 fully loaded rate |
| Rework removed | $2.0M | Reverts and hotfixes down from 140 to 60 a quarter, at about $6,250 each |
| Pipeline spend recovered | $1.3M | Manual reruns down 90%, on the $1.45M a year of pipeline compute that went to reruns |
| Total | $18.4M | Across the 4 workflows in scope |
What stays with people
Agents open tickets, comments, and pull requests. They never merge code or release it. Every merge, every release, every test set aside, every upgrade, every ticket's priority, and the roadmap stay with the engineer, team lead, or product manager who owns them. Every action is logged with the rule it applied.
