Agentic AI Has a Measurement Problem. One Subscription App Counted Only What Finished.

Agentic AI Has a Measurement Problem. One Subscription App Counted Only What Finished.

Every consumer fintech deck in the last eighteen months has an agentic AI slide. Very few of them have a completion rate.

This is not an accident of modesty. Agents that act on a user's behalf in the real world — filling forms, navigating flows, negotiating with a counterparty who does not want to be negotiated with — fail in ways that chatbots do not. A model that produces a fluent answer can be judged on the answer. An agent sent to cancel a gym membership either canceled the gym membership or did not, and the gym is not obliged to tell you which.

So the sector has quietly settled on the metric that flatters it: attempts. Requests submitted. Tasks initiated. Cases opened.

Subpilot, a consumer subscription management app, has this month put a different number on the table. It has canceled 102,099 subscriptions. Each one confirmed complete. Pending requests, in-progress cases, and unconfirmed outcomes are excluded from the total.

For an industry currently building agentic execution layers on top of open banking rails, the interesting artifact here is not the number. It is the counting rule.

The stack: three tiers, one confirmation

Cancellation is a deceptively hostile problem domain because the counterparty is adversarial by design. Retention flows exist to produce abandonment. Multi-step confirmations, "are you sure?" interstitials, and offers calibrated to the moment of hesitation are not defects in the user journey; they are the user journey, optimised against the outcome the user came for.

Subpilot's response is a tiered execution model:

AI agents navigate web cancellation flows directly, working through confirmation loops and retention screens.

Automated email processes handle services that accept written cancellation requests — a lower-variance channel, and a reminder that a substantial share of the subscription economy still terminates by correspondence.

Human specialists take the residual: services that require a phone call during business hours, cases where the flow has changed underneath the agent, negotiations that need a person.

That third tier is what makes the confirmed-only metric defensible. An agentic system without a human fallback has to either report attempts or report a completion rate that includes silent failures. A system with one can afford to count only what finished, because something finishes almost everything.

Coverage today: automated cancellation across hundreds of services, with published step-by-step guides for more than 500 more.

The consent and data model

The architecture is worth noting for anyone building adjacent to it.

Bank connections are read-only. Subpilot ingests recurring charges, invoices and receipts to construct a map of a household's subscription spending — including active free trials before their first conversion — and cannot move money, execute payments, or modify a user's accounts. This is a technical boundary, not a policy commitment, which is the distinction that matters when a user is deciding whether to connect an account.

Execution is request-scoped. No cancellation, refund claim, or negotiation is initiated without an explicit user action, and each terminates in a confirmation surfaced in the dashboard. The agent has no standing authority.

For a category whose central trust problem is what else does it do with my bank connection, collapsing the answer to "read, and only when you ask" is a defensible product decision before it is a compliance one.

Refunds and negotiation: the second surface

Cancellation is the wedge. The economics sit next to it.

Refunds recovered during the cancellation process have returned $504,597 to users, across 16,626 successful requests. Refund recovery is a harder agentic problem than cancellation — it requires persuading a counterparty to reverse a completed transaction — and the gap between requests made and requests succeeded is where the real capability curve lives.

Bill negotiation is the third surface: 4,451 negotiations have produced $95,760 in recurring monthly savings, applied to live bills rather than modelled over a projected lifetime, with cumulative savings from those discounts on pace to exceed $1 million. Subpilot identifies which recurring bills are negotiable, makes the call, and passes the discount through.

More than 60,000 users are on the platform.

"This milestone tells us something important: people are done accepting the status quo on subscriptions."

- Subpilot

Why the counting rule is the story

Consumer trust in agentic finance will not be won on capability demos. It will be won, or lost, on whether the numbers a company reports correspond to things that actually happened to a customer.

Excluding pending and unconfirmed cases from a milestone announcement costs a company the bigger number. It is also the only version of that number a user connecting a bank account has any reason to believe. As agentic products move from summarising a user's financial life to acting inside it, that trade — smaller number, verifiable number — stops being a communications preference and becomes the licence to operate.

The rest of the sector will get there. The question is whether it arrives before or after the first agentic fintech has to explain what "initiated" meant.

Subpilot is a subscription management app. It detects forgotten subscriptions and expiring trials, cancels services users no longer want, secures refunds, and negotiates lower bills. More at subpilot.tech.