PensionBee Study: 1 in 10 AI Pension Answers Risk Financial Harm to Savers

Research from PensionBee reveals that one in ten pension-related answers provided by major AI chatbots could lead savers to financial loss or irreversible errors. As fintech firms increasingly integrate generative AI into customer service, these findings highlight a critical "advice gap" where technical accuracy fails to account for complex regulatory requirements and consumer vulnerability.

What was announced

PensionBee conducted a human-run test of four prominent AI models—Microsoft’s Copilot, OpenAI’s ChatGPT, Google’s Gemini, and Anthropic’s Claude—to evaluate their reliability in providing pension information. The study, described as the first of its kind in the UK, involved asking 45 questions across nine specific pension topics. Human testers used fresh, free-tier accounts to minimize bias, with two experts scoring each of the 539 total responses for both accuracy and potential harm.

While 89% of the answers scored two or more out of three for accuracy, 11% (57 answers) were flagged as potentially harmful. The research found that performance varied significantly depending on the complexity of the query. Questions regarding basic contributions and withdrawals saw accuracy rates above 96%, with harm rates at 5% or lower. However, queries involving significant life events or ambiguous locations resulted in lower accuracy and higher risk.

The data revealed that potentially harmful answers outnumbered inaccurate answers by roughly three to one on topics signaling a saver in difficulty. For example, questions about taking money out of a pension and identifying scams did not produce any answers marked as outright wrong, yet they generated 12 potential harm flags between them. The primary issue identified was the omission of critical regulatory context. Some chatbots failed to mention that transferring a defined benefit pension valued at over £30,000 legally requires regulated advice. Others neglected to note that from April 2027, most unused pension funds will be included in estates for inheritance tax purposes, potentially leading to unexpected tax burdens for beneficiaries who take the AI's answer at face value.

"Using AI chatbots for pension advice can be a bit like playing Russian roulette with your retirement planning. While for the most part it gets things technically right; the confident, helpful tone of answers occasionally masks some worrying omissions, it may fail to detect vulnerability, or just straight up get things wrong."

Becky O'Connor, Head of Pensions at PensionBee.

The companies involved

PensionBee is a leading UK-based online pension provider that allows consumers to combine their various pension pots into a single online plan. Since its inception, the company has positioned itself as a disruptor in the traditional retirement savings market, focusing on transparency and ease of use through its digital platform. It manages billions in assets for hundreds of thousands of customers, advocating for greater consumer control over retirement data and the simplification of the pension transfer process. The firm operates in a highly regulated environment where the distinction between "guidance" and "regulated advice" is a central legal boundary.

The research was supported by AI for Media, an organization led by founder Harriet Meyer. This collaboration highlights the intersection between financial services and the rapidly evolving field of generative artificial intelligence. PensionBee has been a frequent subject of industry analysis, with 112 stories appearing in FF News archives, reflecting its active role in commenting on UK pension policy and consumer protection. As the market moves toward more automated solutions, the role of established providers in defining the safety parameters of these tools becomes increasingly central to maintaining consumer trust in digital-first financial services.

What FF News has reported before

FF News has extensively tracked PensionBee’s efforts to highlight systemic risks and regulatory shifts within the UK retirement sector. Recently, the publication covered how PensionBee Warns of £30,000 Retirement Penalty Amid UK Triple Lock and Social Care Review, emphasizing the financial pressures facing modern retirees. The company has also been vocal about the impact of government policy on individual savings, as seen in the report PensionBee Warns of 'Fiscal Drag' as Savers Demand Personal Allowance Unfreeze in 2026 Budget. Additionally, FF News detailed the firm’s push for technological readiness in the broader industry, reporting that PensionBee Warns Savers to Act Before Pensions Dashboards Connection Deadline. These reports underscore a consistent pattern of the firm identifying gaps between consumer expectations and regulatory reality.

What this means

This research places the fintech industry at a crossroads regarding the deployment of Large Language Models (LLMs). While the efficiency gains of AI are undeniable, the 11% "harm rate" identified by PensionBee suggests that the technology is not yet a viable substitute for regulated advice in high-stakes financial sectors. The industry now faces a significant liability question: if a chatbot provides "broadly accurate" but contextually dangerous information, who is responsible for the resulting financial loss? As regulators scrutinize the "advice gap," firms under pressure to automate must decide whether the reputational risk of a "false friend" AI outweighs the operational savings. The move toward April 2027 tax changes will be a major test of whether these models can update their knowledge bases fast enough to remain safe for the general public.