AI agents for customer service: an honest assessment

An AI agent can handle part of your customer service workload when requests have clear boundaries, information is reliable, and actions are controlled. Published results show potential gains, but they do not prove that autonomous support can replace an entire team. What matters is whether the customer’s problem is actually resolved, at a fully accounted cost, with human support available when needed.

Define the scope: answering, assisting, or taking action

An assistant that explains your return policy does a different job from a system authorized to refund an order. In the first case, you mainly assess whether the answer is accurate. In the second, you also need to verify the customer’s identity, the applicable commercial terms, and the transaction recorded in your system.

AI agent or chatbot: what’s the difference? explains this distinction. Start with the tasks you want to delegate rather than the terminology used in a sales pitch.

Criterion	Assisting a representative	Answering autonomously	Acting autonomously
Expected result	Suggested response reviewed before sending	Response sent to the customer	Change to a case or transaction
Information required	Documentation and ticket context	Reliable sources and explicit boundaries	Sources, business data, and permissions
Main control	Human review	Answer checks and handoff	Technical checks before execution
Failure to watch for	Misleading suggestion accepted	Incorrect information or a dead end	Incorrect, repeated, or unauthorized action

Document retrieval, often called RAG, can supply relevant passages to the model. It does not guarantee that those passages are current or correctly interpreted.

Our AI tools directory, organized by job function, can help you identify available products. Explore those options after defining the scope: a convincing FAQ demonstration does not establish that a system is ready to issue refunds.

Documented case: what Klarna announced

In a press release dated February 27, 2024, Klarna said its assistant had handled 2.3 million conversations during its first month, accounting for two-thirds of its customer service chats. The company also reported a 25% reduction in repeat inquiries and a drop in resolution time from eleven minutes to under two minutes. These figures come from Klarna’s press release.

This is an example of deployment at scale, but the findings are company claims, not an independent evaluation. They describe a specific period in 2024; they do not measure Klarna’s service in October 2026.

The useful lesson is how to interpret the figures. The number of conversations handled tells you about usage; it does not establish that every answer was correct. Short resolution times need to be considered alongside reopened cases, complaints, and satisfaction.

Before applying this case to your own business, ask which requests count toward the results and which are transferred to human support. Also check whether the satisfaction measure covers all customers or only those who complete a survey. These details determine what the results mean: without them, the headline percentage cannot predict your own performance.

Research findings: assisting representatives delivers measured gains

A study published on February 4, 2025, in the Quarterly Journal of Economics examines the introduction of a generative AI assistant for 5,172 customer support representatives. The authors measured an average increase of 15% in problems resolved per hour, with substantial differences depending on employee experience. Less experienced representatives benefited more from the assistance; the most experienced saw small speed gains and slight declines in quality. The source is the paper Generative AI at Work.

This finding concerns people working with assistance. It does not measure the replacement of representatives by an autonomous agent or an equivalent reduction in company costs.

For a small or midsize business, it suggests an approach worth evaluating: provide representatives with source-based draft answers before sending those answers directly to customers. Human review can reveal missing documentation, ambiguous wording, and business rules that are difficult to apply.

This setup has costs of its own. A representative who must check every detail may spend all the time saved on drafting. Measure the complete handling process, including corrections, and break down the results by experience level.

The findings from these cases should remain separate: Klarna reports automated handling, while the academic study measures assistance at work. Combining their results would give a misleading picture of what autonomous systems can achieve.

A reported resolution does not establish success

The contractual definition of a resolution deserves as much attention as its price. Intercom’s official pricing page, accessed on October 24, 2026, lists a price for Fin of $0.99 per billable outcome. It explains that an outcome can include customer confirmation, no request for further help after an answer, or completion of a procedure, including one involving a handoff.

These situations do not necessarily have equal value for your business. Silence may follow a satisfactory answer, but silence alone does not prove that the problem has been solved.

Your dashboard should distinguish between:

• resolutions confirmed by the customer;
• conversations counted as resolved without confirmation;
• procedures completed with a human taking over;
• requests reopened or restated through another channel.

Add checks against your business records: was the refund recorded, was the appointment changed, and did the response follow the applicable rule? Also review a sample of conversations classified as successful. Looking only at visible failures allows unnoticed errors to slip through.

Sierra’s research environments, presented in the official τ-bench repository, bring policies, tools, and tasks together in customer service simulations. They offer a useful approach to evaluation, but cannot replace testing on your own cases.

Calculate the cost of a correctly resolved case

The vendor’s bill is only part of the budget. Add documentation preparation, connections to business software, supervision, and the human time spent taking over cases.

Use this calculation:

Cost per correctly resolved case = total system spending and associated human handling costs, divided by the number of cases correctly completed.

Also set a period for tracking reopened cases. Otherwise, a request closed quickly and picked up again later may appear to save money even though it creates additional work.

Your budget should distinguish between:

• startup costs: documentation cleanup, integration, testing, and training;
• recurring expenses: subscriptions, usage, maintenance, and quality control;
• failure costs: corrections, goodwill compensation, and complaint handling.

Our article on the real cost of an AI agent breaks down these expenses.

Time freed up does not automatically become a financial saving. It may help representatives respond more readily or absorb an increase in requests. And if the agent handles simple cases, difficult cases will make up a larger share of the human workload: average handling time may rise even when representatives are doing their jobs just as well. Assess the service as a whole.

Limit permissions and preserve human handoffs

A customer service agent receives messages and attachments whose contents you do not control. A prompt injection may try to turn that content into instructions. Permissions therefore need to be enforced by software: a written instruction to the model is not enough to protect a sensitive operation.

The principle of least privilege means restricting each tool to the case and action required. For financial operations, include identity verification, eligibility checks, and protection against repeated execution. When uncertainty arises, the agent must be able to pause the action and transfer the case.

Privacy also applies to stored conversations. In its chatbot guidance dated February 19, 2021, France’s data protection authority, CNIL, outlines issues involving personal data and retention. This source establishes principles; it does not certify any current product. AI agents and GDPR: what data leaves, and where does it go? helps you examine those data flows.

Finally, a human handoff must preserve useful context: the original request, checks already performed, sources consulted, and actions attempted. A transfer that forces the customer to repeat everything shifts the burden instead of resolving the problem.

Make the decision through a limited pilot

Start with a category of requests governed by written rules, where errors can be corrected. Replay historical cases, then run the agent without letting it send responses autonomously so you can compare its suggestions with the expected decisions.

Build tests around ordinary requests, but also include missing information, conflicting rules, unavailable tools, and attempts to bypass restrictions. The guide to evaluating an AI agent explains how to organize these checks.

Before opening the system to customers, define the conditions that will trigger a shutdown: an unauthorized action, data disclosure, repeated business errors, or an increase in complaints. Assign someone responsibility for suspending the system and prepare a return to human handling.

Expand the scope according to observed results. The guide to AI agents in business places this pilot within a broader deployment process.

A customer service AI agent can make sense when you know exactly which requests to assign it and how to verify the results. If your rules remain unwritten or nobody can take over cases, start by improving the process and assisting your representatives. Decide on autonomy afterward, one task at a time.

Frequently asked questions

Can an AI agent replace an entire customer service team?

The public results discussed here do not demonstrate that a company can eliminate all human support without reducing service quality. Before considering a reduction in staffing capacity, also measure the workload of the cases left to human representatives and the frequency of repeat contacts.

How can I tell whether my documentation is ready for an AI agent?

Ask a support representative to handle recent requests using only the documents available to the agent. If they need to ask a colleague or rely on unwritten rules, address those gaps before giving customers access.

Should I automate phone support from the start?

You can begin with a written channel to make conversations easier to review and errors easier to analyze. Phone support requires additional evaluation of interruptions, audio comprehension, and transfers to a human representative.

What should I ask to see in a sales demo?

Bring difficult requests from your own business and ask to see the actions performed, the sources used, and the handoffs. Also ask exactly what you will be billed for and how the system behaves when a business tool becomes unavailable.