Data, training complexities make DIY AI agents difficult

AI agents can automate CX processes in customer service, marketing, e-commerce and sales. But for companies doing it themselves, standing up agents to perform that automation is complicated.

That was the upshot of a report released yesterday from Valoir, an independent tech research firm based in Arlington, Va. The report compared do-it-yourself (DIY) routes for deploying AI agents for CX processes with Salesforce Agentforce, which provides built-in tools for simplifying the setup of security and retrieval-augmented generation. RAG enables a generative AI agent to crawl external sources of data to add knowledge to its answers.

The more than 20 companies interviewed for the report said that RAG was challenging to build independently, but it took just a few hours in a Salesforce development sandbox. Without RAG, ChatGPT, for example, could deliver accurate answers to questions only 40% to 60% of the time, said Valoir founder Rebecca Wettemann.

Agentforce, which is powered mostly by OpenAI models and soon will include Google Gemini, could hit 85% to 95% accuracy. Test tasks included an employee HR leave request, which was performed with 95% accuracy; a customer service bot handling more than 1,000 products, 85%; or a customer self-service agent with authentication and personalization based on protected information, 80%; and complex sales coaching using conversational insights for improvement and suggestions and role-playing, 90%.

Fisher & Paykel, a refrigerator manufacturer, gave up on building an agent in Microsoft Copilot because it couldn't get the accuracy it needed. The company trained data models on its catalogs and got its agent to deliver answers in the ballpark -- except for one thing: The agent couldn't discern which of the 300 refrigerators the querent was asking about.

"If you asked the agent, 'How do I replace the air filter on my refrigerator?' no matter how many steps they put into it, they could never get the agent to realize that there were 300 different [ones] and that it needed to determine which refrigerator the customer had before it could tell them," Wettemann said.

"What it would do was go out, look at all the data, sort of munge together this answer from all the different user manuals and give the customer basically gobbledygook as an answer -- with complete confidence," Wettemann added.

Furthermore, security settings could be built to ensure protected customer data was handled compliantly, but one user said it would take "20 or 30 times the effort" over using prebuilt bot platforms such as Agentforce. Wettemann said Valoir's research found that setting up agents in Agentforce was about 16 times faster than the DIY alternative.

While Agentforce has shown promising results, the report stated that customers are still discovering the tuning and maintenance overhead because the product is new.

Customers said that managed packages for AI might be the only way to develop agents for both internal and customer-facing automations for two reasons: One, agent testing and upkeep appears to be "colossal," with DIY, as one user put it; and two, DIY agents sometimes never end up working accurately enough to use, so the projects were abandoned.

Don Fluckinger is a senior news writer for Informa TechTarget. He covers customer experience, digital experience management and end-user computing. Got a tip? Email him.