At last month’s Enterprise Connect conference in Orlando the AI hype train was steadily moving along the tracks. Virtually every keynote and vendor announcement involved some new AI-based feature to improve customer experience, employee collaboration, security, operational management, and more. AI, specifically generative AI (Gen AI), has received the lion’s share of media attention over the last year as vendors race against each other to differentiate and to accelerate in their use of AI to advance their products. However, the rapid, and accelerating advance of AI, has created challenges for IT and business leaders as they move from hype to adoption. These include measuring ROI, change management, and ensuring security, governance, and compliance.
Vendors tout their AI features as improving productivity and customer service, translating into reduced operating costs, increased revenues, and improved workplace efficiency. Some of these benefits are easy to measure. For example, Metrigy’sAI for Business Success: 2024-25 global research study of nearly 700 organizations found that AI was generating tangible benefits, especially for customer service. Study participants, on average, were able to use AI to resolve 41% of customer interactions without live agent support.
AI allowed companies to hire fewer agents compared to those who did not use AI for customer engagement (as shown in the accompanying chart). The reduction in staffing provides quantifiable ROI for AI adoption. AI also saved, on average, 35% of the time spent documenting customer interactions.
On the employee engagement side, AI offers the potential to reduce time spent after meetings by automating action item capture and assignment. It also offers potential to accelerate sales and marketing activities by improving the generation of marketing campaign materials. But simply saving time may not resonate with IT and business leaders who must translate how saving time translates into measurable business benefit. For example, if an AI tool saves the average employee two hours per week, then how is that additional time being used? If I use my two hours to check my NCAA basketball bracket (which was a disaster) and watch game highlights, then did the company achieve a measurable benefit for its investment in AI tools?
To realize ROI, IT and business leaders must use a variety of quantitative and qualitative approaches. These may include looking at average time spent on repeatable activities (e.g., on-boarding customers, conducting a sprint, or generating a marketing campaign) to see if AI saved any time. They can also look at additional metrics such as customer satisfaction, retention, and up-sell, as well as campaign performance to identify additional benefits. Finally, IT and business leaders should use employee experience surveys to understand if employees see benefits from AI. Through surveys, managers will often learn of use cases, and benefits, that they have not previously considered. They will also learn of challenges they may not have anticipated.
Unfortunately, there’s no magic formula for identifying ROI. Examples will vary based on specific industry activities. But vendors, and their implementation partners, are becoming increasingly aware of the need to translate features into benefits. At the recent Google Cloud Next event in Las Vegas there certainly was plenty of discussion about advances in AI technology, but I was heartened to see that a lot of the discussion also focused on implementation and measurement of business value. I expect identifying the business value of AI to be the dominant point of discussion in 2024 (and likely beyond!)
Another key point for me, coming out of both Enterprise Connect and Google Cloud Next, is that change management approaches are getting overwhelmed by the rapid advance of AI. Cloud has already accelerated the rollout of new features from months to weeks to days. AI is accelerating the speed of new feature deployment exponentially.
Top-down approaches of IT evaluating, evangelizing, educating, implementing, and training users on new features no longer works. Rather, business leaders and individual employees proactively seek to understand how AI impacts their specific roles. AI isn’t just for collaboration and customer experience, rather AI features are coming fast and furious across all roles including sales, field, marketing, HR, business operations, and more.
Companies must create the capability for individual lines of business to quickly evaluate potential use cases of AI and obtain necessary IT support to aid in deployment and management. Depending on company size, this could entail the creation of a ‘Chief AI Officer’ role, with liaisons to all business units, to both gain requirements and evangelize potential solutions, across all business activities.
Several sessions at Enterprise Connect discussed concerns related to AI accuracy including hallucinations, transcript accuracy, and grounding. Grounding is becoming an increasing discussion point as those delivering AI solutions look to address user concerns about trust. Of course there was much discussion of AI risks, such as Air Canada’s chatbot recently creating a new refund policy.
As companies deploy AI-enabled tools, they must ensure that the responses being used internally as well as externally are accurate. Here I expect to continue to see more development of grounding approaches (such as what Google introduced at Cloud Next) that add source data to AI responses, enabling users to verify the origin, and accuracy, of AI generated content.
Companies must also take a proactive approach to managing compliance for AI-generated content such as meeting transcripts and summaries, images, and more. AI generated content must be classified, archived, and made available in the same way as any other content to ensure compliance with appropriate regulatory requirements. And, IT leaders must work with their vendors to protect their language models against risk of poisoning (either accidental or deliberate) to minimize risk.
We’re entering a transition phase with respect to AI and generative AI. While the potential benefits of AI are clear, and growing, vendors and their implementation partners must work with business and IT leaders to translate that potential into real-world, measurable ROI including productivity gains, revenue increases, and operational cost savings. This must happen while ensuring that employees are able to take advantage of new capabilities without becoming overwhelmed, and that those AI deployments happen in accordance with governance, security, and compliance requirements.
—
About Metrigy: Metrigy is an innovative research and advisory firm focusing on the rapidly changing areas of workplace collaboration, digital workplace, digital transformation, customer experience and employee experience—along with several related technologies. Metrigy delivers strategic guidance and informative content, backed by primary research metrics and analysis, for technology providers and enterprise organizations.
