After questions of AI ROI have emerged, enterprises should look at private hybrid AI deployments to boost productivity and services—in economical fashion.
If you look at your technology stock screen lately, you might have recognized that we have exited the first stage of the AI hype cycle. A recent correction of 10% (or more) in the largest technology stocks, including Amazon, Google, and Microsoft, has everybody questioning expectations for an AI boom that moved too far, too fast.
But it’s still early days. The AI buildout will be a long journey, with many twists and turn. The largest hyperscale cloud providers deployed billions in new infrastructure to support the creation of large language models to fuel new applications and services, and the market got too hot. The next phase of AI will be characterized by a more practical focus on return on investment, cost, and privacy.
What I expect to unfold next is the evolution of private enterprise AI, a targeted buildout by enterprises to use hybrid cloud AI deployments to boost productivity and services—in economical fashion. After all, while Microsoft and Nvidia may rule the world of the stock market and the S&P 500 Index, they are but suppliers to the rest of the world, which must turn their technology into profits.
We’re not here to say that LLMs and Big AI are a waste—they are part of the market’s evolution. In recent talks with practitioners, I’m finding a lot of interest in experimentation with a range of AI models combined with private infrastructure. There is also a surge in interest in small language models as an economical way to build targeted productivity in enterprises with private data.
The AI cost meme was catalyzed in June of this year, when Sequoia Capital partner David Cahn raised this question in an article entitled “AI’s $600 Billion Question,” asking where the revenue will come from to cover the estimated $600 billion cost of AI.
Cahn made some salient points. While he remains long-term optimistic about AI, using the railroad analogy (if you build it, the trains will come), he points out the huge carrying costs of this infrastructure. And we don’t yet know which services and business models will pay for the infrastructure. The returns will come, but capitalism can be a sloppy process.
So why should we look at private AI? In my talks with CIOs and technology leaders, it’s clear they have reservations about mass adoption of Big AI, due to questions about cost, safety, and economics. But there are ways to make AI work for the enterprise: Lower the cost of the infrastructure, feed it with your own private data, and build proof points for productivity and revenue gains.
Private AI can be deployed in smaller scale in flexible, hybrid cloud infrastructure either on premises or in the cloud, enabling enterprises to experiment with AI without making back-breaking bets.
In one interesting example, networking infrastructure provider Hedgehog explained in a recent Tech Field Day how computer-vision company Luminar deployed AI in its own private infrastructure at one-sixth of the cost of using a cloud AI service. It built this infrastructure to feed its computer vision models with only 20 Nvidia L40 graphics processing units.
Luminar, like most companies, was looking for a productivity payoff that didn’t require huge risks. Many enterprises are confirming such wins, but they are doing it by experimenting with AI in a targeted fashion, using their own data and infrastructure.
There are tactical proof points showing better returns for enterprises with the use of private AI. These use cases can come in many forms, via custom apps built with partners on private infrastructure and SLMs, or through commercial AI services fine-tuned with private data using Retrieval-Augmented Generation.
Audi has produced a great example of this, explaining how it refined a customer service chatbot using its own private data and RAG to reduce hallucinations and provide more relevant information. Audi partnered with software provider Storm Reply to build the chatbot. Training it with Audi’s data gave the chatbot the advantages of better security and more accuracy.
This doesn’t mean that hyperscaler services such as OpenAI’s ChatGPT or Microsoft’s GitHub Copilot will be shunned—but they often need to be improved by combining them with private data, using RAG and other techniques. And enterprises may have their own price points in mind for implementing these services.
In another example, I recently talked about private AI with Faizan Mustafa, a former Toyota CIO who is now vice president of AI and enterprise IT with cloud networking provider Aviatrix. Mustafa told me how Aviatrix is using AI internally to drive productivity, based on its own private data.
Some of the AI applications Aviatrix is targeting include customer relationship management and customer support. Mustafa says the company is using its own data from customer interactions to identify pain points or ways to improve its products.
Mustafa also sees the benefit in helping developers. Aviatrix is using Microsoft’s GitHub Copilot to help developers more quickly generate documentation. “If they don’t have to write long documentation, it helps augment the capacity of software developers. We’ve seen real value,” he said.
Examples like that given by Aviatrix are happening across the world in private experiments. Many of them involve collecting and processing private enterprise data with the applications of RAG.
What will drive the private AI revolution forward? In short: cost reduction. Technology democratization comes through economics. Many enterprises see the scale and size of hyperscaler deployments costing tens of billions of dollars, and they think, we just can’t afford that. They might also find mass-market LLMs too general. They need more targeted AI. As in the Luminar example, they can buy a few dozen AI chips and experiment with their own data on a much smaller scale. They don’t need all the data in the world.
The costs of building private AI clouds could be substantially lower than the use of public cloud models. Consultancy Future Processing estimates the costs of building an AI model for an enterprise can range from $5,000 to $500,000, depending on the complexity. These numbers are a bit lower than what the hyperscalers spend on AI infrastructure. For example, Meta is estimated to have spent $740 million to build its Llama 3 AI model. It’s also spending tens of billions on new AI chips and infrastructure.
For these reasons, we expect interest and experimentation in private AI to explode. A bunch of startups, in addition to Storm Reply, are targeting this area.
Mark McQuade, a former Hugging Face engineer, co-founded a company called Arcee AI, which is targeting SLMs for enterprises. McQuade, who is Arcee’s CEO, told me that SLMs will help drive down the cost of AI and democratize the technology for enterprises.
“We heard customers say they want to own their own data, McQuade told me. “If you can get software and Gen AI that runs in private cloud, that’s the holy grail.”
McQuade says that Arcee has successfully built AI models that run on a single Nvidia A100 GPU. The company has a flexible subscription business model that allows customers to use the software either on-premises or as-a-service in the cloud. The company has 25 employees and has raised $24 million in venture capital.
So, we should stop fretting about whether AI will be “up or down” and think more about where it’s going next. It’s a process of applying the technology to discover ROI at the most economical rate. To get to the next level, AI will need to be four things: secure, private, accurate, and cost-effective. Enterprises are all working on that, and the returns have a high ceiling.
Original Source: https://www.forbes.com/sites/rscottraynovich/2024/08/08/after-the-big-ai-correction-private-enterprise-ai-may-blossom/
