What Microsoft Insiders Really Think About the Company’s AI Future

There’s real concern over whether new Microsoft AI services will create enough value for corporate customers that are paying extra for this nascent technology. In-house AI projects have been killed or questioned, teams have been re-shuffled, while the OpenAI partnership sucks up resources and breeds resentment among some staffers.

However, there’s also a sense of optimism about Microsoft’s ability to steadily improve its AI offerings and capitalize on the trust it has built with thousands of businesses that have used its software for decades. Generative AI might be at the cutting edge, but Microsoft could win precisely because its an old, somewhat boring company, not a volatile new startup.

“Customers trust Microsoft more than OpenAI since they already buy Microsoft’s ecosystem,” a Microsoft AI researcher told BI. “It’s more natural for the enterprise to use Microsoft than offerings from OpenAI. All of these features are not as groundbreaking as ChatGPT when it first came out, but it’s bringing a lot of value to people in the productivity space.”

Microsoft’s opportunity

Consumer-facing chatbots are flashy and fun, but the real money will be made if businesses not only embrace large language models and generative AI, but find it useful for improving operations, sales, and other processes.

The first part is beginning to happen, according to a recent Morgan Stanley survey of chief information officers, the executives who decide how billions of dollars are spent on technology every year. The bank found that AI and machine learning were the top priority for CIOs in the final quarter of 2023. More than two-thirds of these execs said AI had already impacted their IT budgets.

With such a huge opportunity looming, CEO Satya Nadella has rallied practically the entire company around developing AI tools and services based on OpenAI’s GPT models. That’s the foundation for Microsoft’s range of Copilot offerings, which provide automated support to employees working on text documents, spreadsheets, cybersecurity issues, and a host of other tasks.

“Almost everyone I know is working on Copilot to a certain extent,” the Microsoft AI researcher told BI.

The push for practically every team to prioritize generative AI in their applications has created some resentment among rank-and-file employees who say they’ve had to do more with less after in the wake of significant layoffs and a salary freeze.

‘One out of 10 times is magic’

Microsoft also recently rolled out its Copilot products internally, and employees told BI they’ve had mixed early results when it comes to the reliability of outputs. Still, they expect this will improve over time.

Current and former executives said there’s a race to add value to these somewhat unreliable and “gimmicky” AI tools before the excitement wears off and customers ask whether they’re really getting the return they expected out of the much-hyped technology.

“About one out of 10 times is magic,” one Microsoft executive told BI. “The rest of the time it’s, ‘Why do we even try?'”

Microsoft spokesman Frank Shaw said the company is optimistic about its AI tools and customers see a positive impact even at this early stage. Customers get a more finished version of the products than Microsoft employees, and the software giant is implementing feedback from all sources to improve the tools. “These things quickly move away from being party tricks,” he said.

“We’re the ones shipping this,” Shaw added, meaning it’s Microsoft who is bringing AI tools to the market in a meaningful way.

Improving the AI value proposition

Inside OpenAI, meanwhile, there’s growing pressure to become an actual business, rather than an AI research group. That’s led the startup onto Microsoft’s turf. In August, OpenAI launched an GPT Enterprise service that is more robust for businesses to use.

“OpenAI’s gain is not Microsoft’s loss yet,” another Microsoft executive said. “It may happen in the future.”

Microsoft’s answer has been to create Copilots, which can be seen as home-grown ChatGPT-like experiences for existing enterprise customers. The company is bundling these with its existing software, while offering customers access to many different large language models.

These generative AI tools and services are being sold as part of Microsoft’s “digital transformation” pitch, a long-running effort to convince business customers to adopt the latest technology, such as moving on-premise workloads to the cloud. Companies like Walmart and JPMorgan Chase have tapped Microsoft to build generative AI into their operations and products.

From FOMO to ROI

We’re still at the FOMO stage of enterprise AI adoption. Companies are more interested in trying this new technology, and are less worried about how much it costs. But Microsoft insiders know the conversation will soon switch to topics like return on investment.

There’s significant cost associated with this technology, according to one of the executives who spoke to BI, and the value is uncertain in the early stages of the adoption of these tools.

“The return on investment from the experiences they build into their products is unclear, and it’s a nontrivial cost,” this executive said, adding there’s a fear internally that corporate customers will decide what they’re getting isn’t worth the money.

Value, not price

For now, it seems, companies providing this technology are not trying to compete strictly on price. Microsoft Copilots are mostly available as add-ons to existing products for $30 per user per month. Google charges roughly the same for similar AI tools.

“It’s too premature to assume this is going to be a race to the bottom on price,” another Microsoft executive said. “The problem Microsoft sees is we just need to convince people there is value in it. Companies are not sure if this is too expensive. We don’t look at it like we need to make it cheaper, just add more value.”

So far, the feedback from customers on Microsoft AI tools is “mixed,” the executive said. “But it’s still too early. The value proposition has barely begun.”

“Gimmicky,” but saves time

One C-level executive from a large Microsoft customer told BI the company’s Copilot tool for its Microsoft 365 suite of business applications is “gimmicky.”

The tools are limited in what they can do, and while useful, “not yet mind blowing” or indispensable, the person said.

However, the person said the Copilot upgrade to Microsoft 365 does save time by summarizing meetings and long email threads with a reasonable degree of reliability.

Dentsu’s early Copilot views

Advertising agency Dentsu was in an early access program for Microsoft 365 Copilot and plans to add as many as 4,000 seats within the next month, according to Microsoft.

In an interview arranged by Microsoft, Dentsu Global Chief Product and Technology Officer Shiva Vannavada said employees report saving 30 to 40 minutes per day by using Microsoft 365 Copilot for tasks such as summarizing teams chats and generating presentations and executive summaries.

Dentsu conducts monthly surveys to gauge sentiment, according to Vannavada, and 80% of the employees had a very positive view of Copilot.

The remaining 20% had a mix of complaints, he said. Employees on the creative team don’t like the Copilot function in PowerPoint because they want their presentations to look good and the AI versions don’t meet that higher bar. Vannavada said Dentsu has training to help employees use Copilot to create more advanced outputs and applications.

Questions about an in-house GPU effort

Microsoft is trying to make generative AI faster, better, and cheaper. But there are fundamental technology limitations to making the models work accurately, the people who spoke to BI said.

These are also power-hungry workloads that rely heavily on Microsoft buying GPUs from Nvidia. Analysts at DA Davidson estimated Microsoft spent $4.5 billion on Nvidia chips last year. One of the Microsoft executives told BI that this was in the ballpark of its actual spending. Shaw declined to comment on this.

Microsoft has an internal effort to design its own AI chips to reduce reliance on Nvidia, but some employees see that effort as a waste of resources.

“Candidly, these efforts are a couple of years behind Nvidia, and all of these efforts take heavy lifting,” one of the Microsoft executives said. “You’re playing this rat race where the state of the art has moved on by the time you get there.”

Microsoft has become “tech support for OpenAI”

The group at the center of Microsoft’s enterprise AI dream is the “AI Platform” team run by Eric Boyd. This sits within Scott Guthrie’s Cloud + AI organization.

The AI Platform team has had many iterations. What was historically an organization with many in-house AI plays has leaned hard into the OpenAI partnership. That’s created some resentment and led to the departure of some executives who had worked on Microsoft’s home-grown AI initiatives.

Insiders say Microsoft is focused less on the internal services that previously made up Azure AI Services and more on the Azure OpenAI service.

One former executive who left as a result of the changes said products like AI Cognitive Search, Azure AI Bot Service, and Kinect DK are practically gone. Shaw said these services exist in some form, but are either not part of the Azure AI org, have been renamed, or have been bundled with other products.

“The former Azure AI is basically just tech support for OpenAI,” a former Microsoft executive said. “Eric Boyd is effectively maintaining the OpenAI service. It’s less of an innovation engine than it once was. Now it’s more IT for OpenAI. The beating heart of innovation is elsewhere.”

“It’s not very innovative but it’s a good business strategy,” this person added.

The Azure OpenAI service has hundreds of developers supporting customers of Microsoft’s Azure cloud service who use OpenAI’s GPT models. Some Microsoft employees work so closely with OpenAI that they have badges to get into OpenAI’s offices, and some OpenAI employees can badge into Microsoft locations.

Microsoft isn’t limiting itself to OpenAI

The AI Platform team as had a recent leadership shakeup after John Montgomery, an executive who previously ran product, was replaced by former Instacart COO Asha Sharma.

The team provides what’s called “model as a service” in the industry, where companies like Microsoft provide access to AI models via APIs.

Alongside its effort with OpenAI, Microsoft is expanding the number of models it makes available to enterprise customers. The software giant already offers models from Cohere and Meta. It recently invested $16 million in French startup Mistral AI.

Mistral models will be offered to Microsoft customers along with about 1,600 other models including Cohere and Meta’s Llama.

“Microsoft is a big 10 company, they are not going to limit model-as-a-service and cloud services to just OpenAI models,” one of the Microsoft executives said. “Making sure you have the widest variety of models on top of that infrastructure is just good business.”

The Mistral investment may look like Microsoft is strategically moving away from OpenAI. That’s helpful for keeping antitrust regulators at bay, but this person said the size of the investment should clear up that perception.

“$16 million versus $13 billion,” this person said, referring to Microsoft’s investments in Mistral and OpenAI, respectively. “There’s a desire for people to want drama but the math speaks for itself.”

Are you a Microsoft employee or someone else with insight to share?

Contact Ashley Stewart via email ([email protected]), or send a secure message from a non-work device via Signal (+1-425-344-8242).