Microsoft AI System Beats Generalist Physicians in Complex Medical Simulations

Microsoft’s latest AI diagnostic system outperformed generalist physicians in a controlled simulation of complex medical cases.

The system, known as MAI-DxO, achieved an accuracy rate of 83 percent on difficult cases drawn from the New England Journal of Medicine. In comparison, primary care doctors and general internists working alone with reference materials were correct only 36 percent of the time. Mustafa Suleyman, the chief executive officer of Microsoft AI, described the findings as a significant step toward achieving what the company terms medical superintelligence. This new capability represents a major milestone for the tech giant as it competes with other major players in the artificial intelligence space.

The diagnostic tool operates by coordinating multiple AI agents that function similarly to a panel of specialists. These agents question the patient, order necessary tests, and debate potential diagnoses before reaching a final conclusion. While the results are impressive, researchers noted that such complex cases are typically handled by specialists or teams in actual clinical settings, who were not included in this specific comparison. The study also indicated that much of the performance improvement stems from the underlying power of the AI models themselves, with the orchestration system providing additional gains on top of that base performance.

Microsoft tested the system using models from various providers, including OpenAI, Google, Anthropic, Meta, and DeepSeek. The highest results were achieved using OpenAI’s GPT-5 and GPT-5.5 models. Cost analysis showed that running the system with GPT-5 cost approximately $8,400 per case, while the orchestration layer reduced this expense by about $3,000 with a modest four-point accuracy increase. On emergency department cases, however, the addition of the system provided little improvement in either cost or accuracy compared to the standalone model. With the newer GPT-5.5 model, the system significantly boosted accuracy on complex cases from 72 percent to nearly 87 percent, though similar benefits were not observed in emergency scenarios.

The tool proved most useful for less capable models and complex diagnostic challenges. For instance, a weaker model scored 46.4 percent on its own but rose to nearly 69 percent with the assistance of the diagnostic system. Microsoft researchers noted that accuracy gains tend to shrink as models approach their theoretical maximum. The company is currently working with the Mayo Clinic to train a foundation model for healthcare, with a release expected next year. Suleyman believes this effort will enable substantial leaps in the field and position the system as foundational infrastructure for future breakthroughs in clinical care.

Suleyman, who previously co-founded DeepMind, stated that the company expects healthcare models to improve by an order of magnitude in capability within the next 24 to 36 months. He pointed to the rapid progress seen in coding and chat applications as evidence that progress accelerates once models are iterated upon using real-world usage data. A new partnership with Beth Israel Deaconess Medical Center in Boston will evaluate the system’s clinical reasoning across various hospital settings. Patients will receive standard care while consenting participants also engage with the AI tool, allowing researchers to compare the recommendations against human decisions.

This new study coincides with a broader manifesto from Microsoft outlining the requirements for medical superintelligence. The company aims to move beyond benchmarking individual tasks and instead integrate its technology directly with clinicians and healthcare systems. Microsoft already sells dictation and radiology tools to medical professionals and views this new diagnostic system as an expansion of those existing efforts. Suleyman emphasized that the goal is to build the best possible clinical decision support products in the coming years. He described the technology as a critical asset that the company is committed to accelerating into the world to improve patient outcomes and streamline the delivery of medical expertise.