AI’s biggest models have a hidden flaw. There’s a way to fix it without retraining

We found a way to shrink one of OpenAI’s largest open models by roughly a quarter, from about 218 gigabytes down to 164 GB, while making it more accurate on average, not less. No retraining. No extra data. Just a different way of reading a model that already exists.

Normally, improving a large AI model means going back to the drawing board: retraining it from scratch, or fine-tuning it in a process that devours computing power and data most organisations simply don’t have. So, it’s worth asking a simpler question: could you fix a model’s weakness without touching its training at all?

In research being presented at ICML 2026, we found that you often can, by tackling one of the most stubborn problems in modern AI, using a signal that had been sitting inside these models the whole time.

Many of today’s largest AI language models use a design called a Sparse Mixture of Experts (SMoE). Rather than push every calculation through one enormous network, they split the work among many smaller sub-networks called experts, with a traffic controller, the router, sending each chunk of incoming text to just a few of them. Only a fraction of the model activates at any moment, which is how these systems grow so large without becoming impossible to run.

But there’s a hidden inefficiency. As these models train, their experts tend to drift toward sameness. Different experts start producing near-identical, redundant outputs, a problem researchers call representation collapse, or expert collapse. It’s like hiring a roomful of specialists and watching them slowly turn into copies of one another. You’re paying for variety and getting repetition.

We checked ten state-of-the-art models, ranging from 4 billion to 120 billion parameters, including OLMoE, DeepSeek-MoE, Qwen, and OpenAI’s GPT-OSS. Every one of them showed this collapse to some degree.

Fixing it has always meant expensive intervention: retraining a model from scratch or fine-tuning it. Both are slow and costly, which is a large part of why the problem has persisted even in the most advanced systems on the market today.

We had to wonder: could the information needed to route inputs effectively already be present in the model, without leaning on the trained router and without any retraining? That information is there, hiding inside the experts themselves.

Each expert is defined by large tables of numbers, its weights, shaped by everything it learned during training. Using a technique called Singular Value Decomposition SMoE (SSMoE), we extracted the eigenvectors of those weights. Think of them as fingerprints left behind by training, revealing what each expert has come to specialise in.

In several models, pairs of experts that the trained router treated as near interchangeable, sending them a similar mix of inputs, turned out, once we read their fingerprints, to have specialised in noticeably different things: one leaning toward factual recall, the other toward multi-step reasoning. The router had lost the ability to tell them apart. The fingerprints hadn’t.

SSMoE reads those fingerprints directly to help route each input, instead of relying only on the trained router. In effect, it helps the model tell its experts apart again. And because it only reads structure that’s already there, it needs no retraining and can be bolted onto an existing model as a plug-in.

Applied to OpenAI’s GPT-OSS models, SSMoE lifted average performance by around 6% without a single step of retraining, and cut memory use by roughly a quarter, from about 218 GB down to 164 GB for the largest model we tested.That won’t put a 120-billion-parameter model on your laptop; 164 GB still calls for serious hardware. But it’s a substantial saving on a model that’s already trained, with real cost implications for anyone running these systems at scale.

Because so much expert capacity had collapsed into redundancy, we could switch off a quarter of the experts on reasoning tasks, and the trimmed model still outperformed the original.

That’s evidence that a good deal of expert capacity in today’s largest models is pure duplication: not extra ability, just extra weight.

Leaner isn’t the only benefit. Real-world data is rarely as clean as the benchmarks models are usually tested on. To probe this, we deliberately corrupted the inputs, injecting junk character sequences into text and random noise into images, to mimic degraded, real-world conditions. Conventional, heavier representations buckled once conditions turned noisy. Our fingerprint-based approach held its ground, coming out ahead on 16 of 18 corrupted text tasks, and its advantage grew as the input got worse.

We tested this with synthetic corruption rather than real-world datasets, but it points to where SSMoE could help most: exactly the places data is rarely tidy, like OCR’d documents and scanned records, or the typo-strewn queries real users type into a chatbot.

When we trimmed experts aggressively, performance on even primary-school-level maths problems dropped, because some of the experts we discarded were precisely the ones doing the multi-step calculation.

For maths-heavy work, keeping the full set of experts is still the better call. “Redundant” and “unnecessary” turn out not to be the same thing, and knowing the difference matters more than the topline numbers.

We didn’t retrain these models or feed them more data – we read what they’d already learned about themselves.

As AI systems keep growing, and the cost of retraining grows with them, some of the biggest improvements may not require retraining at all, just a closer look at what we’ve already built.

The next test: will this method hold to detect hallucinations Large Vision-Language Models for medicine, with the aim of supporting the safe application and control of AI in medical settings, such as radiology report generation.

The paper is available at: https://arxiv.org/pdf/2605.30992
The code is available at: https://github.com/giangdip2410/SSMoE

Stories worth sharing

The text of this article is licensed under the Creative Commons Attribution (CC BY) 4.0 Internationallicense. We’d love for you to share it, so feel free. Images, videos, graphics and logos are not covered by the CC BY license and may not be used without permission from Deakin or the respective copyright holder. For more information on how to share or reuse this content, please contact researchcomms@deakin.edu.au.