Why can't the large model generate "Ma Jiaqi"? MiniMax full vocabulary scan reveals that nearly 5% of tokens are forgotten during subsequent training.

According to the Beating Monitoring, MiniMax published a technical blog revealing the root cause investigation process for why its M2 series large models cannot output the name “Ma Jiaqi.” The investigation started from a specific example and ultimately uncovered a systemic degradation issue affecting the entire vocabulary.

The root cause was that the tokenizer (the component that splits text into units processed by the model) merged “Jiaqi” into a single independent token during training. During pretraining, the model saw a lot of internet texts and learned this token; but in subsequent fine-tuning with dialogue data, there were fewer than five samples containing “Jiaqi.” During fine-tuning, high-frequency tokens like tool_call markers and code symbols continuously updated their surrounding vector space, pushing low-frequency tokens like “Jiaqi” into the wrong directions. The model still “recognizes” Ma Jiaqi and can accurately answer related information; the only thing lost is the ability to output this token.

The team then conducted a full scan of the complete vocabulary of about 200,000 tokens and found that approximately 4.9% of tokens experienced significant degradation. The most severe degradation was in Japanese: 29.7% of Japanese tokens degraded significantly, far exceeding Korean (3.3%), Russian (3.7%), Chinese (3.9%), and English (3.5%). Other top-ranking degraded tokens included internet SEO spam words like “Legend Private Server” and “Painless Abortion,” which have mechanisms identical to “Jiaqi.”

The severe degradation in Japanese also unraveled an old mystery. Previously, the model occasionally mixed Russian or Korean characters into Japanese conversations, with no clear explanation. This analysis showed that after Japanese token parameter drift, Japanese tokens became confused with tokens from other languages in the vector space, leading to both incorrect activation of Japanese tokens (language mixing) and pushing neighboring low-frequency Chinese tokens out of the normal probability range (token forgetting).

The fix was to construct a comprehensive synthetic dataset covering the entire vocabulary, allowing the model to practice each token with simple repetition tasks. The results were immediate: the proportion of Japanese responses mixed with Russian characters dropped from 47% to 1%, and the stability of output parameters across the entire vocabulary (cosine similarity) increased from a low of 0.329 to above 0.97 for all tokens.