Skip to content

Fix Mistral-Small-3.1-24B contrib: get_neuron_config_cls returns wrong class#112

Open
jimburtoft wants to merge 1 commit intoaws-neuron:mainfrom
jimburtoft:sweeper/fix-mistral-small-config
Open

Fix Mistral-Small-3.1-24B contrib: get_neuron_config_cls returns wrong class#112
jimburtoft wants to merge 1 commit intoaws-neuron:mainfrom
jimburtoft:sweeper/fix-mistral-small-config

Conversation

@jimburtoft
Copy link
Copy Markdown
Contributor

Description

This is a fix to an existing contrib model, not a new contribution.

The Mistral3InferenceConfig.get_neuron_config_cls() method returns the base NeuronConfig instead of Mistral3NeuronConfig. This means the custom attn_cls field defined in Mistral3NeuronConfig is never used.

Changes

One-line fix in modeling_mistral3.py: change get_neuron_config_cls to return Mistral3NeuronConfig instead of NeuronConfig.

This is a correctness fix -- the decoder layer currently hardcodes NeuronMistral3Attention directly so the attn_cls field on Mistral3NeuronConfig is effectively dead code, but the config class should still return the correct subclass for consistency and to enable future use of attn_cls.

Validation Results

Metric Value
Instance trn2.3xlarge (spot, $0.90/hr)
TP 4 (LNC=2)
Compile time 258s
Throughput ~34.5 tok/s
Token match 88% (4/5 prompts 100% match)

Files Changed

File Change
contrib/models/Mistral-Small-3.1-24B-Instruct-2503/src/modeling_mistral3.py Fix get_neuron_config_cls return type

Testing

Compiled and ran on trn2.3xlarge with TP=4, Neuron SDK 2.27, BF16. Model generates coherent text with 88% token match (4 of 5 prompts exact match against HuggingFace CPU reference).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants