Skip to content

Fix llava-v1.5-7b contrib: remove docstring wrapping entire module#113

Open
jimburtoft wants to merge 1 commit intoaws-neuron:mainfrom
jimburtoft:sweeper/fix-llava-docstring
Open

Fix llava-v1.5-7b contrib: remove docstring wrapping entire module#113
jimburtoft wants to merge 1 commit intoaws-neuron:mainfrom
jimburtoft:sweeper/fix-llava-docstring

Conversation

@jimburtoft
Copy link
Copy Markdown
Contributor

Description

This is a fix to an existing contrib model, not a new contribution.

The entire modeling_llava.py file is wrapped in an extraneous """ on line 17, turning all the custom class definitions into a docstring. This causes a SyntaxError on import -- none of the custom classes (NeuronLlavaAttention, NeuronLlavaDecoderLayer, NeuronLlavaForCausalLM, etc.) are actually defined.

The model previously appeared to work only because NeuronLlavaForCausalLM inherits from NeuronBaseForCausalLM via the base class, and the __init__.py imports fell back to defaults. But none of the llava-specific customizations (attention, MLP, decoder layer overrides) were active.

Changes

Removed the extraneous """ on line 17 of modeling_llava.py so the module parses correctly and all custom classes are properly defined.

Validation Results

Metric Value
Instance inf2.8xlarge
TP 2
Compile time 216.6s
Throughput ~14-19 tok/s
Token match N/A (SyntaxError before fix)

Files Changed

File Change
contrib/models/llava-v1.5-7b/src/modeling_llava.py Remove extraneous """ that wrapped entire module

Testing

Compiled and ran on inf2.8xlarge with TP=2, Neuron SDK 2.27, BF16. Model compiles, loads, and generates coherent text. Before the fix, import modeling_llava raised SyntaxError.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants