[Cohere] Fix Cohere2MoE weight loading when using Transformers ≥5.10

[Terrencezzj]

Problem

Cohere2MoeDecoderLayer and Cohere2MoeAttention relied on getattr(config, "first_k_dense_replace", 0) to decide whether a layer uses a dense MLP or MoE.

Under Transformers 5.10+, Cohere2MoeConfig consumes first_k_dense_replace and sets mlp_layer_types = ["dense", "sparse", ...], leaving first_k_dense_replace as None. vLLM then defaulted to 0, built layer 0 as MoE, and failed to load checkpoints with dense layer-0 weights:

KeyError: 'layers.0.mlp.down_proj.weight'

Fix

• Normalize config in Cohere2MoeModel(before make_layers): if mlp_layer_types is absent, derive it from legacy first_k_dense_replace, otherwise default to all "sparse"
• add is_prefix_dense_layer()

Test Plan

Test Result

Deterministic generation produces identical output under Transformers 5.9.0 and 5.10.2

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[mergify]

Hi @Terrencezzj, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

[Terrencezzj]

Hi @Terrencezzj, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

It's failed because of deepseekv4