Skip to content

[Cohere] Cohere2 moe parser fix#44907

Merged
vllm-bot merged 4 commits into
vllm-project:mainfrom
Terrencezzj:cohere2_moe-parser-fix
Jun 9, 2026
Merged

[Cohere] Cohere2 moe parser fix#44907
vllm-bot merged 4 commits into
vllm-project:mainfrom
Terrencezzj:cohere2_moe-parser-fix

Conversation

@Terrencezzj

@Terrencezzj Terrencezzj commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Purpose

  • Normalize Cohere Command streaming tool-call deltas so Melody placeholder fields are omitted from OpenAI-compatible responses.
  • Preserve pending tool-call IDs until the first real function-name delta, preventing clients from seeing invalid empty tool names.
  • Update Cohere2 MoE weight loading to use the current fused_moe_make_expert_params_mapping helper after the FusedMoE API change.

Test Plan

Test Result

Before fix:

tool_delta_1: [{"id":"0","type":"function","index":0,"function":{"name":"","arguments":""}}]
tool_delta_2: [{"id":"","type":"function","index":0,"function":{"name":"bash","arguments":""}}]
tool_delta_3: [{"id":"","type":"function","index":0,"function":{"name":"","arguments":"{\""}}]
tool_delta_4: [{"id":"","type":"function","index":0,"function":{"name":"","arguments":"command"}}]
tool_delta_5: [{"id":"","type":"function","index":0,"function":{"name":"","arguments":"\":"}}]
tool_delta_6: [{"id":"","type":"function","index":0,"function":{"name":"","arguments":" \"pwd\"}"}}]

After fix

tool_delta_1: [{"id":"0","type":"function","index":0,"function":{"name":"bash"}}]
tool_delta_2: [{"index":0,"function":{"arguments":"{\""}}]
tool_delta_3: [{"index":0,"function":{"arguments":"command"}}]
tool_delta_4: [{"index":0,"function":{"arguments":"\":"}}]
tool_delta_5: [{"index":0,"function":{"arguments":" \"pwd\"}"}}]

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
Signed-off-by: Terrencezzj <terrence@cohere.ai>
Signed-off-by: Terrencezzj <terrence@cohere.ai>
]

expert_params_mapping = FusedMoE.make_expert_params_mapping(
expert_params_mapping = fused_moe_make_expert_params_mapping(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this needed for tool parser fix

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, but it breaks our model. The PR caused this was merged this morning #41184

@sfeng33 sfeng33 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@sfeng33 sfeng33 added the ready ONLY add when PR is ready to merge/full CI is needed label Jun 8, 2026
@sfeng33 sfeng33 enabled auto-merge (squash) June 8, 2026 20:55
@vllm-bot vllm-bot merged commit ee4d7df into vllm-project:main Jun 9, 2026
44 of 46 checks passed
ekagra-ranjan pushed a commit to ekagra-ranjan/vllm that referenced this pull request Jun 9, 2026
Signed-off-by: Terrencezzj <terrence@cohere.ai>
Signed-off-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com>
waqahmed-amd-fi pushed a commit to waqahmed-amd-fi/vllm that referenced this pull request Jun 10, 2026
Signed-off-by: Terrencezzj <terrence@cohere.ai>
Signed-off-by: Waqar Ahmed <waqar.ahmed@amd.com>
Saddss pushed a commit to Saddss/vllm that referenced this pull request Jun 14, 2026
Signed-off-by: Terrencezzj <terrence@cohere.ai>
vivek8123 pushed a commit to odh-on-pz/vllm-upstream that referenced this pull request Jun 18, 2026
Signed-off-by: Terrencezzj <terrence@cohere.ai>
divineearthly pushed a commit to divineearthly/vllm that referenced this pull request Jun 19, 2026
Signed-off-by: Terrencezzj <terrence@cohere.ai>
Signed-off-by: divineearthly <divineearthly@gmail.com>
tunglinwood pushed a commit to tunglinwood/vllm that referenced this pull request Jun 22, 2026
Signed-off-by: Terrencezzj <terrence@cohere.ai>
nkzhenhua pushed a commit to nkzhenhua/vllm that referenced this pull request Jun 24, 2026
Signed-off-by: Terrencezzj <terrence@cohere.ai>
ohsono pushed a commit to ohsono/vllm that referenced this pull request Jul 3, 2026
Signed-off-by: Terrencezzj <terrence@cohere.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready ONLY add when PR is ready to merge/full CI is needed tool-calling

4 participants