[bug] MTP continues generating past EOS token #54

New issue

Closed

opened 2026-05-14 23:29:42 +02:00 by sleepy · 1 comment

sleepy commented

2026-05-14 23:29:42 +02:00

Owner

When MTP speculative decoding is active, the model sometimes generates past its EOS token, producing hallucinated content (e.g. user turn markers or tool calls) after the response should have ended.

Partial fix in commit 9a0d599 (stop-token guard for accepted draft tokens). Safety net added in stable/pre-q4kv commit b124ccf.

Acceptance criteria:

MTP step never emits tokens past a stop token
Draft tokens that are stop tokens are caught immediately on accept
Bonus tokens that are stop tokens are caught before emission
Verified tokens on rejection path are caught
Safety net filter in fallthrough output path

When MTP speculative decoding is active, the model sometimes generates past its EOS token, producing hallucinated content (e.g. user turn markers or tool calls) after the response should have ended. Partial fix in commit 9a0d599 (stop-token guard for accepted draft tokens). Safety net added in stable/pre-q4kv commit b124ccf. Acceptance criteria: - MTP step never emits tokens past a stop token - Draft tokens that are stop tokens are caught immediately on accept - Bonus tokens that are stop tokens are caught before emission - Verified tokens on rejection path are caught - Safety net filter in fallthrough output path

sleepy added the

bug

label

2026-05-14 23:29:42 +02:00

sleepy referenced this issue from a pull request that will close it,

2026-05-15 00:59:10 +02:00

[mtp] Fix EOS token leak in output_token_ids (#54) #60

sleepy referenced this issue from a commit

2026-05-15 01:01:43 +02:00

[mtp] Fix EOS token leak in output_token_ids (#54) (#60)

sleepy closed this issue

2026-05-15 01:01:44 +02:00

sleepy commented

2026-05-15 01:01:52 +02:00

Author

Owner

Merged via PR #60 (squash). Fix: trim stop token from output_token_ids on 3 early-return finish paths. Verified with Qwen3.6-27B-MXFP4-MTP: 5/5 prompts clean (was 4/5 leaking <|im_end|>).

sleepy referenced this issue

2026-05-15 01:50:13 +02:00

Tool response token leakage #17

sleepy referenced this issue

2026-05-15 01:50:14 +02:00

[bug] Tool response token leakage (#17 reopen) #56