[bug] Tool response token leakage (#17 reopen)

sleepy commented

2026-05-14 23:29:42 +02:00

Owner

Issue #17 was about tool response tokens leaking into output. Check if this is still reproducible on current main, especially with MTP enabled (which generates tokens in pairs and may miss multi-token stop sequences).

Acceptance criteria:

Tool response tokens never appear in streamed output
Works correctly with both standard batching and MTP paths

Issue #17 was about tool response tokens leaking into output. Check if this is still reproducible on current main, especially with MTP enabled (which generates tokens in pairs and may miss multi-token stop sequences). Acceptance criteria: - Tool response tokens never appear in streamed output - Works correctly with both standard batching and MTP paths

sleepy added the

bug

label

2026-05-14 23:29:42 +02:00

sleepy commented

2026-05-15 01:50:14 +02:00

Author

Owner

PR #60 (issue #54) fixes the <|im_end|> leak. The structural token leak (user markers) from MTP heads is a separate issue. Needs server-level testing with tool calling to reproduce and fix. The 4B mxfp4 test model is broken (produces token 0). Cannot verify without a working MTP model at smaller size.

sleepy referenced this issue from a commit

2026-05-15 17:55:18 +02:00

fix(mtp): integrate output parser into MTP step to prevent token leakage (#56)

sleepy referenced this issue from a pull request that will close it,

2026-05-15 17:58:03 +02:00

[mtp] Integrate output parser into MTP step to prevent token leakage (#56) #63

sleepy referenced this issue from a commit

2026-05-15 18:09:40 +02:00

fix(mtp): integrate output parser into MTP step to prevent token leakage (#56)

sleepy referenced this issue from a commit

2026-05-15 18:11:02 +02:00

[mtp] Integrate output parser into MTP step to prevent token leakage (#56) (#63)

sleepy closed this issue

2026-05-15 18:11:02 +02:00

sleepy commented

2026-05-15 18:11:27 +02:00

Author

Owner

Merged via squash (PR #63). Root cause: _mtp_step bypassed _get_output_parser_session(), so Harmony models had no channel-aware token filtering during MTP generation. Fix: route all MTP token emission through parser session with 3 extracted helpers. 81/82 scheduler tests pass (1 pre-existing failure on boundary snapshots).

Merged via squash (PR #63). Root cause: `_mtp_step` bypassed `_get_output_parser_session()`, so Harmony models had no channel-aware token filtering during MTP generation. Fix: route all MTP token emission through parser session with 3 extracted helpers. 81/82 scheduler tests pass (1 pre-existing failure on boundary snapshots).

sleepy referenced this issue

2026-05-15 18:32:01 +02:00

Tool response token leakage #17

Rows
Columns

[bug] Tool response token leakage (#17 reopen) #56