[feature] Complete mixed MTP + batched generation (phase 3)

sleepy commented

2026-05-14 23:28:56 +02:00

Owner

Branch: feature/10-mixed-mtp-batch (3 commits ahead of main)

Allows MTP speculative decoding and standard batched generation to run concurrently.

Commits:

8a6e2f1 phase 1: skip MTP to batch transition on new arrivals
c4ecef4 phase 2: run both MTP and batch paths in mixed mode
19811d4 phase 3: find MTP-enabled request in _mtp_step

Acceptance criteria:

MTP request continues speculating while batch handles new arrivals
No throughput regression for single-request MTP
Correct output when transitioning between MTP and batched modes
Tests for concurrent MTP + batch scenarios

Notes: Phases 1-2 were merged via PRs #9/#11. Phase 3 was not merged.

Branch: feature/10-mixed-mtp-batch (3 commits ahead of main) Allows MTP speculative decoding and standard batched generation to run concurrently. Commits: - 8a6e2f1 phase 1: skip MTP to batch transition on new arrivals - c4ecef4 phase 2: run both MTP and batch paths in mixed mode - 19811d4 phase 3: find MTP-enabled request in _mtp_step Acceptance criteria: - MTP request continues speculating while batch handles new arrivals - No throughput regression for single-request MTP - Correct output when transitioning between MTP and batched modes - Tests for concurrent MTP + batch scenarios Notes: Phases 1-2 were merged via PRs #9/#11. Phase 3 was not merged.

sleepy referenced this issue from a commit

2026-05-15 18:17:44 +02:00

feat(mtp): mixed MTP + batched generation phases 1-3 (#49)

sleepy referenced this issue from a pull request that will close it,

2026-05-15 18:18:35 +02:00

[mtp] Mixed MTP + batched generation phases 1-3 (#49) #64

sleepy referenced this issue from a commit

2026-05-15 18:26:11 +02:00

feat(mtp): mixed MTP + batched generation phases 1-3 (#49)

sleepy referenced this issue from a commit

2026-05-15 18:26:59 +02:00

[mtp] Mixed MTP + batched generation phases 1-3 (#49) (#64)

sleepy closed this issue

2026-05-15 18:26:59 +02:00

sleepy commented

2026-05-15 18:27:34 +02:00

Author

Owner

Merged via squash (PR #64). Three phases implemented: (1) skip MTP→batch transition, (2) run MTP alongside batch in mixed mode, (3) find MTP-enabled request in _mtp_step. Reviewer caught 2 bugs (missing cache extraction, dangling batch generator) — both fixed. 81/82 tests pass.

Rows
Columns

[feature] Complete mixed MTP + batched generation (phase 3) #49