[mtp] Mixed MTP + batched generation phases 1-3 (#49) #64

Merged

sleepy merged 1 commit from feature/49-mixed-mtp-batch into main

2026-05-15 18:26:58 +02:00

sleepy commented

2026-05-15 18:18:35 +02:00

Owner

Summary

Fixes #49. Allows MTP speculative decoding and standard batched generation to run concurrently.

Changes

Phase 1: Remove MTP→batch transition — MTP requests stay in MTP mode when new requests arrive
Phase 2: Run _mtp_step() for MTP-enabled requests alongside batch generator in mixed mode
Phase 3: _mtp_step finds MTP-enabled request instead of always taking first running request

Test Results

81/82 scheduler tests pass. 1 failure is pre-existing (boundary snapshot token tracking).

Diff

+32/-40 lines in scheduler.py (net reduction). Clean, focused changes.

## Summary Fixes #49. Allows MTP speculative decoding and standard batched generation to run concurrently. ## Changes 1. **Phase 1**: Remove MTP→batch transition — MTP requests stay in MTP mode when new requests arrive 2. **Phase 2**: Run `_mtp_step()` for MTP-enabled requests alongside batch generator in mixed mode 3. **Phase 3**: `_mtp_step` finds MTP-enabled request instead of always taking first running request ## Test Results 81/82 scheduler tests pass. 1 failure is pre-existing (boundary snapshot token tracking). ## Diff +32/-40 lines in scheduler.py (net reduction). Clean, focused changes.

sleepy added 1 commit

2026-05-15 18:18:35 +02:00