[feature] SpecPrefill attention-based sparse prefill for MoE #58

New issue

Closed

opened 2026-05-14 23:29:42 +02:00 by sleepy · 1 comment

sleepy commented

2026-05-14 23:29:42 +02:00

Owner

The scheduler has SpecPrefill infrastructure (commit c5081e3) but it needs work to be production-ready for MoE models like Mixtral, Qwen3-MoE.

Current state: draft code exists, disabled by default.

Acceptance criteria:

SpecPrefill drops irrelevant expert computations during prefill
Measurable prefill speedup on MoE models (>20%)
No quality regression (perplexity within 0.1%)
Works with continuous batching

The scheduler has SpecPrefill infrastructure (commit c5081e3) but it needs work to be production-ready for MoE models like Mixtral, Qwen3-MoE. Current state: draft code exists, disabled by default. Acceptance criteria: - SpecPrefill drops irrelevant expert computations during prefill - Measurable prefill speedup on MoE models (>20%) - No quality regression (perplexity within 0.1%) - Works with continuous batching

sleepy added the

feature

label

2026-05-14 23:29:42 +02:00

sleepy closed this issue

2026-05-15 21:20:56 +02:00

sleepy commented

2026-05-15 21:20:56 +02:00

Author

Owner

Closed — not prioritized.