sleepy/sleepy-llm
1
0
Fork
You've already forked sleepy-llm
 0
Code Issues 11 Pull requests 3 Projects Releases Packages Wiki Activity Actions

feat: GPU-accelerated prefill path #45

New issue
Open
opened 2026-05-15 19:02:26 +02:00 by sleepy · 0 comments
sleepy commented 2026-05-15 19:02:26 +02:00
Owner
Copy link

Current prefill runs on CPU (forward_cpu). This is slow for long prompts.

Plan:

  1. Move prefill to GPU using the batched matmul kernels (matmul_bf16_batch with 2D tiled approach)
  2. Batched attention for full attention layers (seq_len > 1 SDPA)
  3. Linear attention prefill stays CPU (forward_with_state) until we can port conv1d + delta rule to batched GPU
  4. Expected speedup: 10-50x for long prompts
Current prefill runs on CPU (forward_cpu). This is slow for long prompts. Plan: 1. Move prefill to GPU using the batched matmul kernels (matmul_bf16_batch with 2D tiled approach) 2. Batched attention for full attention layers (seq_len > 1 SDPA) 3. Linear attention prefill stays CPU (forward_with_state) until we can port conv1d + delta rule to batched GPU 4. Expected speedup: 10-50x for long prompts
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
perf/gpu-prefill
perf/cpu-gpu-pipeline
perf/lmhead-topk
perf/reduce-dispatch-count
perf/profile-decode
perf/mlx-style-gemv
perf/fuse-gate-up-matmul
perf/rms-norm-parallel-reduction
perf/46-eliminate-residual-copies
fix/62-model-correctness
fix/60-runtime-metal-compilation
fix/56-garbage-output
fix/57-tokenizer-test-path
fix/55-bench-command
perf/53-gemv-parity
perf/41-gpu-argmax
perf/43-fuse-norm-matmul
perf/40-gemv-bandwidth-optimization
perf/37-tune-threadgroup-sizes
perf/38-batched-qnorm-knorm
perf/36-single-cb-decode
feat/35-gpu-linear-attention-kernels
perf/gpu-only-rewrite
master
perf/27-kv-cache
perf/26-zero-copy-bf16
fix/29-bench-quick-mode
perf/gpu-hw-alignment
perf/37-decode-speed
fix/30-forward-pass-correctness
gpu-persistent-buffers
gpu-matmul-projections
fix/metal-encoder-commit
fix/benchmark-tokenizer-memory
fix/25-missing-field
No results found.
Labels
Clear labels   
bug
Something isn't working

  
doc
Documentation

  
feature
New feature or request

  
infra
Build/CI/infrastructure

  
perf
Performance improvement

  
refactor
Code cleanup or restructuring

No labels
bug
doc
feature
infra
perf
refactor
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
sleepy/sleepy-llm#45
No description provided.