sleepy/sleepy-llm
1
0
Fork
You've already forked sleepy-llm
 0
Code Issues 11 Pull requests 3 Projects Releases Packages Wiki Activity Actions

[fix] Update bench command to use forward_prefill and forward_decode (#55) #58

Open
sleepy wants to merge 1 commit from fix/55-bench-command into main
pull from: fix/55-bench-command
merge into: sleepy:main
Conversation 0 Commits 1 Files changed 1 +23 -22
sleepy commented 2026-05-21 00:06:42 +02:00
Owner
Copy link

Summary

Fixes the bench subcommand which was calling model.forward() (which now returns error.UseForwardPrefillDecode).

Changes

  • bench_prefill: Uses model.forward_prefill() instead of model.forward()
  • bench_decode: Uses model.forward_prefill() for prompt, then model.forward_decode() for token generation
  • Added sync_linear_states_to_gpu() after prefill to match engine pattern

Test Results

  • zig build compiles cleanly
## Summary Fixes the `bench` subcommand which was calling `model.forward()` (which now returns `error.UseForwardPrefillDecode`). ## Changes - `bench_prefill`: Uses `model.forward_prefill()` instead of `model.forward()` - `bench_decode`: Uses `model.forward_prefill()` for prompt, then `model.forward_decode()` for token generation - Added `sync_linear_states_to_gpu()` after prefill to match engine pattern ## Test Results - `zig build` compiles cleanly
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin fix/55-bench-command:fix/55-bench-command
git switch fix/55-bench-command

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff fix/55-bench-command
git switch fix/55-bench-command
git rebase main
git switch main
git merge --ff-only fix/55-bench-command
git switch fix/55-bench-command
git rebase main
git switch main
git merge --no-ff fix/55-bench-command
git switch main
git merge --squash fix/55-bench-command
git switch main
git merge --ff-only fix/55-bench-command
git switch main
git merge fix/55-bench-command
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bug
Something isn't working

  
doc
Documentation

  
feature
New feature or request

  
infra
Build/CI/infrastructure

  
perf
Performance improvement

  
refactor
Code cleanup or restructuring

No labels
bug
doc
feature
infra
perf
refactor
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
sleepy/sleepy-llm!58
No description provided.