bug: bench command fails with error.UseForwardPrefillDecode #55

New issue

Closed

opened 2026-05-20 21:02:19 +02:00 by sleepy · 0 comments

sleepy commented

2026-05-20 21:02:19 +02:00

Owner

The bench subcommand in src/bench/bench.zig calls model.forward() which returns error.UseForwardPrefillDecode. It should use forward_prefill() and forward_decode() instead.

Reproduction

./zig-out/bin/sleepy-llm bench --model ~/.sleepy-llm/models/Qwen3.5-0.8B --prompt "Hello" --max-tokens 128

Result: Error: UseForwardPrefillDecode

Expected

Benchmark should run and report prefill + decode tok/s.

Fix

Update bench_prefill to call model.forward_prefill() and bench_decode to call model.forward_decode() with proper KV-cache management.

The `bench` subcommand in `src/bench/bench.zig` calls `model.forward()` which returns `error.UseForwardPrefillDecode`. It should use `forward_prefill()` and `forward_decode()` instead. ## Reproduction ``` ./zig-out/bin/sleepy-llm bench --model ~/.sleepy-llm/models/Qwen3.5-0.8B --prompt "Hello" --max-tokens 128 ``` Result: `Error: UseForwardPrefillDecode` ## Expected Benchmark should run and report prefill + decode tok/s. ## Fix Update `bench_prefill` to call `model.forward_prefill()` and `bench_decode` to call `model.forward_decode()` with proper KV-cache management.

sleepy added the

bug

label

2026-05-20 21:07:29 +02:00

sleepy referenced this issue

2026-05-20 23:58:25 +02:00

bug: generate command produces garbage output on Qwen3.5-0.8B #57

sleepy referenced this issue from a commit

2026-05-21 00:04:59 +02:00

fix(#55): update bench command to use forward_prefill and forward_decode

sleepy referenced this issue

2026-05-21 00:06:42 +02:00