[cache] Skip q4 KV quantization on restored prefix caches (#48) #62

Merged

sleepy merged 2 commits from fix/48-q4-cache-prefix-restore into stable/pre-q4kv

2026-05-15 01:37:41 +02:00

sleepy commented

2026-05-15 01:33:32 +02:00

Owner

Summary

Guard _apply_quantized_kv() in _do_external_prefill() to skip when existing_cache is provided
Root cause: restored prefix caches carry fp16 KV data that must be preserved; converting them to QuantizedKVCache destroys the cached data and causes cache misses

Closes #48

Test results

80/80 scheduler tests pass

## Summary - Guard `_apply_quantized_kv()` in `_do_external_prefill()` to skip when `existing_cache` is provided - Root cause: restored prefix caches carry fp16 KV data that must be preserved; converting them to QuantizedKVCache destroys the cached data and causes cache misses Closes #48 ## Test results - 80/80 scheduler tests pass

sleepy added 1 commit

2026-05-15 01:33:32 +02:00

fix: skip KV quantization for restored prefix caches (#48 ) 83a625b322

When existing_cache is provided (restored prefix cache), _apply_quantized_kv
was replacing fp16 KVCache objects with QuantizedKVCache, destroying the
restored prefix data and causing effective cache misses. Skip quantization
for restored caches to preserve the prefix data intact.

sleepy added 1 commit

2026-05-15 01:37:33 +02:00

fix(cache): guard _transition_to_mtp quantized KV conversion on restored caches bb3c4e3700

sleepy merged commit bc2e714dd3 into stable/pre-q4kv

2026-05-15 01:37:41 +02:00

sleepy referenced this pull request from a commit

2026-05-15 01:37:41 +02:00

[cache] Skip q4 KV quantization on restored prefix caches (#48) (#62)

sleepy referenced this pull request

2026-05-15 01:37:51 +02:00

CRITICAL: Cache misses with q4 KV quant — model stops before tool calls #48