[cache] Default max_kv_size to 32768 (#4) #6

Merged

sleepy merged 2 commits from perf/4-default-max-kv-size into main

2026-05-15 20:40:11 +02:00

sleepy commented

2026-05-15 20:34:45 +02:00

Owner

Summary

Changes max_kv_size default from None to 32768 across all generation entry points. Combined with PR #5 pre-allocation, KV cache memory is now flat and bounded by default.

Changes

generate_step(), BatchGenerator.__init__(), make_prompt_cache() — default 32768
CLI args in generate.py, chat.py, cache_prompt.py — default 32768
Tests updated to pass max_kv_size=None where unbounded behavior is needed

Test results

49/49 passed

## Summary Changes `max_kv_size` default from `None` to `32768` across all generation entry points. Combined with PR #5 pre-allocation, KV cache memory is now flat and bounded by default. ## Changes - `generate_step()`, `BatchGenerator.__init__()`, `make_prompt_cache()` — default 32768 - CLI args in `generate.py`, `chat.py`, `cache_prompt.py` — default 32768 - Tests updated to pass `max_kv_size=None` where unbounded behavior is needed ## Test results 49/49 passed

sleepy added 1 commit

2026-05-15 20:34:45 +02:00

Default max_kv_size to 32768 for bounded KV cache memory

Build and Test / check_lint (pull_request) Waiting to run

Details

Build and Test / mac_build_and_test (pull_request) Blocked by required conditions

Details

b1617383f1

Change default max_kv_size from None to 32768 across all public
generation entry points. This pairs with the pre-allocation from PR #5
to give flat memory usage by default while still allowing users to
pass None for unbounded behavior.

sleepy added 1 commit

2026-05-15 20:38:52 +02:00

Fix wasteful RotatingKVCache allocation in _make_new_cache and update docstring

Build and Test / check_lint (pull_request) Has been cancelled

Details

Build and Test / mac_build_and_test (pull_request) Has been cancelled

Details

48ec392f27

Pass max_kv_size=None to make_prompt_cache in the bounded path so it
creates plain KVCache objects instead of throwaway RotatingKVCache(keep=4)
that are immediately replaced. Also clarify make_prompt_cache docstring
to note the 32768 default and that None disables bounded cache.

sleepy referenced this pull request

2026-05-15 20:39:29 +02:00

[perf] Default max_kv_size to 32768 #4

sleepy merged commit df607cab1a into main

2026-05-15 20:40:11 +02:00