sleepy/llama.cpp
1
0
Fork 0
Code Issues 6 Pull Requests Actions 182 Packages Projects Releases Wiki Activity

server: Add cached_tokens info to oaicompat responses (#19361)

Browse Source 
* tests : fix fetch_server_test_models.py

* server: to_json_oaicompat cached_tokens

Adds OpenAI and Anthropic compatible information about the
number of cached prompt tokens used in a response.
This commit is contained in:
Ryan Goulden
2026-03-19 11:09:33 -07:00
committed by GitHub
parent 76f2dc70c3
commit 26c9ce1288
6 changed files with 61 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tools/server/tests/unit/test_compat_anthropic.py
+2
View File
@@ -63,8 +63,10 @@ def test_anthropic_messages_basic():
assert "text" in res.body["content"][0], "Text content block missing 'text' field"
assert res.body["stop_reason"] in ["end_turn", "max_tokens"], f"Invalid stop_reason: {res.body.get('stop_reason')}"
assert "usage" in res.body, "Missing 'usage' field"
assert "cache_read_input_tokens" in res.body["usage"], "Missing usage.cache_read_input_tokens"
assert "input_tokens" in res.body["usage"], "Missing usage.input_tokens"
assert "output_tokens" in res.body["usage"], "Missing usage.output_tokens"
assert isinstance(res.body["usage"]["cache_read_input_tokens"], int), "cache_read_input_tokens should be integer"
assert isinstance(res.body["usage"]["input_tokens"], int), "input_tokens should be integer"
assert isinstance(res.body["usage"]["output_tokens"], int), "output_tokens should be integer"
assert res.body["usage"]["output_tokens"] > 0, "Should have generated some tokens"