[models] Qwen3.5 model implementation (#6) #14

Merged

sleepy merged 4 commits from feature/6-qwen3_5-model into master

2026-05-10 14:20:12 +02:00

sleepy commented

2026-05-10 14:11:13 +02:00

Owner

Summary

Qwen3.5 model family with config, tokenizer, attention, MLP, MTP, and registry.

Changes

src/models/qwen3_5/config.zig — Model config with JSON parsing and comptime assertions
src/models/qwen3_5/tokenizer.zig — BPE tokenizer with encode/decode
src/models/qwen3_5/attention.zig — GQA attention layer with CPU reference forward
src/models/qwen3_5/mlp.zig — SwiGLU MLP layer
src/models/qwen3_5/mtp.zig — MTP head placeholder
src/models/qwen3_5/model.zig — Full model with embedding, layers, KV cache, forward pass
src/models/registry.zig — Model family registration

Verification

zig build ✅
zig build test ✅ (54/54 tests pass)
zig build lint ✅

Closes #6

## Summary Qwen3.5 model family with config, tokenizer, attention, MLP, MTP, and registry. ## Changes - src/models/qwen3_5/config.zig — Model config with JSON parsing and comptime assertions - src/models/qwen3_5/tokenizer.zig — BPE tokenizer with encode/decode - src/models/qwen3_5/attention.zig — GQA attention layer with CPU reference forward - src/models/qwen3_5/mlp.zig — SwiGLU MLP layer - src/models/qwen3_5/mtp.zig — MTP head placeholder - src/models/qwen3_5/model.zig — Full model with embedding, layers, KV cache, forward pass - src/models/registry.zig — Model family registration ## Verification - `zig build` ✅ - `zig build test` ✅ (54/54 tests pass) - `zig build lint` ✅ Closes #6

sleepy added 2 commits

2026-05-10 14:11:13 +02:00

chore: remove accidental comptime_test file de61104782

feat: implement Qwen3.5 model family 2f9a31ca5e

- Add Config with JSON parsing and comptime assertions
- Add BPE tokenizer with encode/decode
- Add GQA AttentionLayer with CPU reference forward
- Add SwiGLU MLPLayer
- Add MTPHead placeholder
- Add full Model with embed, layers, RMSNorm, LM head
- Add model Registry with factory pattern
- Wire all modules into main.zig for test discovery

Refs: #6

sleepy commented

2026-05-10 14:13:06 +02:00

Author

Owner

CHANGES_REQUESTED

model.zig:73-90 — Layer allocation leak on error. errdefer only frees array, not Layer contents. Add per-layer cleanup.
model.zig:92-98 — Same leak if MTPHead.init fails after loop.
model.zig:239 — KV cache created but never used. Either implement KV cache in AttentionLayer.forward or remove until functional.
attention.zig:97-140 — Missing causal masking for autoregressive generation. Position s should only attend to 0..s.
tokenizer.zig:109 — Unknown tokens silently map to ID 0. Return error or use proper unk token.

CHANGES_REQUESTED 1. model.zig:73-90 — Layer allocation leak on error. errdefer only frees array, not Layer contents. Add per-layer cleanup. 2. model.zig:92-98 — Same leak if MTPHead.init fails after loop. 3. model.zig:239 — KV cache created but never used. Either implement KV cache in AttentionLayer.forward or remove until functional. 4. attention.zig:97-140 — Missing causal masking for autoregressive generation. Position s should only attend to 0..s. 5. tokenizer.zig:109 — Unknown tokens silently map to ID 0. Return error or use proper unk token.

sleepy referenced this pull request from a commit

2026-05-10 14:16:23 +02:00

Fix PR #14 review feedback: memory leaks, causal masking, unknown token error, remove unused KVCache

sleepy added 1 commit

2026-05-10 14:16:23 +02:00

Fix PR #14 review feedback: memory leaks, causal masking, unknown token error, remove unused KVCache ed6cc40d44

sleepy added 1 commit

2026-05-10 14:20:02 +02:00

fix: remove unused KVCache struct ed235b999e

sleepy merged commit 7cc66c60b0 into master

2026-05-10 14:20:12 +02:00

sleepy referenced this pull request from a commit