Qwen3.5 model implementation #6

New issue

Closed

opened 2026-05-10 12:01:31 +02:00 by sleepy · 1 comment

sleepy commented

2026-05-10 12:01:31 +02:00

Owner

Implement the Qwen3.5 model family:

src/models/qwen3_5/model.zig — Model struct, load(), forward()
src/models/qwen3_5/config.zig — Hyperparameters (comptime where possible)
src/models/qwen3_5/attention.zig — GQA dispatch to attention.metal
src/models/qwen3_5/mlp.zig — SwiGLU dispatch to swiglu.metal
src/models/qwen3_5/tokenizer.zig — BPE from tokenizer.json
src/models/qwen3_5/mtp.zig — MTP head dispatch
src/models/qwen3_5/README.md
src/models/registry.zig — Model family registration hook

Acceptance criteria:

Model can load Qwen3.5-4B weights
Forward pass runs end-to-end
Tokenizer produces correct token IDs
MTP head works (if weights present)
Unit tests pass
No file exceeds ~400 lines

Max 2 attempts.

Implement the Qwen3.5 model family: - src/models/qwen3_5/model.zig — Model struct, load(), forward() - src/models/qwen3_5/config.zig — Hyperparameters (comptime where possible) - src/models/qwen3_5/attention.zig — GQA dispatch to attention.metal - src/models/qwen3_5/mlp.zig — SwiGLU dispatch to swiglu.metal - src/models/qwen3_5/tokenizer.zig — BPE from tokenizer.json - src/models/qwen3_5/mtp.zig — MTP head dispatch - src/models/qwen3_5/README.md - src/models/registry.zig — Model family registration hook Acceptance criteria: - Model can load Qwen3.5-4B weights - Forward pass runs end-to-end - Tokenizer produces correct token IDs - MTP head works (if weights present) - Unit tests pass - No file exceeds ~400 lines Max 2 attempts.

sleepy referenced this issue from a commit

2026-05-10 14:10:34 +02:00

feat: implement Qwen3.5 model family

sleepy referenced this issue from a pull request that will close it,

2026-05-10 14:11:13 +02:00

[models] Qwen3.5 model implementation (#6) #14

sleepy referenced this issue from a commit

2026-05-10 14:20:12 +02:00

[models] Qwen3.5 model implementation (#6) (#14)

sleepy closed this issue

2026-05-10 14:20:12 +02:00

sleepy commented

2026-05-10 14:20:26 +02:00

Author

Owner

Merged via squash in PR #14.

Config parsing with comptime assertions
BPE tokenizer with encode/decode
GQA attention layer with causal masking
SwiGLU MLP layer
Full model with embedding, stacked layers, forward pass
Model registry with factory pattern
54/54 tests pass
zig build ✅, zig build test ✅, zig build lint ✅

Merged via squash in PR #14. - Config parsing with comptime assertions - BPE tokenizer with encode/decode - GQA attention layer with causal masking - SwiGLU MLP layer - Full model with embedding, stacked layers, forward pass - Model registry with factory pattern - 54/54 tests pass - `zig build` ✅, `zig build test` ✅, `zig build lint` ✅

sleepy referenced this issue

2026-05-15 19:02:26 +02:00

perf: Pipeline decode — overlap GPU commit with next token embedding #47