perf: Tune threadgroup sizes per matmul shape #37

New issue

Closed

opened 2026-05-15 19:02:26 +02:00 by sleepy · 1 comment

sleepy commented

2026-05-15 19:02:26 +02:00

Owner

Current dispatch uses fixed ROWS_PER_TG=4 for all matmul shapes. Different shapes need different tuning:

lm_head (N=152960, K=2560): Needs many threadgroups, ROWS_PER_TG=1 with larger SG stride may be better
gate/up proj (N=9216, K=2560): Medium, current config may be ok
small projections (N=32, K=2560): Only 32 output rows — ROWS_PER_TG=4 gives just 8 threadgroups. Use ROWS_PER_TG=1 with a larger reduction group
MLP down (N=2560, K=9216): K is large, more lanes in reduction helps

Autotune or use a lookup table based on (N, K) to select optimal (ROWS_PER_TG, threads_per_row).

Current dispatch uses fixed ROWS_PER_TG=4 for all matmul shapes. Different shapes need different tuning: - lm_head (N=152960, K=2560): Needs many threadgroups, ROWS_PER_TG=1 with larger SG stride may be better - gate/up proj (N=9216, K=2560): Medium, current config may be ok - small projections (N=32, K=2560): Only 32 output rows — ROWS_PER_TG=4 gives just 8 threadgroups. Use ROWS_PER_TG=1 with a larger reduction group - MLP down (N=2560, K=9216): K is large, more lanes in reduction helps Autotune or use a lookup table based on (N, K) to select optimal (ROWS_PER_TG, threads_per_row).

sleepy referenced this issue from a commit

2026-05-15 19:32:30 +02:00

perf(#37): make ROWS_PER_TG a runtime parameter, tune per matmul shape

sleepy referenced this issue from a pull request that will close it,

2026-05-15 19:34:02 +02:00

[perf] Make ROWS_PER_TG runtime parameter, tune per matmul shape (#37) #50

sleepy referenced this issue from a commit

2026-05-15 19:34:50 +02:00

[perf] Make ROWS_PER_TG runtime parameter, tune per matmul shape (#37) (#50)

sleepy closed this issue

2026-05-15 19:34:50 +02:00

sleepy commented

2026-05-15 19:35:01 +02:00

Author

Owner

Merged via PR #50 (squash). Coherence: verified. Decode: 40ms→41ms (noise, no regression). ROWS_PER_TG is now runtime — enables future autotuning. Threadgroup tuning alone cannot break the 45% bandwidth ceiling; requires #40/#42.