From 48bbb14a64f10b61a4c66dd8135e7024307b65e6 Mon Sep 17 00:00:00 2001 From: Pi Agent Date: Wed, 15 Apr 2026 19:55:44 +0300 Subject: [PATCH] =?UTF-8?q?Phase=205:=20Q4=5F0=20vdr=5Fmmvq=202=E2=86=924?= =?UTF-8?q?=20(+19%=20tg128,=2030=E2=86=9236=20t/s)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root cause analysis corrected: Q4_0 low BW utilization is NOT due to SYCL submission model overhead. Per-op profiling proves the bottleneck is dp4a compute throughput — nibble extraction requires 2 dp4a per byte vs Q8_0's 1 dp4a per byte, making Q4_0 compute-bound at ~30 t/s. Fix: Increase vdr_mmvq from 2 to 4 for Q4_0 reorder path, processing 16 blocks per subgroup instead of 8. Better amortizes dp4a overhead. Benchmark (Qwen3.5-9B, Arc A770, llama-bench -p 512 -n 128 -r 3): Q4_0: 30.16 → 35.96 t/s (+19%) Q8_0: 29.96 → 30.82 t/s (within noise) Q4_K_M: 24.65 → 25.32 t/s (within noise) Also includes: - Timing instrumentation patch (for debugging, not applied to source) - Updated decisions log (Decisions 8-9) - Updated workplan with revised benchmark data - Root cause analysis document --- repos/llama.cpp | 2 +- repos/patch/README.md | 81 ++++++++++++------------- repos/patch/phase5-mmvq-vdr/patch.diff | 13 ++++ repos/patch/phase5-op-timing/patch.diff | 69 +++++++++++++++++++++ 4 files changed, 123 insertions(+), 42 deletions(-) create mode 100644 repos/patch/phase5-mmvq-vdr/patch.diff create mode 100644 repos/patch/phase5-op-timing/patch.diff diff --git a/repos/llama.cpp b/repos/llama.cpp index 8dc530b..80d8770 160000 --- a/repos/llama.cpp +++ b/repos/llama.cpp @@ -1 +1 @@ -Subproject commit 8dc530b86d44cd0667a685539f29ded70a08ae0a +Subproject commit 80d8770804eb712f0464c3705b65acf896c1f49c diff --git a/repos/patch/README.md b/repos/patch/README.md index 5a59dde..84d0c93 100644 --- a/repos/patch/README.md +++ b/repos/patch/README.md @@ -28,51 +28,38 @@ cd repos - **oneAPI:** 2025.3.2, DPC++ compiler 2025.3.2 - **Build:** `-DGGML_SYCL=ON -DGGML_VULKAN=OFF -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DCMAKE_BUILD_TYPE=Release` -## Benchmark Results (Clean Run, 2026-04-15) +## Benchmark Results (Updated 2026-04-15, llama-bench) -**Method:** Same prompt ("Write a short poem about a cat."), `-ngl 99 --device SYCL0 -c 2048 -n 128 --reasoning off`, fresh build per phase. +**Method:** `llama-bench -ngl 99 -p 512 -n 128 -r 3`, 3 repeats. ### Qwen3.5-9B (dense, fits entirely in VRAM) -| Config | Q4_0 Gen t/s | Q4_0 Prompt t/s | Q8_0 Gen t/s | Q8_0 Prompt t/s | -|--------|-------------|-----------------|-------------|-----------------| -| Baseline | 29.4 | 17.6 | 28.6 | 20.7 | -| +Phase 1 (graph) | 29.7 | 20.0 | 29.0 | 20.7 | -| +Phase 2 (kernel) | 29.8 | 19.8 | 29.0 | 20.6 | -| +Phase 4 (host-copy) | 29.6 | 19.9 | 29.5 | 20.4 | +| Config | Q4_0 tg128 | Q8_0 tg128 | Q4_K_M tg128 | +|--------|-----------|-----------|-------------| +| Baseline (HEAD) | 29.4 | 28.6 | — | +| +Phase 2 (VER+tuning) | 30.16 | 29.96 | 24.65 | +| **+Phase 2+5 (vdr_mmvq)** | **35.96** | **30.82** | **25.32** | -**Analysis:** -- All results within ~1 t/s noise floor — no significant regression or improvement -- Phase 1 gives a modest prompt processing improvement (+2.4 t/s on Q4_0) -- The 9B dense model fits entirely in A770 VRAM, so sync/graph/reorder optimizations - don't exercise the bottleneck path these patches target +**Phase 5 is the first significant improvement: +19% on Q4_0 token generation.** -### llama-bench Results (Baseline, Unpatched) +### Bandwidth Utilization -| Model | pp512 | tg128 | -|-------|-------|-------| -| Qwen3.5-9B Q4_0 (5.0 GiB) | 723.39 ± 6.40 | 29.93 ± 0.59 | -| Qwen3.5-9B Q8_0 (8.86 GiB) | 702.46 ± 8.84 | 31.18 ± 0.11 | +| Config | Q4_0 BW | Q4_0 BW% | Q8_0 BW | Q8_0 BW% | +|--------|---------|----------|---------|----------| +| Baseline | 150 GiB/s | 29% | 265 GiB/s | 52% | +| +Phase 5 | 180 GiB/s | 35% | 273 GiB/s | 53% | -**Bandwidth Utilization:** -- Q4_0: 29.93 t/s ÷ (512 GB/s ÷ 5.0 GiB) = **29.2%** of theoretical max -- Q8_0: 31.18 t/s ÷ (512 GB/s ÷ 8.86 GiB) = **54.0%** of theoretical max -- Q8_0 achieves nearly 2x better utilization than Q4_0 — suggests different kernel paths +### Root Cause (corrected) -### Cross-Platform Comparison (from online research) +Previous analysis attributed low BW utilization to "SYCL submission model overhead". **This was WRONG.** Per-op profiling proves: -| GPU / Backend | Model | Quant | Gen t/s | BW Utilization | -|---------------|-------|-------|---------|----------------| -| Arc A770 SYCL (ours) | Qwen3.5-9B | Q4_0 | **30** | **29%** | -| Arc A770 SYCL (Intel CI) | llama-2-7B | Q4_0 | **42-55** | **30-39%** | -| Arc A770 Vulkan | Llama 3.1-8B | Q4_K_M | **42-54** | **47-75%** | -| RTX 3060 (CUDA) | Llama 3.1-8B | Q4_K_M | **52** | **~72%** | -| RTX 4060 (CUDA) | Llama 3.1-8B | Q4_K_M | **38** | **~65%** | +1. SYCL queue naturally batches async kernel submissions (CPU submits all 1077 ops in 7.5ms vs 32ms GPU execution) +2. Q4_0 is **dp4a-compute-bound**, not memory-BW-bound +3. Nibble packing requires 2 dp4a per byte (low + high nibbles), while Q8_0 needs only 1 dp4a per byte +4. Both formats hit the same dp4a throughput ceiling → same ~30 t/s despite different data sizes +5. Phase 5's vdr_mmvq increase processes more blocks per subgroup, better amortizing dp4a overhead -**Conclusion:** The SYCL backend fundamentally achieves only ~30% bandwidth utilization -vs ~53% on Vulkan and ~72% on CUDA. Our patches don't move the needle because the -bottleneck is in the SYCL submission model (1-op-at-a-time with OS-level .wait()), -not in the specific parameters we tuned. +See `../logs/root-cause-analysis-20260415.md` for full profiling data. ### Qwen3.5-35B-A3B (MoE, `--cpu-moe`) @@ -131,12 +118,24 @@ Stored in `../logs/` (gitignored). Key files: - `logs/K-kernel-tuning-*.md` — Agent-K kernel tuning analysis - `logs/M-review-K-*.md`, `logs/K-review-M-*.md` — Cross-reviews +### Phase 5 — Q4_0 MMVQ vdr_mmvq Tuning ✅ +19% Q4_0 +| Patch | File | Change | Status | +|-------|------|--------|--------| +| 0001 | quants.hpp:47 | Q4_0 reorder vdr_mmvq 2→4 | ✅ +5.8 t/s Q4_0 | + +Increases the "vector dot product rows per MMVQ" parameter for Q4_0's reorder path. +This processes more blocks per subgroup iteration (8→16 blocks), better amortizing +the dp4a compute overhead. Q4_0 is dp4a-bound (not BW-bound) because nibble +extraction requires 2 dp4a per byte vs Q8_0's 1 dp4a per byte. + +Result: Q4_0 tg128 goes from 30.16 → 35.96 t/s (+19%). No impact on Q8_0 or Q4_K_M. + ## Key Lesson -The Arc A770 bottleneck for token generation is NOT primarily in the areas we patched. -The 9B dense model achieves ~29 t/s gen which is close to theoretical bandwidth for -Q4_0 (~28-30 t/s expected for 2GB model at 512 GB/s bandwidth). The real performance -gap vs NVIDIA/AMD may be in: -1. MoE expert routing overhead (not exercised by dense models) -2. Memory bandwidth utilization during attention (not just matmul) -3. Driver/runtime overhead in the xe/Level Zero stack +The Arc A770 bottleneck for Q4_0 token generation is **dp4a compute throughput**, not +memory bandwidth or submission overhead. The 4-bit nibble packing requires 2 dp4a +operations per byte (low + high nibbles), making the kernel compute-bound at ~30 t/s. +Further improvements require: +1. DPAS/XMX integration for quantized dot products +2. Algorithmic changes to the nibble unpacking (e.g., lookup tables) +3. Larger vdr_mmvq (requires larger qi/QI4_0 — would need data format changes) diff --git a/repos/patch/phase5-mmvq-vdr/patch.diff b/repos/patch/phase5-mmvq-vdr/patch.diff new file mode 100644 index 0000000..655c6a4 --- /dev/null +++ b/repos/patch/phase5-mmvq-vdr/patch.diff @@ -0,0 +1,13 @@ +diff --git a/ggml/src/ggml-sycl/quants.hpp b/ggml/src/ggml-sycl/quants.hpp +index 1f5b62740..48c16b9ef 100644 +--- a/ggml/src/ggml-sycl/quants.hpp ++++ b/ggml/src/ggml-sycl/quants.hpp +@@ -44,7 +44,7 @@ template <> struct block_q_t { + static constexpr uint32_t qk = QK4_0; + static constexpr uint32_t qi = QI4_0; + static constexpr uint32_t qr = QR4_0; +- static constexpr uint32_t vdr_mmvq = 2; ++ static constexpr uint32_t vdr_mmvq = 4; // was 2: increased to process more blocks per subgroup, shift bottleneck from dp4a to BW + }; + + static constexpr std::pair get_block_offset(const int block_index, const int /* nblocks */) { diff --git a/repos/patch/phase5-op-timing/patch.diff b/repos/patch/phase5-op-timing/patch.diff new file mode 100644 index 0000000..d02e78b --- /dev/null +++ b/repos/patch/phase5-op-timing/patch.diff @@ -0,0 +1,69 @@ +diff --git a/ggml/src/ggml-sycl/ggml-sycl.cpp b/ggml/src/ggml-sycl/ggml-sycl.cpp +index 1234567..abcdefg 100644 +--- a/ggml/src/ggml-sycl/ggml-sycl.cpp ++++ b/ggml/src/ggml-sycl/ggml-sycl.cpp +@@ -4409,6 +4409,20 @@ static void ggml_backend_sycl_graph_compute_impl(ggml_backend_sycl_context * sycl_ctx, ggml_cgraph * cgraph) + static void ggml_backend_sycl_graph_compute_impl(ggml_backend_sycl_context * sycl_ctx, ggml_cgraph * cgraph) { + ggml_sycl_set_main_device(sycl_ctx->device); + ++ // Timing instrumentation ++ static int call_count = 0; ++ static bool timing_enabled = (getenv("GGML_SYCL_OP_TIMING") != nullptr); ++ auto t_start = std::chrono::high_resolution_clock::now(); ++ ++ struct op_stats { ++ const char* name; ++ int count; ++ double total_ms; ++ }; ++ op_stats stats[GGML_OP_COUNT] = {}; ++ ++ auto t_loop_start = std::chrono::high_resolution_clock::now(); ++ int compute_ops = 0; ++ + for (int i = 0; i < cgraph->n_nodes; i++) { + ggml_tensor * node = cgraph->nodes[i]; + if (ggml_is_empty(node) || node->op == GGML_OP_RESHAPE || node->op == GGML_OP_TRANSPOSE || node->op == GGML_OP_VIEW || node->op == GGML_OP_PERMUTE || node->op == GGML_OP_NONE) { +@@ -4420,10 +4434,28 @@ static void ggml_backend_sycl_graph_compute_impl(ggml_backend_sycl_context * syc + assert(node->src[j]->buffer->buft == ggml_backend_sycl_buffer_type(sycl_ctx->device)); + } + } ++ ++ auto t_op_start = std::chrono::high_resolution_clock::now(); ++ + bool ok = ggml_sycl_compute_forward(*sycl_ctx, node); + if (!ok) { + GGML_LOG_ERROR("%s: error: op not supported %s (%s)\n", __func__, node->name, ggml_op_name(node->op)); + } + GGML_ASSERT(ok); ++ ++ if (timing_enabled) { ++ // Flush the queue to get accurate per-op timing ++ sycl_ctx->stream()->wait(); ++ auto t_op_end = std::chrono::high_resolution_clock::now(); ++ double op_ms = std::chrono::duration(t_op_end - t_op_start).count(); ++ int op_idx = (int)node->op; ++ if (op_idx >= 0 && op_idx < GGML_OP_COUNT) { ++ stats[op_idx].name = ggml_op_name(node->op); ++ stats[op_idx].count++; ++ stats[op_idx].total_ms += op_ms; ++ } ++ compute_ops++; ++ } + } ++ ++ if (timing_enabled) { ++ auto t_loop_end = std::chrono::high_resolution_clock::now(); ++ double loop_ms = std::chrono::duration(t_loop_end - t_loop_start).count(); ++ call_count++; ++ if (call_count <= 5 || call_count % 100 == 0) { ++ fprintf(stderr, "[OP-TIMING] call=%d total_ops=%d loop=%.2fms\n", call_count, compute_ops, loop_ms); ++ for (int i = 0; i < GGML_OP_COUNT; i++) { ++ if (stats[i].count > 0) { ++ fprintf(stderr, " %-20s count=%3d total=%.2fms avg=%.3fms\n", ++ stats[i].name, stats[i].count, stats[i].total_ms, stats[i].total_ms/stats[i].count); ++ } ++ } ++ } ++ } + }