sleepy/intel-gpu-llm-diagnosis
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Phase 5: Q4_0 vdr_mmvq 2→4 (+19% tg128, 30→36 t/s)

Browse Source 
Root cause analysis corrected: Q4_0 low BW utilization is NOT due to
SYCL submission model overhead. Per-op profiling proves the bottleneck
is dp4a compute throughput — nibble extraction requires 2 dp4a per byte
vs Q8_0's 1 dp4a per byte, making Q4_0 compute-bound at ~30 t/s.

Fix: Increase vdr_mmvq from 2 to 4 for Q4_0 reorder path, processing
16 blocks per subgroup instead of 8. Better amortizes dp4a overhead.

Benchmark (Qwen3.5-9B, Arc A770, llama-bench -p 512 -n 128 -r 3):
  Q4_0:  30.16 → 35.96 t/s (+19%)
  Q8_0:  29.96 → 30.82 t/s (within noise)
  Q4_K_M: 24.65 → 25.32 t/s (within noise)

Also includes:
- Timing instrumentation patch (for debugging, not applied to source)
- Updated decisions log (Decisions 8-9)
- Updated workplan with revised benchmark data
- Root cause analysis document
This commit is contained in:
Pi Agent
2026-04-15 19:55:44 +03:00
parent f3caef26f7
commit 48bbb14a64
4 changed files with 123 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files
repos/llama.cpp
+1 -1
Submodule repos/llama.cpp updated: 8dc530b86d...80d8770804
repos/patch/README.md
+40 -41
View File
@@ -28,51 +28,38 @@ cd repos
- **oneAPI:** 2025.3.2, DPC++ compiler 2025.3.2 - **oneAPI:** 2025.3.2, DPC++ compiler 2025.3.2
- **Build:** `-DGGML_SYCL=ON -DGGML_VULKAN=OFF -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DCMAKE_BUILD_TYPE=Release` - **Build:** `-DGGML_SYCL=ON -DGGML_VULKAN=OFF -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DCMAKE_BUILD_TYPE=Release`
## Benchmark Results (Clean Run, 2026-04-15) ## Benchmark Results (Updated 2026-04-15, llama-bench)
**Method:** Same prompt ("Write a short poem about a cat."), `-ngl 99 --device SYCL0 -c 2048 -n 128 --reasoning off`, fresh build per phase. **Method:** `llama-bench -ngl 99 -p 512 -n 128 -r 3`, 3 repeats.
### Qwen3.5-9B (dense, fits entirely in VRAM) ### Qwen3.5-9B (dense, fits entirely in VRAM)
| Config | Q4_0 Gen t/s | Q4_0 Prompt t/s | Q8_0 Gen t/s | Q8_0 Prompt t/s | | Config | Q4_0 tg128 | Q8_0 tg128 | Q4_K_M tg128 |
|--------|-------------|-----------------|-------------|-----------------| |--------|-----------|-----------|-------------|
| Baseline | 29.4 | 17.6 | 28.6 | 20.7 | | Baseline (HEAD) | 29.4 | 28.6 | — |
| +Phase 1 (graph) | 29.7 | 20.0 | 29.0 | 20.7 | | +Phase 2 (VER+tuning) | 30.16 | 29.96 | 24.65 |
| +Phase 2 (kernel) | 29.8 | 19.8 | 29.0 | 20.6 | | **+Phase 2+5 (vdr_mmvq)** | **35.96** | **30.82** | **25.32** |
| +Phase 4 (host-copy) | 29.6 | 19.9 | 29.5 | 20.4 |
**Analysis:** **Phase 5 is the first significant improvement: +19% on Q4_0 token generation.**
- All results within ~1 t/s noise floor — no significant regression or improvement
- Phase 1 gives a modest prompt processing improvement (+2.4 t/s on Q4_0)
- The 9B dense model fits entirely in A770 VRAM, so sync/graph/reorder optimizations
don't exercise the bottleneck path these patches target
### llama-bench Results (Baseline, Unpatched) ### Bandwidth Utilization
| Model | pp512 | tg128 | | Config | Q4_0 BW | Q4_0 BW% | Q8_0 BW | Q8_0 BW% |
|-------|-------|-------| |--------|---------|----------|---------|----------|
| Qwen3.5-9B Q4_0 (5.0 GiB) | 723.39 ± 6.40 | 29.93 ± 0.59 | | Baseline | 150 GiB/s | 29% | 265 GiB/s | 52% |
| Qwen3.5-9B Q8_0 (8.86 GiB) | 702.46 ± 8.84 | 31.18 ± 0.11 | | +Phase 5 | 180 GiB/s | 35% | 273 GiB/s | 53% |
**Bandwidth Utilization:** ### Root Cause (corrected)
- Q4_0: 29.93 t/s ÷ (512 GB/s ÷ 5.0 GiB) = **29.2%** of theoretical max
- Q8_0: 31.18 t/s ÷ (512 GB/s ÷ 8.86 GiB) = **54.0%** of theoretical max
- Q8_0 achieves nearly 2x better utilization than Q4_0 — suggests different kernel paths
### Cross-Platform Comparison (from online research) Previous analysis attributed low BW utilization to "SYCL submission model overhead". **This was WRONG.** Per-op profiling proves:
| GPU / Backend | Model | Quant | Gen t/s | BW Utilization | 1. SYCL queue naturally batches async kernel submissions (CPU submits all 1077 ops in 7.5ms vs 32ms GPU execution)
|---------------|-------|-------|---------|----------------| 2. Q4_0 is **dp4a-compute-bound**, not memory-BW-bound
| Arc A770 SYCL (ours) | Qwen3.5-9B | Q4_0 | **30** | **29%** | 3. Nibble packing requires 2 dp4a per byte (low + high nibbles), while Q8_0 needs only 1 dp4a per byte
| Arc A770 SYCL (Intel CI) | llama-2-7B | Q4_0 | **42-55** | **30-39%** | 4. Both formats hit the same dp4a throughput ceiling → same ~30 t/s despite different data sizes
| Arc A770 Vulkan | Llama 3.1-8B | Q4_K_M | **42-54** | **47-75%** | 5. Phase 5's vdr_mmvq increase processes more blocks per subgroup, better amortizing dp4a overhead
| RTX 3060 (CUDA) | Llama 3.1-8B | Q4_K_M | **52** | **~72%** |
| RTX 4060 (CUDA) | Llama 3.1-8B | Q4_K_M | **38** | **~65%** |
**Conclusion:** The SYCL backend fundamentally achieves only ~30% bandwidth utilization See `../logs/root-cause-analysis-20260415.md` for full profiling data.
vs ~53% on Vulkan and ~72% on CUDA. Our patches don't move the needle because the
bottleneck is in the SYCL submission model (1-op-at-a-time with OS-level .wait()),
not in the specific parameters we tuned.
### Qwen3.5-35B-A3B (MoE, `--cpu-moe`) ### Qwen3.5-35B-A3B (MoE, `--cpu-moe`)
@@ -131,12 +118,24 @@ Stored in `../logs/` (gitignored). Key files:
- `logs/K-kernel-tuning-*.md` — Agent-K kernel tuning analysis - `logs/K-kernel-tuning-*.md` — Agent-K kernel tuning analysis
- `logs/M-review-K-*.md`, `logs/K-review-M-*.md` — Cross-reviews - `logs/M-review-K-*.md`, `logs/K-review-M-*.md` — Cross-reviews
### Phase 5 — Q4_0 MMVQ vdr_mmvq Tuning ✅ +19% Q4_0
| Patch | File | Change | Status |
|-------|------|--------|--------|
| 0001 | quants.hpp:47 | Q4_0 reorder vdr_mmvq 2→4 | ✅ +5.8 t/s Q4_0 |
Increases the "vector dot product rows per MMVQ" parameter for Q4_0's reorder path.
This processes more blocks per subgroup iteration (8→16 blocks), better amortizing
the dp4a compute overhead. Q4_0 is dp4a-bound (not BW-bound) because nibble
extraction requires 2 dp4a per byte vs Q8_0's 1 dp4a per byte.
Result: Q4_0 tg128 goes from 30.16 → 35.96 t/s (+19%). No impact on Q8_0 or Q4_K_M.
## Key Lesson ## Key Lesson
The Arc A770 bottleneck for token generation is NOT primarily in the areas we patched. The Arc A770 bottleneck for Q4_0 token generation is **dp4a compute throughput**, not
The 9B dense model achieves ~29 t/s gen which is close to theoretical bandwidth for memory bandwidth or submission overhead. The 4-bit nibble packing requires 2 dp4a
Q4_0 (~28-30 t/s expected for 2GB model at 512 GB/s bandwidth). The real performance operations per byte (low + high nibbles), making the kernel compute-bound at ~30 t/s.
gap vs NVIDIA/AMD may be in: Further improvements require:
1. MoE expert routing overhead (not exercised by dense models) 1. DPAS/XMX integration for quantized dot products
2. Memory bandwidth utilization during attention (not just matmul) 2. Algorithmic changes to the nibble unpacking (e.g., lookup tables)
3. Driver/runtime overhead in the xe/Level Zero stack 3. Larger vdr_mmvq (requires larger qi/QI4_0 — would need data format changes)
repos/patch/phase5-mmvq-vdr/patch.diff
+13
View File
@@ -0,0 +1,13 @@
diff --git a/ggml/src/ggml-sycl/quants.hpp b/ggml/src/ggml-sycl/quants.hpp
index 1f5b62740..48c16b9ef 100644
--- a/ggml/src/ggml-sycl/quants.hpp
+++ b/ggml/src/ggml-sycl/quants.hpp
@@ -44,7 +44,7 @@ template <> struct block_q_t<GGML_TYPE_Q4_0> {
static constexpr uint32_t qk = QK4_0;
static constexpr uint32_t qi = QI4_0;
static constexpr uint32_t qr = QR4_0;
- static constexpr uint32_t vdr_mmvq = 2;
+ static constexpr uint32_t vdr_mmvq = 4; // was 2: increased to process more blocks per subgroup, shift bottleneck from dp4a to BW
};
static constexpr std::pair<int, int> get_block_offset(const int block_index, const int /* nblocks */) {
repos/patch/phase5-op-timing/patch.diff
+69
View File
@@ -0,0 +1,69 @@
diff --git a/ggml/src/ggml-sycl/ggml-sycl.cpp b/ggml/src/ggml-sycl/ggml-sycl.cpp
index 1234567..abcdefg 100644
--- a/ggml/src/ggml-sycl/ggml-sycl.cpp
+++ b/ggml/src/ggml-sycl/ggml-sycl.cpp
@@ -4409,6 +4409,20 @@ static void ggml_backend_sycl_graph_compute_impl(ggml_backend_sycl_context * sycl_ctx, ggml_cgraph * cgraph)
static void ggml_backend_sycl_graph_compute_impl(ggml_backend_sycl_context * sycl_ctx, ggml_cgraph * cgraph) {
ggml_sycl_set_main_device(sycl_ctx->device);
+ // Timing instrumentation
+ static int call_count = 0;
+ static bool timing_enabled = (getenv("GGML_SYCL_OP_TIMING") != nullptr);
+ auto t_start = std::chrono::high_resolution_clock::now();
+
+ struct op_stats {
+ const char* name;
+ int count;
+ double total_ms;
+ };
+ op_stats stats[GGML_OP_COUNT] = {};
+
+ auto t_loop_start = std::chrono::high_resolution_clock::now();
+ int compute_ops = 0;
+
for (int i = 0; i < cgraph->n_nodes; i++) {
ggml_tensor * node = cgraph->nodes[i];
if (ggml_is_empty(node) || node->op == GGML_OP_RESHAPE || node->op == GGML_OP_TRANSPOSE || node->op == GGML_OP_VIEW || node->op == GGML_OP_PERMUTE || node->op == GGML_OP_NONE) {
@@ -4420,10 +4434,28 @@ static void ggml_backend_sycl_graph_compute_impl(ggml_backend_sycl_context * syc
assert(node->src[j]->buffer->buft == ggml_backend_sycl_buffer_type(sycl_ctx->device));
}
}
+
+ auto t_op_start = std::chrono::high_resolution_clock::now();
+
bool ok = ggml_sycl_compute_forward(*sycl_ctx, node);
if (!ok) {
GGML_LOG_ERROR("%s: error: op not supported %s (%s)\n", __func__, node->name, ggml_op_name(node->op));
}
GGML_ASSERT(ok);
+
+ if (timing_enabled) {
+ // Flush the queue to get accurate per-op timing
+ sycl_ctx->stream()->wait();
+ auto t_op_end = std::chrono::high_resolution_clock::now();
+ double op_ms = std::chrono::duration<double, std::milli>(t_op_end - t_op_start).count();
+ int op_idx = (int)node->op;
+ if (op_idx >= 0 && op_idx < GGML_OP_COUNT) {
+ stats[op_idx].name = ggml_op_name(node->op);
+ stats[op_idx].count++;
+ stats[op_idx].total_ms += op_ms;
+ }
+ compute_ops++;
+ }
}
+
+ if (timing_enabled) {
+ auto t_loop_end = std::chrono::high_resolution_clock::now();
+ double loop_ms = std::chrono::duration<double, std::milli>(t_loop_end - t_loop_start).count();
+ call_count++;
+ if (call_count <= 5 || call_count % 100 == 0) {
+ fprintf(stderr, "[OP-TIMING] call=%d total_ops=%d loop=%.2fms\n", call_count, compute_ops, loop_ms);
+ for (int i = 0; i < GGML_OP_COUNT; i++) {
+ if (stats[i].count > 0) {
+ fprintf(stderr, " %-20s count=%3d total=%.2fms avg=%.3fms\n",
+ stats[i].name, stats[i].count, stats[i].total_ms, stats[i].total_ms/stats[i].count);
+ }
+ }
+ }
+ }
}