intel-gpu-llm-diagnosis/overview_kimi.md

# Intel Arc GPU LLM Inference: Driver & Software Stack Research Overview

**Date:** 2026-04-15  
**Scope:** Research-only phase. No code or driver modifications were made. This document collects online community discourse, bug reports, and architectural analysis to identify why Intel Arc GPUs underperform on quantized LLM inference and where the relevant software stacks are misaligned.

---

## 1. Executive Summary

Intel Arc GPUs (Alchemist / Battlemage) are mechanically capable LLM inference cards on paper—large VRAM pools (up to 24 GB), wide memory buses (456–608 GB/s), and dedicated XMX matrix engines. In practice, community reports consistently describe **severe performance cliffs on quantized models**, **backend-specific kernel inefficiencies**, and **driver/runtime instability** that prevent Arc from reaching the bandwidth utilization seen on NVIDIA/AMD hardware.

The core observation across Reddit, GitHub Issues, and Intel documentation is that **token generation (TG) on quantized GGUF models often achieves only 20–40% of theoretical memory bandwidth**, while prompt processing (PP) can swing wildly depending on which backend (SYCL, Vulkan, OpenVINO, IPEX-LLM) is used. The problem is **not a single driver bug**; it is a misalignment between:

- **Kernel implementations** in `llama.cpp`’s SYCL/Vulkan backends that were optimized for NVIDIA/AMD data layouts and not fully ported for Intel Xe architecture.
- **The Intel graphics runtime stack** (Compute Runtime / Level Zero / IGC) which exposes the hardware but relies on user-space kernels to extract performance.
- **Rapid ecosystem churn**: IPEX-LLM was archived in Jan 2026, PyTorch XPU support moved upstream, and vLLM is mid-migration from IPEX to a new `vllm-xpu-kernels` backend, leaving users with fragmented, often conflicting setup instructions.

---

## 2. The Driver & Runtime Stack

For LLM inference on Intel Arc, the following layers are involved:

| Layer | Project / Driver | Role |
|-------|------------------|------|
| **Kernel driver** | `i915` / `xe` (Linux), Intel Graphics Driver (Windows) | Base GPU scheduling, memory management |
| **Compute Runtime** | `intel/compute-runtime` (NEO) | OpenCL + Level Zero driver; exposes SYCL devices |
| **Graphics Compiler** | Intel Graphics Compiler (IGC) | JIT-compiles SYCL kernels to Xe ISA |
| **Math libs** | oneMKL + oneDNN | GEMM/SDPA backends for PyTorch/SYCL |
| **Mesa Vulkan** | `anv` (Intel Vulkan driver) | Backend for `llama.cpp` Vulkan path |
| **Framework integrations** | `ipex-llm` (archived), `intel-extension-for-pytorch`, `vllm-xpu-kernels`, upstream `vllm` | User-facing inference stacks |

### Key Repositories Pulled

```
repos/
├── llama.cpp                      # SYCL & Vulkan backends, GGUF quantization kernels
├── ipex-llm                       # Intel’s former integration layer (archived Jan 2026)
├── intel-extension-for-pytorch    # PyTorch XPU extension (also archived / deprecated)
├── compute-runtime                # Intel Level Zero / OpenCL driver (NEO)
├── oneDNN                         # Intel deep-learning primitive library
├── vllm                           # vLLM mainline (XPU backend in flux)
└── vllm-xpu-kernels               # New dedicated Intel kernel repo for vLLM
```

---

## 3. Problems Identified in Community Discourse

### 3.1 Quantized Model Performance Is Disproportionately Bad

The most repeated complaint is that **quantized models run far slower than they should** given their reduced size.

**Example: `llama.cpp` SYCL backend on Arc Pro B70 (Xe2 / Battlemage)**  
*(GitHub Issue #21517, Reddit r/LocalLLAMA)*

| Quant | Model Size | TG (t/s) | Effective BW | % of Peak |
|-------|------------|----------|--------------|-----------|
| Q4_K_M | 15.6 GiB | 20.56 | 321 GB/s | 53% |
| Q8_0 | 26.6 GiB | 4.88 | 130 GB/s | **21%** |

**Critical finding:** Q8_0 is **4× slower** than Q4_K_M despite moving only **1.7× more bytes**. This rules out pure memory-bandwidth limits and points to **kernel-level inefficiency**.

The same issue affects both SYCL and Vulkan backends equally, and persists when splitting across two GPUs with abundant free VRAM. Updating the Compute Runtime / IGC had **no effect** on Q8_0 token-generation speed, confirming the bottleneck is in the inference-framework kernels, not the compiler or driver.

#### Inverse anomaly on older Arc A770 (Xe1 / Alchemist)
On Vulkan, some users report the *opposite*: Q8_0 **outperforms** Q4/Q6 in prompt processing (Issue #19887: ~600 t/s for Q8_0 vs ~200 t/s for Q6_K). This suggests the quantization-kernel imbalance is **architecture-specific**, not universal.

### 3.2 Missing or Partial "Reorder" Optimizations

`llama.cpp`’s SYCL backend introduced a **reorder optimization** (PR #12035, Feb 2025) that separates quantized weights from their scale factors so the GPU can load them with coalesced memory access. This optimization was implemented **only for Q4_0** and later extended to Q4_K and Q6_K. **Q8_0 was never added** to the reorder/MMVQ fast path.

Consequences:
- Q8_0 falls back to the generic **DMMV** kernel with `iter_stride = 64` (2 values per thread per iteration).
- The reorder path for Q4_0 uses `iter_stride = 512` (16 values per thread per iteration).
- A forced DMMV path for Q4_K_M drops its TG speed by ~40%, but forcing MMVQ on Q8_0 does **not** recover performance—both paths are slow for Q8_0 on Xe2.

Community speculation: the 34-byte `block_q8_0` layout is not a power-of-two, making it harder to vectorize efficiently on Intel’s EU/SIMD width without explicit data-layout rewriting.

### 3.3 XMX Matrix Engines Are Underutilized for Token Generation

Intel Arc has **Xe Matrix Extensions (XMX)**—analogous to NVIDIA Tensor Cores. They are used automatically by oneMKL/oneDNN for **FP16/BF16 GEMM**, which is why prompt processing (compute-bound) sees large speedups when `-DGGML_SYCL_F16=ON` is enabled (reported ~2.4× improvement: 302 → 725 t/s).

However, **token generation uses DMMV/MMVQ**, which are memory-bandwidth-bound dequantize-and-dot kernels. The SYCL backend does **not** currently route these small quantized matrix-vector operations through XMX/DPAS instructions. Developers (Codeplay/Intel contributors in Discussion #12570) note they are investigating DPAS directly, but consider the kernel "memory bound" in current configurations, so adding matrix engines may not help until memory-access patterns are fixed first.

### 3.4 Vulkan Backend Regressions and Driver Sensitivity

The Vulkan path in `llama.cpp` is maintained independently and sees recurring Intel-specific regressions:

- **Mesa version sensitivity**: Newer Mesa versions have reportedly **slowed TG on Intel** (Discussion #10879).
- **Performance degradation between builds**: A drop from 53 → 42 t/s on A770 was bisected to changes between `b7189` and `b7209` (Issue #17628).
- **Cooperative matrix TDRs**: Windows drivers (101.8509/101.8531) cause GPU timeouts when `VK_KHR_cooperative_matrix` is enabled with `llama.cpp` Vulkan (Issue #20554).
- **BMG (Battlemage) support lag**: B580 GPUs require very recent Mesa + kernel combinations; users often see fallback-to-CPU behavior or "unsupported device" warnings (Discussion #12570).

### 3.5 IPEX-LLM Deprecation and Ecosystem Fragmentation

`intel/ipex-llm` was **archived on January 28, 2026**. It was the primary documented way to run Ollama/llama.cpp on Intel Arc via Docker. Since archiving:

- Open issues (prompt-processing slowdowns, container GPU visibility, B580 model-load failures) are frozen.
- Users are left choosing between:
  1. **Upstream `llama.cpp` SYCL** (manual oneAPI setup, variable quant performance).
  2. **OpenVINO** (Intel-recommended, but not GGUF-native).
  3. **vLLM + IPEX** (deprecated; vLLM is migrating to `vllm-xpu-kernels`).
  4. **Vulkan `llama.cpp`** (easier setup, but lower peak performance and regressions).

### 3.6 Compute Runtime / Level Zero Issues

The `intel/compute-runtime` repository contains long-standing issues that affect LLM workloads:

- **4 GB allocation limit** on Arc A770 16 GB (`CL_DEVICE_MAX_MEM_ALLOC_SIZE`) — Issue #627. Large single buffers must be split or use host paging.
- **Incorrect free-memory reporting** — Issue #750. Runtime reports `free_memory == global_mem_size` even when VRAM is in use, confusing memory managers in `llama.cpp` and vLLM.
- **BAR / SVM allocation failures on Arrow Lake** — Issue #890. OEM laptops with fixed 256 MB BAR hang or fail to allocate >4 GB, verified on both Linux and Windows.
- **iGPU + dGPU conflicts**: SYCL initialization can fail or select the wrong device when both an Intel iGPU and Arc dGPU are present (Issues #13775, #9106).

### 3.7 vLLM on Intel XPU Is Still Immature

vLLM upstream added Intel XPU support, but user reports highlight:

- **B-series crashes during model inspection** (Issue #27408): `SIGABRT` inside `drm_neo.cpp` on Battlemage.
- **Dual-GPU scaling breaks TG**: Two A770s in tensor/pipeline parallel double throughput but **halve text-generation speed** to 3–4 t/s (Issue #12190).
- **AWQ/INT4 pre-quantized models fail** because the torchao codepath is CUDA-only (Issue #269 in `intel/llm-scaler`).
- **IPEX deprecation**: vLLM release notes explicitly "deprecated IPEX for XPU, switched to vllm-xpu-kernels" (v0.11.x+), but the new kernel repo is still catching up on feature parity.

---

## 4. Speculations on Root Causes

Based on the collected discourse, the following hypotheses best explain the observed gaps:

### A. Kernel Data Layouts Are CUDA-Centric
`llama.cpp`’s quantized kernels (DMMV, MMVQ) were originally written and tuned for NVIDIA warp sizes, shared-memory banking, and coalesced-load patterns. The Intel Xe architecture has **different SIMD widths, cache-line behavior, and scatter-gather characteristics**. The "reorder" fix for Q4_0 proves that **rewriting the data layout specifically for Intel** yields large gains, but this work was not systematically extended to all quant types.

### B. SYCL Is a Thick Abstraction with Opaque Performance Characteristics
Developers (both community and Intel-affiliated) note that the SYCL stack (DPC++ compiler → IGC → Level Zero) works, but makes it **difficult to reason about whether generated ISA actually uses XMX/DPAS or falls back to scalar ALU paths**. The IGC JIT compiler does auto-vectorization, but for non-power-of-two block sizes (e.g., Q8_0’s 34 bytes), the generated code may serialize loads and leave EUs idle.

### C. The Driver Stack Correctly Exposes Hardware, But Does Not Hide Its Quirks
Intel Compute Runtime is functionally correct—models load, kernels execute, and results are valid—but it does **not** provide the same transparent memory-management or profiling feedback that CUDA/Rocm drivers offer. Issues like the 4 GB `MAX_MEM_ALLOC_SIZE`, wrong free-memory reporting, and iGPU+dGPU device-selection bugs force **application-layer workarounds** that are inconsistently implemented across frameworks.

### D. Rapid Corporate Strategy Shifts Create Maintenance Debt
The shift from **BigDL-LLM → IPEX-LLM → archived IPEX-LLM**, and the parallel shift from **IPEX → native PyTorch XPU → vllm-xpu-kernels**, means optimizations and bug fixes were repeatedly abandoned mid-stream. Community contributors (e.g., Codeplay, private maintainers in Discussion #12570) are left carrying the SYCL backend in `llama.cpp` with limited resources.

---

## 5. Potential Solutions & Lines of Investigation

**No code changes were made in this phase.** The following are research-backed proposals for a subsequent implementation phase.

### 5.1 Complete the Reorder Optimization for Q8_0 and K-Quants in `ggml-sycl`
- Implement `dequantize_block_q8_0_reorder` and add Q8_0 to `ggml_sycl_supports_reorder_mmvq()`.
- Increase `iter_stride` for Q8_0 DMMV to match the 8× factor used in Q4_0 reorder kernels.
- **Expected impact:** Close the 4× TG gap between Q8_0 and Q4_K_M on Xe2 GPUs.

### 5.2 Profile SYCL Kernels with Intel-Specific Tools
- Use **Intel VTune** or **ze_tracer** on the slow Q8_0 DMMV kernel to measure:
  - EU utilization
  - L3 cache miss rates
  - Memory-latency hiding efficiency
- **Goal:** Determine whether the bottleneck is load coalescing, register pressure, or insufficient occupancy.

### 5.3 Implement DPAS/XMX Kernels for Quantized Matrix-Vector Multiplication
- Target **DPAS/DPASW** instructions directly (or via lightweight SYCL extensions) for INT8/INT4 dot products in MMVQ.
- oneDNN/oneMKL already use XMX for FP16 GEMM; the gap is in the **custom quantized DMMV/MMVQ kernels** inside `llama.cpp`.

### 5.4 Audit and Fix Intel Compute Runtime Memory Reporting
- Patch `compute-runtime` to return accurate `free_memory` via Level Zero (`zesMemoryGetState`) so that `llama.cpp` and vLLM can make correct offloading decisions without the `ZES_ENABLE_SYSMAN=1` workaround.
- Investigate the 4 GB `CL_DEVICE_MAX_MEM_ALLOC_SIZE` cap on 16 GB cards.

### 5.5 Stabilize the Vulkan Backend for Xe2
- Add BMG/B580 device IDs and tune pipeline layouts in `ggml-vulkan`.
- Bisect and revert or adapt the Mesa change that caused TG regression on Intel Arc.

### 5.6 Align vLLM XPU Quantization Roadmap
- Ensure `vllm-xpu-kernels` supports the same GGUF/AWQ/GPTQ paths that CUDA does, or at minimum documents unsupported formats clearly.
- Fix the B-series model-inspection crash (`drm_neo.cpp` abort) before broader Battlemage deployment.

---

## 6. Key Data Sources & References

| Source | Significance |
|--------|--------------|
| `ggml-org/llama.cpp#21517` | Rigorous Q8_0 bandwidth analysis on Arc Pro B70; proves kernel inefficiency |
| `ggml-org/llama.cpp#12035` | Original Q4_0 reorder optimization; template for fixing Q8_0 |
| `ggml-org/llama.cpp#12570` | Maintainer discussion on SYCL vs Vulkan, XMX, DPAS, and Codeplay roadmap |
| `ggml-org/llama.cpp#19887` | Inverse quant anomaly on A770 Vulkan (Q8_0 faster than Q4/Q6) |
| `intel/ipex-llm#12632` | Prompt-processing slowdown on Arc A380 in Docker (frozen, repo archived) |
| `intel/ipex-llm#12994` | B580 model load failures with Level Zero |
| `intel/compute-runtime#627` | 4 GB allocation limit on Arc A770 16 GB |
| `intel/compute-runtime#890` | Arrow Lake >4 GB allocation failures / system hangs |
| `vllm-project/vllm#27408` | B-series SIGABRT during model inspection |
| `vllm-project/vllm#33214` | Migration from IPEX to `vllm-xpu-kernels` |
| Reddit r/LocalLLAMA (2025) | User-reported bandwidth math, “3.1× Q8_0 speedup” thread |
| Reddit r/IntelArc | Benchmark comparisons: Vulkan vs SYCL, Ollama setup guides |

---

## 7. Conclusion

Intel Arc GPUs are **not fundamentally broken** for LLM inference, but they are **victims of incomplete kernel optimization and ecosystem churn**. The hardware has competitive bandwidth and matrix engines, yet the dominant open-source inference path (`llama.cpp` SYCL) only optimized Q4_0’s data layout, leaving Q8_0 and several K-quants on a slow generic fallback. Simultaneously, Intel’s own integration layer (`ipex-llm`) was archived while downstream projects (vLLM, PyTorch) are mid-transition to new backends.

The highest-leverage fixes, in order, appear to be:
1. **Kernel-side:** Extend the reorder/MMVD/MMVQ optimizations to Q8_0 and remaining quants in `ggml-sycl`.
2. **Driver-side:** Fix memory reporting and large-buffer allocation limits in Compute Runtime.
3. **Ecosystem-side:** Stabilize `vllm-xpu-kernels` and document a single, maintained path for Arc users.

All relevant repositories have been pulled locally for the next phase of analysis.