[docs] add coherence tests, MLX benchmarking, onboarding, Gitea API

GIT.md

@@ -52,9 +52,59 @@ Issue labels:
 2. Make changes, commit with `[area] description` conventions (see below)
 3. Push branch: `git push gitea feature/<name>`
 4. Create PR on Gitea targeting `master`
-5. Before merge: build, benchmark (record in BENCHMARKS.md), perplexity check if kernel changed
+5. Before merge: build, benchmark (record in BENCHMARKS.md), perplexity check if kernel changed, **coherence test** (see below)
 6. Squash-merge to master
 
+## Pre-Merge Coherence Tests
+
+**Mandatory before any merge to master.** Run on both `master` and the PR branch to detect silent correctness regressions.
+
+> **IMPORTANT:** macOS has no `timeout` command. Use `gtimeout` (from `brew install coreutils`).
+
+### Quick Coherence Test (4B model, ~30s)
+
+```bash
+gtimeout 60 ./build-build/bin/llama-cli \
+  -m ~/.llama/models/Qwen3.5-4B-Q4_0.gguf \
+  -n 64 -p "Once upon a time" \
+  --temp 0 -s 42 -st
+```
+
+### Perplexity Check (kernel changes only)
+
+```bash
+gtimeout 120 ./build-build/bin/llama-perplexity \
+  -m ~/.llama/models/Qwen3.5-4B-Q4_0.gguf \
+  -f /tmp/coherence_test.txt -t 1 --chunks 1 -c 128
+```
+
+### Verification
+
+- **llama-cli**: PR output must be coherent speech (not gibberish). Does not need to be bit-perfect vs master.
+- **perplexity**: PR perplexity must match master within floating-point tolerance (<0.1% delta)
+- **Gibberish output = block merge.** Re-dispatch with specific feedback.
+
+### Timeout Policy
+
+**All test commands MUST use `gtimeout`** to prevent hangs:
+- Inference/cli: `gtimeout 60` (60s)
+- Perplexity: `gtimeout 120` (120s)
+- Benchmark: `gtimeout 300` (5min)
+- Build: `gtimeout 300` (5min)
+
+A hung test is a test failure. Do not retry without investigating the hang.
+
+### IMPORTANT: llama-cli interactive mode
+
+llama-cli enters interactive REPL after generating, flooding output with `>` prompts. This is NOT a correctness failure — it's the CLI waiting for input.
+**Always use `--single-turn` (`-st`) flag to prevent this:**
+
+```bash
+gtimeout 60 ./build-build/bin/llama-cli -m ~/.llama/models/Qwen3.5-4B-Q4_0.gguf -n 64 -p "Once upon a time" --temp 0 -s 42 -st
+```
+
+Without `-st`, you will see `>` garbage and the process will hang. DO NOT attempt to "fix" the kernel because of this.
+
 ## Commit Messages
 
 Format: `[area] short description (max 72 chars)`
@@ -77,15 +127,23 @@ When working autonomously, agents MUST:
 2. **Create a branch** for any code change: `feature/<issue-number>-<short-desc>`
 3. **Reference the issue** in commits: `[area] description (#123)`
 4. **Run benchmarks** before/after kernel changes and record in BENCHMARKS.md
-5. **Run perplexity** to verify correctness after any kernel change:
+5. **Run perplexity** to verify correctness after any kernel change (with timeout):
-   ```bash
+    ```bash
-   ./build-build/bin/llama-perplexity -m MODEL.gguf -f /tmp/coherence_test.txt -t 1 --chunks 1 -c 128
+    gtimeout 120 ./build-build/bin/llama-perplexity -m MODEL.gguf -f /tmp/coherence_test.txt -t 1 --chunks 1 -c 128
-   ```
+    ```
-6. **Build succeeds** before pushing:
+6. **Run coherence test** before any merge (with timeout):
-   ```bash
+    ```bash
-   cmake --build build-build -j$(sysctl -n hw.ncpu)
+    gtimeout 60 ./build-build/bin/llama-cli -m ~/.llama/models/Qwen3.5-4B-Q4_0.gguf -n 64 -p "Once upon a time" --temp 0 -s 42 -st
-   ```
+    ```
-7. **Push branch** to gitea, then **create PR via Gitea API** (not via git push)
+    Output must be coherent speech (not gibberish).
+7. **Build succeeds** before pushing (with timeout):
+    ```bash
+    gtimeout 300 cmake --build build-build -j$(sysctl -n hw.ncpu)
+    ```
+8. **Push branch** to gitea, then **create PR via Gitea API** (not via git push)
+
+> **NOTE:** macOS has no `timeout` command. Always use `gtimeout` (from `brew install coreutils`).
+> **NOTE:** Always use `-st` flag with llama-cli to prevent interactive mode `>` prompts.
 
 ## Build
 
@@ -113,6 +171,20 @@ cmake --build build-build --target llama-eval-callback-profile -j$(sysctl -n hw.
 ./build-build/bin/llama-perplexity -m MODEL.gguf -f /tmp/coherence_test.txt -t 1 --chunks 1 -c 128
 ```
 
+## MLX Benchmarking
+
+MLX-lm is the performance target. Models at `~/.omlx/models/`.
+
+```bash
+# Quick generation test
+mlx_lm.generate --model ~/.omlx/models/Qwen3.6-27B-Q4_0 --prompt "Once upon a time" --max-tokens 128
+
+# Benchmark with timing
+time mlx_lm.generate --model ~/.omlx/models/Qwen3.6-27B-Q4_0 --prompt "Once upon a time" --max-tokens 4096
+```
+
+Compare llama.cpp results against MLX baselines. Record in BENCHMARKS.md.
+
 ## Profiling Tools
 
 | Tool | What it does |
@@ -152,19 +224,47 @@ ANALYSIS_QWEN3_5_MXFP4.md                     — MXFP4 format analysis
 ## Gitea API
 
 Base: `https://git.kokoham.com/api/v1`
-Token in `~/.gitea_token` (not committed).
+Token in `~/Documents/personal/projects/.env` as `GITEA_TOKEN`.
-Local API from server: `http://127.0.0.1:18431/api/v1`
 
 ```bash
+export $(grep -v '^#' ~/Documents/personal/projects/.env | xargs)
+
 # Create issue
-curl -X POST "http://127.0.0.1:18431/api/v1/repos/sleepy/llama.cpp/issues" \
+curl -X POST "https://git.kokoham.com/api/v1/repos/sleepy/llama.cpp/issues" \
-  -H "Authorization: token $(cat ~/.gitea_token)" \
+  -H "Authorization: token $GITEA_TOKEN" \
   -H "Content-Type: application/json" \
   -d '{"title":"...","body":"...","labels":["perf"]}'
 
 # Create PR
-curl -X POST "http://127.0.0.1:18431/api/v1/repos/sleepy/llama.cpp/pulls" \
+curl -X POST "https://git.kokoham.com/api/v1/repos/sleepy/llama.cpp/pulls" \
-  -H "Authorization: token $(cat ~/.gitea_token)" \
+  -H "Authorization: token $GITEA_TOKEN" \
   -H "Content-Type: application/json" \
   -d '{"title":"...","body":"...","head":"feature/xyz","base":"master"}'
+
+# Comment on issue/PR
+curl -X POST "https://git.kokoham.com/api/v1/repos/sleepy/llama.cpp/issues/{number}/comments" \
+  -H "Authorization: token $GITEA_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{"body":"..."}'
+
+# Close issue
+curl -X PATCH "https://git.kokoham.com/api/v1/repos/sleepy/llama.cpp/issues/{number}" \
+  -H "Authorization: token $GITEA_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{"state":"closed"}'
+
+# Merge PR
+curl -X POST "https://git.kokoham.com/api/v1/repos/sleepy/llama.cpp/pulls/{number}/merge" \
+  -H "Authorization: token $GITEA_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{"do_force_merge":false,"merge_title":"..."}'
 ```
+
+## Onboarding — What to Read
+
+For new agents or sessions, read in this order:
+1. **GIT.md** — this file (workflow, tests, commands, file locations)
+2. **BENCHMARKS.md** — all benchmark results, track progress toward 22 t/s
+3. **ANALYSIS_QWEN3_5_MXFP4.md** — MXFP4 format analysis (if relevant)
+4. **Issue #40** — target t/s goal and MLX comparison guidelines
+5. **MLX reference**: `../mlx-lm/mlx/include/mlx/backend/metal/kernels/quantized.h` — qmv_fast_impl

ggml/src/ggml-metal/ggml-metal-common.cpp

@@ -394,19 +394,29 @@ void ggml_graph_optimize(ggml_cgraph * gf) {
         // fuse only ops that start with these operations
         // can be expanded when needed
         if (node.op() == GGML_OP_ADD ||
+            node.op() == GGML_OP_SUB ||
+            node.op() == GGML_OP_MUL ||
+            node.op() == GGML_OP_DIV ||
             node.op() == GGML_OP_NORM ||
             node.op() == GGML_OP_RMS_NORM) {
             ops[0] = node.op();
 
             int f = i + 1;
             while (f < n && f < i + MAX_FUSE) {
-                // conservatively allow fusing only these ops
+                // bin ops (ADD/SUB/MUL/DIV) must be same type to fuse
-                // can be expanded when needed
+                // NORM/RMS_NORM can chain with MUL/ADD
-                if (gf->nodes[f]->op != GGML_OP_ADD &&
+                if (node.op() == GGML_OP_ADD ||
-                    gf->nodes[f]->op != GGML_OP_MUL &&
+                    node.op() == GGML_OP_SUB ||
-                    gf->nodes[f]->op != GGML_OP_NORM &&
+                    node.op() == GGML_OP_MUL ||
-                    gf->nodes[f]->op != GGML_OP_RMS_NORM) {
+                    node.op() == GGML_OP_DIV) {
-                    break;
+                    if (gf->nodes[f]->op != node.op()) break;
+                } else {
+                    if (gf->nodes[f]->op != GGML_OP_ADD &&
+                        gf->nodes[f]->op != GGML_OP_MUL &&
+                        gf->nodes[f]->op != GGML_OP_NORM &&
+                        gf->nodes[f]->op != GGML_OP_RMS_NORM) {
+                        break;
+                    }
                 }
                 ops[f - i] = gf->nodes[f]->op;
                 f++;

ggml/src/ggml-metal/ggml-metal-ops.cpp

@@ -3118,19 +3118,15 @@ int ggml_metal_op_bin(ggml_metal_op_t ctx, int idx) {
 
     int n_fuse = 1;
 
-    // c[0] = add(a,    b[0])
+    // c[0] = op(a,     b[0])
-    // c[1] = add(c[0], b[1])
+    // c[1] = op(c[0],  b[1])
-    // c[2] = add(c[1], b[2])
+    // c[2] = op(c[1],  b[2])
     // ...
     if (use_fusion) {
-        fops[0] = GGML_OP_ADD;
+        ggml_op cur_op = op->op;
-        fops[1] = GGML_OP_ADD;
+        for (int i = 0; i < 8; ++i) {
-        fops[2] = GGML_OP_ADD;
+            fops[i] = cur_op;
-        fops[3] = GGML_OP_ADD;
+        }
-        fops[4] = GGML_OP_ADD;
-        fops[5] = GGML_OP_ADD;
-        fops[6] = GGML_OP_ADD;
-        fops[7] = GGML_OP_ADD;
 
         // note: in metal, we sometimes encode the graph in parallel so we have to avoid fusing ops
         //       across splits. idx_end indicates the last node in the current split
@@ -3165,7 +3161,7 @@ int ggml_metal_op_bin(ggml_metal_op_t ctx, int idx) {
         ++n_fuse;
 
         if (debug_fusion > 1 && n_fuse > 1) {
-            GGML_LOG_DEBUG("%s: fuse: ADD x %d\n", __func__, n_fuse);
+            GGML_LOG_DEBUG("%s: fuse: %s x %d\n", __func__, ggml_op_name(cur_op), n_fuse);
         }
     }