Add 'Last Updated: April 9, 2026' to all markdown files missing dates

Update README.md: Mark research run as complete, add conclusion.md reference, clean up folder structure
Add conclusion.md: Comprehensive analysis of harness suitability for local models with research-backed recommendations
2026-04-09 17:27:57 +02:00 · 2026-04-09 17:25:30 +02:00 · 2026-04-09 17:22:42 +02:00 · 2026-04-09 17:15:36 +02:00 · 2026-04-09 17:14:27 +02:00 · 2026-04-09 16:25:19 +02:00
27 changed files with 2584 additions and 323 deletions
@@ -1,5 +1,7 @@
 # AGENTS.md

+**Last Updated:** April 9, 2026
+
 ## Research Project: Coding Agent Harness Analysis

 ### Objective
@@ -0,0 +1,160 @@
+# Feedback File Structure Template
+
+Use this structure for all feedback files to maintain consistency across the repository.
+
+## Standard Header
+
+```markdown
+# [Model Name] with [Harness] - Feedback Report
+
+**Model:** [Full model name]
+**Size:** [Parameters, e.g., 27B, 30B-A3B MoE]
+**Provider:** [Company/API, e.g., OpenAI, Anthropic, Ollama]
+**Harness:** [Harness name, e.g., OpenCode, Hermes, ForgeCode, pi]
+**Date Compiled:** [YYYY-MM-DD]
+**Source References:** [Primary sources]
+
+---
+```
+
+## Required Sections
+
+### 1. Quick Reference
+
+```markdown
+## Quick Reference
+
+| Attribute | Value |
+|-----------|-------|
+| Model | [Name] |
+| Size | [Parameters] |
+| Context Window | [e.g., 128K, 1M] |
+| Best For | [Use case summary] |
+| Cost | [If applicable] |
+```
+
+### 2. Benchmark Results
+
+```markdown
+## Benchmark Results
+
+### [Benchmark Name]
+- **Score:** [X%] (Rank #Y)
+- **Harness:** [If Terminal-Bench or harness-specific]
+- **Date:** [When tested]
+- **Note:** [Any important context]
+
+**Important:** For Terminal-Bench, always note that scores are harness+model combinations.
+```
+
+### 3. What Worked Well
+
+```markdown
+## What Worked Well
+
+1. **[Key Point]**
+   - Detailed explanation
+   - Supporting evidence
+
+2. **[Key Point]**
+   - Details
+```
+
+### 4. Issues Encountered
+
+```markdown
+## Issues Encountered
+
+1. **[Issue Title]**
+   - **Severity:** [Critical/Major/Minor]
+   - **Description:** Details
+   - **Workaround:** If any
+
+2. **[Issue Title]**
+   - Details
+```
+
+### 5. Configuration (Optional)
+
+```markdown
+## Configuration
+
+```json
+[Configuration example]
+```
+
+Or for CLI flags:
+
+```bash
+[Command line options]
+```
+```
+
+### 6. Source References
+
+```markdown
+## Source References
+
+1. **[Source Name]**: [URL]
+   - [Brief description of what it covers]
+
+2. **[Source Name]**: [URL]
+   - Description
+```
+
+## For Multi-Model Files
+
+If a file covers multiple models, use this structure:
+
+```markdown
+# [Topic] Feedback for [Harness]
+
+**Date Compiled:** [YYYY-MM-DD]
+**Source References:** [Primary sources]
+
+---
+
+## Model Reference Guide
+
+| Model | Size | Provider | Notes |
+|-------|------|----------|-------|
+| [Name] | [Size] | [Provider] | [Key info] |
+
+---
+
+## [Model 1]
+
+[Follow standard sections above]
+
+---
+
+## [Model 2]
+
+[Follow standard sections above]
+```
+
+## Style Guidelines
+
+1. **Use tables** for comparative data
+2. **Use bullet points** for lists
+3. **Use numbered lists** for sequential steps or ranked items
+4. **Bold** key terms and metrics
+5. **Italic** for emphasis
+6. `Code formatting` for commands, file names, and technical terms
+7. **Always cite sources** with full URLs
+8. **Note dates** for time-sensitive information
+
+## Special Notes
+
+### Terminal-Bench
+Always clarify that Terminal-Bench scores represent **harness+model** combinations, not raw model capability. Include the harness name in the benchmark table.
+
+### Qwen Models
+Include the Model Reference Guide when discussing Qwen models to avoid confusion between Qwen3, Qwen 3.5, and Qwen2.5 families.
+
+Current Qwen 3.5 MoE models include: 27B, 35B-A3B, 122B-A10B, 397B-A17B.
+
+### Verified vs Self-Reported
+Note when benchmark scores are:
+- **Verified:** Independently validated (e.g., SWE-bench Verified)
+- **Self-Reported:** Submitted by the harness developers themselves
@@ -1,61 +1,73 @@
 # Coding Harness Feedback Analysis

-Research on four coding agent harnesses to understand what works best for different model sizes, particularly smaller/local models.
+**Last Updated:** April 9, 2026
+
+Research analyzing four coding agent harnesses (opencode, pi, hermes, forgecode) to understand what works best for local/smaller models (7B-27B parameters).
+
+## What Was Done
+
+1. **Repository Analysis**: Each harness was analyzed for prompts, tools, parsing, and skills system suitability for local models
+2. **Community Feedback Synthesis**: GitHub issues, Reddit discussions, and Discord reports compiled per harness
+3. **Research Integration**: Findings cross-referenced with agent systems research (prompting, orchestration, evaluation)
+
+## Key Output
+
+**`conclusion.md`** — Comprehensive analysis covering:
+- What's working well across all four harnesses
+- Critical gaps for local model compatibility
+- Research-backed recommendations with citations
+- Priority fixes (immediate, short-term, medium-term)

 ## Folder Structure

 ```
-├── AGENTS.md              # Project overview and data collection strategy
-├── Research*.md           # Prompt research and orchestration strategies
+├── conclusion.md          # Main findings and recommendations
 │
-├── opencode/              # Go-based coding agent
-│   ├── feedback/
-│   │   ├── frontier/      # GPT-5.4, Claude Opus, Gemini feedback
-│   │   └── localllm/      # Local model feedback (prompting, tool handling)
-│   └── repo/              # Source code (submodule)
+├── AGENTS.md              # Original project scope and strategy
 │
-├── pi/                    # Minimal terminal coding harness by Mario Zechner
-│   ├── feedback/
-│   │   ├── frontier/      # (empty - in progress)
-│   │   └── localllm/      # (empty - in progress)
-│   └── repo/              # Source code (submodule)
+├── opencode/
+│   ├── REPO_FEEDBACK.md   # Repository analysis (prompts, tools, parsing)
+│   └── feedback/          # Community feedback by model tier
+│       ├── frontier/      # GPT-5.4, Claude, Gemini
+│       └── localllm/      # Qwen, Gemma, local model issues
 │
-├── hermes/                # Nous Research's agent
-│   ├── feedback/
-│   │   ├── frontier/      # Claude, GPT, budget provider feedback
-│   │   ├── localllm/      # Qwen, Gemma, local model feedback
-│   │   └── general/       # Bug reports, benchmarks, features
-│   └── repo/              # Source code (submodule)
+├── pi/
+│   ├── REPO_FEEDBACK.md   # Repository analysis
+│   └── feedback/
+│       ├── frontier/      # Frontier model feedback
+│       └── localllm/      # Local model feedback
 │
-└── forgecode/             # AI pair programmer with sub-agents
-    ├── feedback/
-    │   ├── frontier/      # GPT-5.4, Claude, Gemini, pricing, security
-    │   └── localllm/      # Qwen, MiniMax, GLM, DeepSeek feedback
-    └── repo/              # Source code (submodule)
+├── hermes/
+│   ├── REPO_FEEDBACK.md   # Repository analysis
+│   └── feedback/
+│       ├── frontier/      # Claude, GPT feedback
+│       ├── localllm/      # Qwen, Gemma, local setup
+│       └── general/       # Bug reports, benchmarks
+│
+└── forgecode/
+    ├── REPO_FEEDBACK.md   # Repository analysis
+    └── feedback/
+        ├── frontier/      # GPT-5.4, Claude, pricing
+        └── localllm/      # Qwen, MiniMax, GLM, DeepSeek
 ```

-## Quick Navigation
+## Quick Reference

-| Harness | Feedback Location | Key Topics |
-|---------|------------------|------------|
-| **opencode** | `opencode/feedback/` | Tool calling, local model prompting |
-| **pi** | `pi/feedback/` | (Being researched) |
-| **hermes** | `hermes/feedback/` | Terminal-bench results, local setup |
-| **forgecode** | `forgecode/feedback/` | Pricing, benchmarks, security |
+| Harness | Best For | Key Limitation |
+|---------|----------|----------------|
+| **pi-mono** | Local models (7B+) | Minimal overhead, needs JSON retry layer |
+| **hermes** | Frontier & 27B+ | 14K token overhead, needs tiered toolsets |
+| **forgecode** | Sub-agent workflows | Multiple system messages break Qwen3.5 |
+| **opencode** | Frontier models | Verbose prompts, no JSON repair |

-## Feedback Format
+## Research Sources

-Each feedback file includes:
- Model name/size/provider
- Task performance or benchmark results
- Issues encountered
- What worked well
- Source reference (URL, Discord, GitHub issues)
+Analysis cross-references findings from:
+- SOLVE-Med / MATA (small-model orchestration)
+- ATLAS (generate-verify-repair with 14B models)
+- StateFlow (FSM-based agent loops)
+- JetBrains (observation masking vs summarization)
+- Anthropic (Building Effective AI Agents)
+- Anthropic (Harness Design for Long-Running Apps)

-## Research Focus
-
- Tool handling and capabilities
- Skills system effectiveness  
- Prompt engineering strategies
- Context management
- Error recovery
+See `../entropy/Research/md/` for full research notes.
@@ -0,0 +1,366 @@
+# Coding Agent Harness Analysis: Conclusions
+
+**Date:** April 9, 2026  
+**Scope:** opencode, hermes, forgecode, pi-mono  
+**Based on:** Repository feedback analysis + Research literature
+
+---
+
+## Executive Summary
+
+This analysis synthesizes findings from four coding agent harnesses against current research on agent design, prompting, and orchestration. The goal is to identify what architectural patterns work well for local/smaller models (7B-27B parameters) and where the current approaches fall short.
+
+**Key Finding:** The gap between frontier-optimized and local-suitable harnesses is substantial but bridgeable. Current harnesses prioritize capability over efficiency, leaving significant room for local-model-specific optimizations.
+
+---
+
+## What Works Well
+
+### 1. Skills System Design
+
+All four harnesses implement some form of skills/sub-agent system, and this pattern is consistently well-designed:
+
+- **pi-mono**: XML-formatted skills with clear delimiters (`<available_skills>`, `<skill>`), on-demand loading prevents context bloat (`disableModelInvocation` flag)
+- **Hermes**: Skills caching, progressive disclosure (Level 0: names only, Level 1: full content when needed via `skill_view()`)
+- **ForgeCode**: Clean skill invocation pattern, dynamic loading via tool call
+- **OpenCode**: Sub-agents use minimal TaskPrompt (~17 lines) instead of verbose CoderPrompt (~220 lines)
+
+**Research Support:** This aligns with the "Principled Instructions" finding that structured, hierarchical information reduces cognitive load [Research-prompt.md §14]. The XML formatting specifically leverages the finding that "XML tags for complex prompts" reduce misinterpretation vs ambiguous delimiters [Research-prompt.md §12, §20].
+
+**Why It Helps Local Models:** Specialized skills reduce the cognitive load on the main prompt. The model sees skill names/descriptions but only loads full content when explicitly invoked, keeping the working context minimal.
+
+---
+
+### 2. Model-Specific Prompting
+
+Both Hermes and OpenCode implement model-aware prompting:
+
+- **Hermes**: `TOOL_USE_ENFORCEMENT_GUIDANCE` applied conditionally based on model family (GPT, Gemini, Gemma, Grok) [hermes/REPO_FEEDBACK.md §5]
+- **OpenCode**: Different prompt structures for Anthropic vs OpenAI endpoints [opencode/REPO_FEEDBACK.md §1.1]
+
+**Research Support:** The research emphasizes that "performance swings of up to 76 accuracy points from single-character formatting differences" occur, and "format effects do not transfer across models" [Research-prompt.md §12]. Model-specific prompting is not optional—it's a reliability requirement.
+
+**Why It Helps Local Models:** Local models often have different chat templates, instruction-following patterns, and tool-calling formats. One-size-fits-all prompts fail more often on smaller models.
+
+---
+
+### 3. Tool Schema Normalization
+
+**ForgeCode** stands out with a sophisticated transformer pipeline:
+- `normalize_tool_schema.rs`: Removes duplicate `description` and `title` from parameters
+- `enforce_strict_schema.rs`: Adds `additionalProperties: false` for stricter JSON compliance
+- `enforce_strict_tool_schema.rs`: Converts nullable enums to OpenAI-compatible format [forgecode/REPO_FEEDBACK.md §2]
+
+**Research Support:** Schema-enforced constrained decoding "removes syntactic failures" compared to prompt-only structured output which has a "5–20% failure rate" [Research-prompt.md §17]. The finding that adding a `reasoning` field first in JSON schemas improves semantic quality applies here—simpler schemas leave more room for model reasoning.
+
+**Why It Helps Local Models:** Simplified, strict schemas reduce parsing errors. Smaller models struggle with deeply nested or ambiguous schemas; normalization removes cognitive overhead.
+
+---
+
+### 4. Minimal System Prompts
+
+**pi-mono** achieves the best results here with a deliberately minimal approach:
+- System prompt: ~1000 tokens [pi/REPO_FEEDBACK.md §1]
+- Clear, direct language without excessive constraints
+- Task instruction at the end (document-first, query-last ordering)
+
+**Research Support:** The "Lost in the Middle" research shows "30%+ degradation on content buried in the middle" of long contexts, and placement of documents first with query last yields "up to 30% quality improvement" [Research-prompt.md §11]. LLMLingua-2 achieves "20x compression with only ~1.5 accuracy point drop" when compressing context/documents rather than instructions [Research-prompt.md §19].
+
+**Why It Helps Local Models:** Smaller context windows (4K-32K) mean every token counts. Minimal prompts preserve working memory for the actual task.
+
+---
+
+### 5. Context Compaction / Management
+
+**pi-mono** implements sophisticated compaction:
+- Structured summaries with sections: Goal, Constraints, Current State, File Operations, etc.
+- File operation tracking for context awareness [pi/REPO_FEEDBACK.md §4.2]
+
+**Research Support:** JetBrains Research found that "observation masking matched or outperformed LLM summarization in 4 of 5 configurations, at lower complexity" and achieved "2.6% higher solve rates while being 52% cheaper" with Qwen3-Coder [Research-orchestration.md §11]. The research recommends triggering compaction at 70–80% of context limit, preserving original task spec and recent N turns verbatim.
+
+**Why It Helps Local Models:** Small context windows fill quickly. Well-designed compaction preserves essential state while removing noise.
+
+---
+
+### 6. Auto-Discovery of Local Endpoints
+
+**OpenCode** and **Hermes** automatically discover local models:
+- Query v1/models and api/v0/models endpoints
+- Auto-configure context windows and defaults
+
+**Why It Helps Local Models:** Reduces manual configuration burden, which is a significant barrier to local model adoption.
+
+---
+
+## What's Weak / Needs New Ideas
+
+### 1. Token Overhead (Critical)
+
+| Harness | Fixed Overhead | Impact on Local Models |
+|---------|---------------|------------------------|
+| **Hermes** | ~14K tokens (31 tools + system prompt) | Leaves only 20% of 4K context for actual work |
+| **OpenCode** | CoderPrompt ~170 lines, bash tool ~3500 chars | Designed for frontier models; 27B+ threshold |
+| **ForgeCode** | 58+ line prompts, 12+ rules | Exceeds comprehension capacity of <14B models |
+| **pi-mono** | ~1000 tokens | ✅ Acceptable |
+
+**The Problem:** Hermes explicitly acknowledges this as a "fundamental architectural constraint, not a bug" [hermes/REPO_FEEDBACK.md §1]. OpenCode's bash tool description alone is ~3500 characters with embedded git/PR workflows [opencode/REPO_FEEDBACK.md §2.2].
+
+**Research Gap:** While the research emphasizes prompt compression (LLMLingua-2) and placement (Lost in the Middle), there's no systematic study on **dynamic prompt tiering** based on model capacity. The harnesses treat all models the same.
+
+**New Idea Needed:** Dynamic prompt compression tiering:
+- Detect or configure model size/tier
+- Strip examples and reduce verbosity for smaller models
+- Abbreviate tool descriptions (especially bash/edit)
+- Create "essential tools only" mode for <14B models
+
+This aligns with SOLVE-Med/MATA findings that "small specialized models, when orchestrated well, can outperform much larger standalone systems" [Research.md §12, Research-orchestration.md §5].
+
+---
+
+### 2. JSON Parsing / Tool Calling Reliability (Critical)
+
+| Harness | Issue |
+|---------|-------|
+| **OpenCode** | No JSON repair layer; relies entirely on SDK/provider; "NO resilience for JSON repair/relaxation" [opencode/REPO_FEEDBACK.md §3.2] |
+| **Hermes** | llama-server returns `dict` instead of JSON string → crashes on `.strip()` (Issue #1071) [hermes/REPO_FEEDBACK.md §2] |
+| **ForgeCode** | XML tool wrapper `<forge_tool_call>` confuses local models (Qwen3.5 specifically) [forgecode/REPO_FEEDBACK.md §2] |
+| **pi-mono** | ~1.6 retries per prompt for JSON compliance [pi/REPO_FEEDBACK.md §1] |
+
+**Research Support:** The research notes that "schema-enforced constrained decoding removes syntactic failures" but this requires API-level support [Research-prompt.md §17]. For local models without this support, we're in a gap.
+
+**New Idea Needed:** A JSON extraction/repair layer with:
+- Extract JSON blocks from text output using regex/fenced code blocks
+- Fuzzy tool name matching (Levenshtein distance) against registered tools
+- Parameter type coercion (string→number, etc.)
+- Retry loops with prompt refinement on parse failure
+- Tool fallback: route to bash for simple file operations when structured calls fail
+
+This aligns with StateFlow findings that "error handling as a named state is critical" and "removing the explicit Error state caused a 5% success rate decline" [Research-orchestration.md §12].
+
+---
+
+### 3. Monolithic Toolsets (High Priority)
+
+**Hermes**: All 31 tools loaded eagerly; no granularity for resource constraints [hermes/REPO_FEEDBACK.md §9]  
+**OpenCode**: 11 core tools all presented together; no subset selection [opencode/REPO_FEEDBACK.md §2.1]
+
+**Research Support:** The research explicitly recommends routing "grep/read/run/simple classification to cheaper lanes" and reserving "expensive models for hard reasoning" [Research.md §12, Research-orchestration.md §5]. Difficulty-Aware Agentic Orchestration (DAAO) shows "a variational autoencoder estimating query difficulty + a cost/performance-aware router gives near-frontier quality at significantly lower cost" [Research-orchestration.md §6].
+
+**New Idea Needed:** Tiered toolsets based on model capability:
+- `minimal`: 5-8 essential tools (terminal, file, read, write, patch, search)
+- `standard`: 15-20 tools for 14B+ models
+- `full`: All tools for frontier models
+
+This should be configurable, not hardcoded. The harness should detect or allow users to specify model tier and adjust tool availability accordingly.
+
+---
+
+### 4. Context Window Mismanagement (High Priority)
+
+**OpenCode**: 4K default fallback is "too small for verbose prompts" [opencode/REPO_FEEDBACK.md §5.2]  
+**Hermes**: Manual sync required between Ollama `num_ctx` and Hermes config; users report "context exceeded your setting" errors [hermes/REPO_FEEDBACK.md §4]
+
+**Research Support:** The research recommends "set a hard token budget before each agent turn" and "trigger compaction when projected input exceeds 70–80% of the context limit" [Research-orchestration.md §11].
+
+**New Idea Needed:** Automatic context negotiation:
+- Query the model's actual context window from the endpoint
+- Dynamically adjust tool availability and prompt size
+- Warn users when context configuration mismatches are detected
+- Set 32K default for local models (not 4K)
+
+---
+
+### 5. KV Cache / Session Instability (Medium Priority)
+
+**OpenCode**: KV cache invalidated when sub-agent spawns (no reuse between parent/child) [opencode/REPO_FEEDBACK.md §4.2]  
+**pi-mono**: Session hangs after extended use (Issue #2422); no health monitoring [pi/REPO_FEEDBACK.md §3]
+
+**Research Support:** The StateFlow research emphasizes that "error handling as a named state is critical" [Research-orchestration.md §12]. The retry/fallback/circuit breaker pattern is recommended: "After 2 consecutive failures on the same action, force a planning reset" [Research-orchestration.md §15].
+
+**New Idea Needed:** Session health monitoring with:
+- Periodic heartbeat/ping to detect hung sessions
+- Automatic recovery with state preservation
+- Circuit breaker pattern for systematic degradation
+- Graceful shutdown handlers
+
+---
+
+### 6. Multiple System Messages (ForgeCode Specific)
+
+**ForgeCode** generates two separate system messages (`static_block` + `non_static_block`) which breaks Qwen3.5 and models with strict chat templates (Issue #2894) [forgecode/REPO_FEEDBACK.md §1].
+
+**New Idea Needed:** Combine into single system message or make second message optional via config.
+
+---
+
+## Strong Signals from Research for Local/Smaller Models
+
+Based on the research literature, here are high-confidence patterns that should be implemented in harnesses targeting local models:
+
+### 1. Difficulty-Based Model Routing (Strong Signal)
+
+**Source:** DAAO (Difficulty-Aware Agentic Orchestration) [Research-orchestration.md §6]
+
+**Finding:** A variational autoencoder estimating query difficulty + a cost/performance-aware router gives near-frontier quality at significantly lower cost.
+
+**Application:** Before dispatching to a model, classify task difficulty:
+- Simple classification → 7B model
+- Code generation → 14B model
+- Complex synthesis → 27B+ model
+
+This is the principled version of the "route to cheap specialists" heuristic.
+
+---
+
+### 2. Generate → Verify → Repair Pattern (Strong Signal)
+
+**Source:** ATLAS [Research.md §12, Research-orchestration.md §6]
+
+**Finding:** ATLAS achieves 74.6% on LiveCodeBench using a quantized 14B Qwen model on a single consumer GPU (16GB VRAM) via a three-phase pipeline:
+1. **Generate**: PlanSearch extracts constraints, produces diverse candidates; Budget Forcing controls token spend
+2. **Verify**: "Geometric Lens" scores candidates with energy field (87.8% selection accuracy) + sandboxed execution
+3. **Repair**: Self-generated test cases + iterative refinement via PR-CoT
+
+This doubles baseline pass rate (38% → 74.6%) entirely through infrastructure, not model scale.
+
+**Application:** Implement external verification for coding tasks:
+- Generate multiple candidate solutions
+- Score candidates with a lightweight verifier (could be smaller model + tests)
+- Repair failing candidates with targeted refinement
+
+---
+
+### 3. Observation Masking Over Summarization (Strong Signal)
+
+**Source:** JetBrains Research [Research-orchestration.md §11]
+
+**Finding:** Observation masking (replace old tool outputs with placeholders, keep reasoning chain) matched or outperformed LLM summarization in 4 of 5 configurations. With Qwen3-Coder 480B, masking achieved 2.6% *higher* solve rates while being 52% cheaper.
+
+**Key Insight:** LLM summarization paradoxically caused agents to run ~15% longer trajectories because summaries gave false confidence to keep going.
+
+**Application:** Default to observation masking for context compaction. Only use LLM summarization as a fallback for single oversized responses.
+
+---
+
+### 4. Small Specialists for Mechanical Subproblems (Strong Signal)
+
+**Source:** SOLVE-Med / MATA [Research.md §12, Research-orchestration.md §5]
+
+**Finding:** Small specialized models, when orchestrated well, can outperform or match much larger standalone systems.
+
+**Application:** Route mechanical tasks to small models:
+- grep/read/run → 4B model
+- simple classification → 7B model
+- code generation → 14B+ model
+- synthesis/integration → 27B+ model
+
+Reserve expensive models for hard reasoning or integration steps.
+
+---
+
+### 5. State Machine Modeling with Explicit Error States (Strong Signal)
+
+**Source:** StateFlow [Research-orchestration.md §12]
+
+**Finding:** Modeling tasks as finite state machines (FSM) with explicit states yielded 63.73% success on SQL tasks vs 40.3% for ReAct, at 5.8x lower cost. Removing the explicit Error state caused a 5% success rate decline.
+
+**Recommended Minimum States:**
+- `PLANNING`
+- `EXECUTING`
+- `OBSERVING`
+- `ERROR_RECOVERY`
+- `DONE`
+
+**Application:** Model agent loops as explicit FSMs with named error states and deterministic transitions in code (not in LLM prompts).
+
+---
+
+### 6. Bounded Loops with Fixed Retry Budgets (Strong Signal)
+
+**Source:** Portkey/Maxim on retries, fallbacks, circuit breakers [Research-orchestration.md §15]
+
+**Finding:** Three patterns form the production resilience stack:
+- **Retries**: Exponential backoff + jitter, max 3 attempts
+- **Fallbacks**: Switch to alternate model/provider
+- **Circuit breakers**: Remove endpoint from routing when failure rate exceeds threshold
+
+**Application:**
+- Define per-task max-retry budgets (not just per-call)
+- After 2 consecutive tool failures on the same action, force planning reset
+- Return structured error observations to agent, not crashes
+
+---
+
+### 7. XML Structure Over Free-Form Text (Strong Signal)
+
+**Source:** Anthropic Prompting Best Practices, POSIX Prompt Sensitivity Index [Research-prompt.md §12, §20]
+
+**Finding:** XML tags for structure reduce brittleness. "Prompt-only structured output has a 5–20% failure rate." Adding even one few-shot example dramatically reduces prompt sensitivity.
+
+**Application:**
+- Use XML tags for complex prompts: `<instructions>`, `<context>`, `<examples>`, `<input>`
+- Use `<example>` tags with 3–5 diverse examples focused on output format
+- Keep examples short; they're for format alignment, not reasoning
+
+---
+
+### 8. Episodic Memory with Structured Metadata (Strong Signal)
+
+**Source:** A-MEM, Episodic Memory research [Research-orchestration.md §9-10]
+
+**Finding:** A Zettelkasten-style memory network (structured notes with attributes, keywords, tags) doubled complex reasoning performance vs flat vector store baselines at lower token cost.
+
+**Application:**
+- Build two layers: short-term in-context working buffer + persistent episodic store
+- Enrich every stored memory with metadata at write time (task context, success/failure, timestamps, tags)
+- After task completion, abstract successful patterns from episode trace into reusable rules
+
+---
+
+## Summary Table: Harness Suitability for Local Models
+
+| Harness | Token Overhead | Tool Reliability | Context Mgmt | Overall |
+|---------|---------------|------------------|--------------|---------|
+| **pi-mono** | ✅ ~1000 tokens | ⚠️ Needs retry layer | ✅ Sophisticated | **Best suited** |
+| **Hermes** | ❌ ~14K tokens | ⚠️ Bug #1071 | ⚠️ Manual config | Needs tiering |
+| **ForgeCode** | ⚠️ Complex prompts | ⚠️ XML issues | ⚠️ 4K default | Fix #2894 first |
+| **OpenCode** | ❌ Verbose | ❌ No JSON repair | ⚠️ 4K default | Needs compression |
+
+---
+
+## Recommendations Priority
+
+### Immediate (High Impact, Low Effort)
+1. **Fix Hermes llama-server bug** (#1071): Type-check arguments before `.strip()`
+2. **Fix ForgeCode multiple system messages** (#2894): Combine into single message
+3. **Set 32K default context** for local models in all harnesses
+
+### Short-term (High Impact, Medium Effort)
+4. **Implement JSON extraction/repair layer** with fuzzy tool matching
+5. **Create tiered toolsets**: minimal (5-8 tools), standard (15-20), full (all)
+6. **Add session health monitoring** with heartbeat and circuit breakers
+
+### Medium-term (High Impact, High Effort)
+7. **Dynamic prompt compression** based on model tier
+8. **Implement generate → verify → repair** pipeline for coding tasks
+9. **Difficulty-based model routing** for multi-model deployments
+10. **Observation masking** as default compaction strategy
+
+---
+
+## References
+
+### Repository Feedback
+- opencode/REPO_FEEDBACK.md
+- hermes/REPO_FEEDBACK.md
+- forgecode/REPO_FEEDBACK.md
+- pi/REPO_FEEDBACK.md
+
+### Research Literature
+- Research.md: Core agent systems research (SOLVE-Med, MATA, ATLAS, SWE-agent, Agentless)
+- Research-prompt.md: Prompt design and single-agent strategies (Lost in the Middle, POSIX, Principled Instructions, LLMLingua)
+- Research-orchestration.md: Multi-agent design, memory, context management (StateFlow, A-MEM, JetBrains context research)
+
+---
+
+*Analysis conducted April 9, 2026. Strong conclusions backed by multiple verified reports and research citations; recommendations prioritized by impact/effort ratio.*
@@ -1,5 +1,7 @@
 # AGENTS.md

+**Last Updated:** April 9, 2026
+
 ## Research/Analysis Folder for forgecode

 This is the research and analysis folder for the **forgecode** coding harness.
@@ -1,5 +1,7 @@
 # ForgeCode Research & Analysis Folder

+**Last Updated:** April 9, 2026
+
 This folder contains comprehensive research and analysis of the **ForgeCode** coding harness from antinomyhq.

 ---
@@ -0,0 +1,307 @@
+# ForgeCode Repository Feedback Analysis
+
+**Date:** April 9, 2026  
+**Scope:** Analysis of forgecode codebase for local model compatibility  
+**Focus Areas:** Prompts, tools, parsing, skills  
+**Model Focus:** Local models (Qwen 3.5, Gemma 4, MiniMax, GLM, DeepSeek)
+
+---
+
+## Executive Summary
+
+ForgeCode has a **sophisticated but complex** architecture that presents both opportunities and challenges for local models. The harness implements numerous optimizations for tool calling reliability, but many of these rely on infrastructure that may not be available or performant with smaller models.
+
+**Key Finding:** The harness's **tool calling layer** is the primary concern for local models, followed by **prompt complexity** and **context management**. The skills system is well-designed but adds overhead.
+
+---
+
+## What Works Well for Local Models
+
+### 1. **Modular Prompt Architecture** ✅
+
+**Evidence:**
+- Templates are modular and composable (`forge-custom-agent-template.md`, `forge-partial-*.md`)
+- System context is re-rendered on each turn (plan: `2025-04-02-system-context-rendering-v2.md`)
+- Variables can be passed to prompts
+
+**Why This Helps Local Models:**
+- Smaller prompts = less context pressure
+- Re-rendering allows dynamic updates (time, environment)
+- Variables enable customization without full prompt rewrites
+
+**Strength:** **Strong** - This is well-documented and implemented in the codebase.
+
+---
+
+### 2. **Tool Schema Normalization** ✅
+
+**Evidence:**
+- `normalize_tool_schema.rs` removes duplicate `description` and `title` from parameters
+- `enforce_strict_schema.rs` adds `additionalProperties: false` for stricter JSON schema compliance
+- `enforce_strict_tool_schema.rs` converts nullable enums to OpenAI-compatible format
+
+**Why This Helps Local Models:**
+- Simplified schemas reduce parsing errors
+- Strict schemas are more predictable for smaller models
+- Nullable enum handling prevents schema validation failures
+
+**Strength:** **Strong** - Multiple transformers ensure schemas are optimized before reaching the model.
+
+---
+
+### 3. **Parallel Tool Calls** ✅
+
+**Evidence:**
+- `supports_parallel_tool_calls` flag in `system_prompt.rs`
+- Instructions in `forge-custom-agent-template.md`: "invoke all relevant tools simultaneously"
+
+**Why This Helps Local Models:**
+- Reduces total turns needed for multi-step tasks
+- Faster task completion = less context accumulation
+- Parallelism reduces timeout risk
+
+**Strength:** **Moderate** - Depends on model support; local models may not reliably support parallel calls.
+
+---
+
+### 4. **Skills System** ✅
+
+**Evidence:**
+- `forge-partial-skill-instructions.md` provides clear invocation pattern
+- Skills are loaded dynamically via tool call
+- Skills provide domain-specific workflows
+
+**Why This Helps Local Models:**
+- Specialized skills reduce cognitive load on main prompt
+- Reusable workflows = less prompt engineering overhead
+- Clear invocation pattern (`skill` tool with name only)
+
+**Strength:** **Strong** - Well-designed and documented. Skills can be invoked with minimal context.
+
+---
+
+## Problematic Areas for Local Models
+
+### 1. **Multiple System Messages** ❌
+
+**Evidence:**
+- GitHub Issue #2894: "Multiple system messages break models with strict chat templates (e.g. Qwen3.5)"
+- `system_prompt.rs` line 128: `context.set_system_messages(vec![static_block, non_static_block])`
+- Two system messages are set: `static_block` and `non_static_block`
+
+**Impact:**
+- **BREAKS** Qwen3.5 and Qwen3 models
+- Models with strict chat templates fail to parse message structure
+- Tool calling becomes unpredictable
+
+**Root Cause:**
+The harness generates two separate system messages:
+1. `static_block` - from `system_prompt.template`
+2. `non_static_block` - from `forge-custom-agent-template.md`
+
+These are concatenated into two separate system messages, which breaks models that expect a single system message.
+
+**Strength:** **Strong** - This is a confirmed bug with an open GitHub issue.
+
+**Workaround:** None yet; use different model or await fix.
+
+---
+
+### 2. **Tool Calling Format Complexity** ⚠️
+
+**Evidence:**
+- `forge-partial-tool-use-example.md` shows `<forge_tool_call>` XML wrapper
+- Tool calls must be in JSON format inside XML tags
+- Example: `<forge_tool_call>{"name": "read", "arguments": {...}}</forge_tool_call>`
+
+**Why This Is Problematic:**
+- Local models trained on varied data may not recognize custom XML wrapper
+- Qwen3.5 specifically struggles with XML tool parsing (community feedback)
+- LM Studio 0.4.9+ reportedly handles this better than raw llama.cpp
+
+**Strength:** **Moderate** - This is a known issue with community workarounds (LM Studio > raw llama.cpp).
+
+---
+
+### 3. **Context Window Pressure** ⚠️
+
+**Evidence:**
+- `system_prompt.rs` includes:
+  - Full tool definitions (`tool_information`)
+  - File list (`files`)
+  - Extension statistics (`extensions`)
+  - Custom rules (`custom_rules`)
+  - Skills list (`skills`)
+  - README content (not shown but referenced)
+
+**Impact:**
+- Local models often have smaller context windows (4K-32K)
+- Default Ollama context is 4K (too small)
+- Context can exceed 100% while still appearing to work
+
+**Strength:** **Strong** - Well-documented in `general-local-models.md`:
+> "Ollama/Qwen3 runs with 4K context window by default (too small)"
+> "Need explicit configuration to increase context"
+
+---
+
+### 4. **Prompt Complexity** ⚠️
+
+**Evidence:**
+- `forge-custom-agent-template.md` is 58 lines with complex rules
+- `non_negotiable_rules` section has 12+ rules with examples
+- `forge-command-generator-prompt.md` is 113 lines with 6+ edge case categories
+
+**Why This Is Problematic:**
+- Smaller models (<14B) struggle with long, complex prompts
+- Qwen3.5 requires higher-quality quantization for reliable parsing
+- Context pressure increases with prompt length
+
+**Strength:** **Moderate** - Community feedback suggests:
+> "30B+ recommended for serious coding work"
+> "<7B models: Generally insufficient for reliable agentic tool use"
+
+---
+
+### 5. **Tool Naming Conventions** ⚠️
+
+**Evidence:**
+- `tool-calling-reliability.md`: "Models pattern-match against training data first"
+- Renaming edit tool to `old_string`/`new_string` "measurably dropped tool-call error rates"
+
+**Why This Is Problematic:**
+- ForgeCode's tool names may not match training data patterns
+- Local models rely more on pattern matching than frontier models
+- Custom tool names increase error rate
+
+**Strength:** **Moderate** - This is a known issue with a known fix (use established names).
+
+---
+
+## Codebase Quality Assessment
+
+### **Good: Architecture & Design**
+
+1. **Transformer Pipeline** (`crates/forge_app/src/dto/`)
+   - Multiple transformers for different providers (Anthropic, OpenAI, Google)
+   - Each transformer is focused and testable
+   - Example: `enforce_schema.rs`, `normalize_tool_schema.rs`
+
+2. **Tool Registry** (`tool_registry.rs`)
+   - Clear separation of concerns
+   - Timeout handling built-in
+   - Permission checking before execution
+
+3. **Template Engine** (`system_prompt.rs`)
+   - Handlebars-style templating
+   - Variables passed to templates
+   - Re-rendering on each turn
+
+### **Concerning: Complexity**
+
+1. **Multiple Layers of Abstraction**
+   - `ToolRegistry` → `ToolExecutor` → `ToolCatalog`
+   - `SystemPrompt` → `TemplateEngine` → `Template`
+   - Each layer adds overhead and potential failure points
+
+2. **Generic Type Parameters**
+   - `ToolRegistry<S>` where `S: Services + EnvironmentInfra`
+   - Complex trait bounds make debugging harder
+   - Local models may struggle with the resulting prompts
+
+3. **Async Complexity**
+   - Heavy use of `async/await` and `tokio`
+   - `join_all` for parallel tool calls
+   - Timeout handling adds latency
+
+---
+
+## Recommendations for Local Models
+
+### **Immediate Fixes (High Priority)**
+
+1. **Fix Multiple System Messages** (#2894)
+   - Combine `static_block` and `non_static_block` into single message
+   - Or make second message optional via config
+
+2. **Add Context Window Config**
+   - Allow users to specify context window size
+   - Default to 32K for local models (not 4K)
+
+3. **Simplify Tool Call Format**
+   - Add option for pure JSON (no XML wrapper)
+   - Let users choose based on model compatibility
+
+### **Medium Priority**
+
+4. **Tool Name Optimization**
+   - Use established names (`old_string`/`new_string`)
+   - Document tool naming conventions for users
+
+5. **Context Compaction**
+   - Implement automatic context compression
+   - Add warning when context exceeds 80%
+
+6. **Quantization Guidance**
+   - Document recommended quantizations per model
+   - Q8_0 for tool calling, Q4_K_M for basic tasks
+
+### **Lower Priority**
+
+7. **Skills System Optimization**
+   - Lazy-load skills (only when needed)
+   - Cache skill content to reduce prompt size
+
+8. **Parallel Tool Call Fallback**
+   - Detect model support for parallel calls
+   - Fall back to sequential if not supported
+
+---
+
+## Conclusions
+
+### **Strong Conclusions (Based on Direct Evidence)**
+
+1. **Multiple system messages break Qwen3.5** - Confirmed via GitHub issue #2894
+2. **4K default context is insufficient** - Documented in `general-local-models.md`
+3. **Tool schema normalization helps** - Multiple transformers ensure strict compliance
+4. **30B+ recommended for serious work** - Community consensus from Reddit r/LocalLLaMA
+
+### **Moderate Conclusions (Based on Code Analysis + Community Feedback)**
+
+1. **XML tool wrapper may confuse local models** - Qwen3.5 struggles with XML parsing
+2. **Prompt complexity exceeds local model capacity** - 58+ line prompts with 12+ rules
+3. **Pattern matching on tool names matters** - Renaming improves reliability
+4. **Parallel calls reduce context pressure** - But may not be supported by all models
+
+### **Weaker Conclusions (Speculative)**
+
+1. **Generic type parameters add overhead** - Plausible but not directly measured
+2. **Async complexity affects local models** - Indirect impact via prompt size
+3. **Skills system adds latency** - Not measured, but plausible
+
+---
+
+## Source References
+
+1. **GitHub Issue #2894:** https://github.com/antinomyhq/forgecode/issues/2894
+2. **Reddit r/LocalLLaMA:** https://www.reddit.com/r/LocalLLaMA/comments/1qz5uww/qwen3_coder_next_as_first_usable_coding_model_60/
+3. **ForgeCode Blog:** https://forgecode.dev/blog/benchmarks-dont-matter/
+4. **Tool Calling Reliability:** `forgecode/feedback/localllm/tool-calling-reliability.md`
+5. **General Local Models:** `forgecode/feedback/localllm/general-local-models.md`
+
+---
+
+## Appendix: Key Code Locations
+
+| Component | File Path | Local Model Impact |
+|-----------|-----------|-------------------|
+| Multiple System Messages | `crates/forge_app/src/system_prompt.rs:128` | **HIGH** - Breaks Qwen3.5 |
+| Tool Schema Normalization | `crates/forge_app/src/dto/openai/transformers/normalize_tool_schema.rs` | **POSITIVE** - Helps all models |
+| Parallel Tool Calls | `crates/forge_app/src/system_prompt.rs:114` | **MODERATE** - Depends on model |
+| Skills System | `crates/forge_app/src/system_prompt.rs:95` | **POSITIVE** - Well-designed |
+| Context Rendering | `plans/2025-04-02-system-context-rendering-v2.md` | **POSITIVE** - Dynamic updates |
+
+---
+
+**Author's Note:** This analysis combines direct code inspection with community feedback. Strong conclusions are backed by both code and external sources. Weaker conclusions are based on code patterns and reasonable inference. Always verify with your specific model/backend combination.
@@ -18,12 +18,26 @@ ForgeCode achieved **81.8% on TermBench 2.0** (tied with GPT 5.4 and Opus 4.6),

 ## TermBench 2.0 Results

+### Current Leaderboard (Harness + Model Combinations)
+
+**Important:** Terminal-Bench measures agent harness + model combinations, not raw model capability.
+
+| Rank | Harness | Model | Score | Date |
+|------|---------|-------|-------|------|
+| 1 | Pilot | Claude Opus 4.6 | 82.9% | 2026-04-01 |
+| 2 | ForgeCode | GPT 5.4 | 81.8% | 2026-03-12 |
+| 3 | ForgeCode | Claude Opus 4.6 | 81.8% | 2026-03-12 |
+| 4 | TongAgents | Gemini 3.1 Pro | 80.2% | 2026-03-13 |
+| 5 | SageAgent | GPT-5.3-Codex | 78.4% | 2026-03-13 |
+
+**Source:** https://www.tbench.ai/leaderboard/terminal-bench/2.0
+
 ### Self-Reported (via ForgeCode at tbench.ai)
-| Configuration | Score | Rank |
-|--------------|-------|------|
-| ForgeCode + GPT 5.4 | 81.8% | #1 |
-| ForgeCode + Opus 4.6 | 81.8% | #1 |
-| Claude Code + Opus 4.6 | 58.0% | #39 |
+| Configuration | Score |
+|--------------|-------|
+| ForgeCode + GPT 5.4 | 81.8% |
+| ForgeCode + Claude Opus 4.6 | 81.8% |
+| Claude Code + Claude Opus 4.6 | 58.0% |

 ### Independent SWE-bench (Princeton/UChicago)
 | Configuration | Score |
@@ -88,17 +102,16 @@ ForgeCode transparently documented their journey:
 ## Independent Terminal-Bench Data

 From llm-stats.com (April 9, 2026):
- **23 models evaluated**
- **Average score:** 0.345 (34.5%)
- **Best score:** 0.500 (50.0%) - Claude Sonnet 4.5
- **All results self-reported** (0 verified)
+- **28+ models evaluated**
+- **Average score:** Varies significantly by harness
+- **All results self-reported** (0 verified on independent platforms)

-**Top 3:**
-1. Claude Sonnet 4.5: 50.0%
-2. MiniMax M2.1: 47.9%
-3. Kimi K2-Thinking: 47.1%
+**Key Point:** Terminal-Bench scores are inherently harness-specific. The same model (e.g., Claude Opus 4.6) achieves different scores with different harnesses:
+- Pilot + Claude Opus 4.6: 82.9%
+- ForgeCode + Claude Opus 4.6: 81.8%
+- Claude Code + Claude Opus 4.6: 58.0%

-**Note:** ForgeCode's 81.8% is not on this independent leaderboard; it was self-reported on tbench.ai.
+**Note:** The 24-point gap between ForgeCode and Claude Code on the same model illustrates how harness engineering significantly impacts benchmark scores.

 ---

@@ -1,64 +1,88 @@
 # Claude Opus 4.6 with ForgeCode - Feedback Report

 **Model:** Claude Opus 4.6  
+**Size:** [Not specified]  
 **Provider:** Anthropic  
 **Harness:** ForgeCode  
-**Source References:** DEV Community (Liran Baba), ForgeCode Blog, Reddit r/ClaudeCode  
-**Date Compiled:** April 9, 2026
+**Date Compiled:** April 9, 2026  
+**Source References:** DEV Community (Liran Baba), ForgeCode Blog, Reddit r/ClaudeCode

 ---

-## Benchmark Performance
+## Quick Reference

-### TermBench 2.0 (Self-Reported via ForgeCode)
+| Attribute | Value |
+|-----------|-------|
+| Model | Claude Opus 4.6 |
+| Provider | Anthropic |
+| Context Window | 200K tokens |
+| Best For | Complex reasoning, large codebases, long-horizon tasks |
+| Cost | ~$15/M input, ~$75/M output |
+
+---
+
+## Benchmark Results
+
+### Terminal-Bench 2.0 (Harness-Specific)
 - **Score:** 81.8% (tied for #1)
- **Comparison:** Claude Code + Opus 4.6 scored 58.0% (Rank #39)
+- **Harness:** ForgeCode
+- **Comparison:** Claude Code + Opus 4.6: 58.0% (Rank #39)
 - **Gap:** ~24 percentage points in favor of ForgeCode harness
+- **Note:** Score reflects harness+model combination, not raw model capability

-### SWE-bench Verified (Independent - Princeton/UChicago)
+### SWE-Bench Verified (Independent)
 - **ForgeCode + Claude 4:** 72.7%
 - **Claude Code + Claude 3.7 Sonnet (extended thinking):** 70.3%
- **Gap:** Only 2.4 percentage points
+- **Gap:** Only 2.4 percentage points on independent validation
+- **Source:** Princeton/UChicago

-**Key Insight:** The benchmark gap narrows significantly on independent validation. TermBench 2.0 results are self-reported by ForgeCode itself.
+### SWE-Bench Pro
+- **Score:** 57.3% (Rank varies)
+- **Behind:** Claude Mythos Preview (77.8%), GLM-5.1 (58.4%), GPT-5.4 (57.7%)
+- **Source:** llm-stats.com

---
-
-## Real-World Performance Feedback
-
-### Speed
- **Observation:** "Noticeably faster than Claude Code. Not marginal, real."
- **Test Case:** Adding post counter to blog index (Astro 6, ~30 files)
-  - Claude Code: ~90 seconds
-  - ForgeCode + Opus 4.6: <30 seconds
- **Consistency:** Multi-file renames, component additions, layout restructuring all showed faster performance
-
-### Why Faster
-1. **Rust binary** vs Claude Code's TypeScript (better startup/memory)
-2. **Context engine:** Indexes function signatures and module boundaries instead of dumping raw files (~90% context size reduction)
-3. **Selective context:** Pulls only what the agent needs
-
-### Stability
- **Assessment:** Excellent stability with Opus 4.6 through ForgeCode
- **No tool call failures reported** (unlike GPT 5.4 experience)
- Consistent performance across different task types
+**Key Insight:** The benchmark gap narrows significantly on independent validation. Terminal-Bench results are self-reported by harness developers.

 ---

 ## What Worked Well

-1. **Multi-file refactoring:** Handles complex changes across file boundaries efficiently
-2. **Code comprehension:** Strong understanding of Astro/React components
-3. **Speed on complex tasks:** Consistently 3x faster than Claude Code on identical tasks
-4. **Planning with muse:** Plan output felt "more detailed and verbose than Claude Code's plan mode"
+1. **Speed**
+   - **Observation:** "Noticeably faster than Claude Code. Not marginal, real."
+   - **Test Case:** Adding post counter to blog index (Astro 6, ~30 files)
+     - Claude Code: ~90 seconds
+     - ForgeCode + Opus 4.6: <30 seconds
+   - **Consistency:** Multi-file renames, component additions, layout restructuring all showed faster performance
+   - **Why:** Rust binary vs TypeScript, context engine indexes signatures (~90% size reduction), selective context
+
+2. **Multi-file Refactoring**
+   - Handles complex changes across file boundaries efficiently
+   - Strong understanding of Astro/React components
+   - Consistently 3x faster than Claude Code on identical tasks
+
+3. **Planning with Muse**
+   - Plan output felt "more detailed and verbose than Claude Code's plan mode"
+
+4. **Stability**
+   - Excellent stability with Opus 4.6 through ForgeCode
+   - No tool call failures reported (unlike GPT 5.4 experience)
+   - Consistent performance across different task types

 ---

 ## Issues Encountered

-1. **Ecosystem gaps:** No IDE extensions, no hooks, no checkpoints/rewind
-2. **No auto-memory:** Context doesn't persist between sessions
-3. **No built-in sandbox:** Requires manual `--sandbox` flag for isolation
+1. **Ecosystem Gaps** (Major)
+   - **Description:** No IDE extensions, no hooks, no checkpoints/rewind
+   - **Impact:** Less integrated workflow compared to Claude Code
+
+2. **No Auto-Memory** (Minor)
+   - **Description:** Context doesn't persist between sessions
+   - **Impact:** Requires re-contextualization on new sessions
+
+3. **No Built-in Sandbox** (Minor)
+   - **Description:** Requires manual `--sandbox` flag for isolation
+   - **Impact:** Security requires explicit configuration

 ---

@@ -76,6 +100,11 @@

 ## Source References

-1. **DEV Community:** https://dev.to/liran_baba/forgecode-vs-claude-code-which-ai-coding-agent-actually-wins-36c
-2. **ForgeCode Blog:** https://forgecode.dev/blog/benchmarks-dont-matter/
-3. **Reddit r/ClaudeCode:** https://www.reddit.com/r/ClaudeCode/comments/1royhni/someone_is_using_forgecodedev/
+1. **DEV Community - ForgeCode vs Claude Code**: https://dev.to/liran_baba/forgecode-vs-claude-code-which-ai-coding-agent-actually-wins-36c
+   - Real-world performance comparison by Liran Baba
+
+2. **ForgeCode Blog - Benchmarks Don't Matter**: https://forgecode.dev/blog/benchmarks-dont-matter/
+   - Documentation of harness optimizations and benchmark methodology
+
+3. **Reddit r/ClaudeCode**: https://www.reddit.com/r/ClaudeCode/comments/1royhni/someone_is_using_forgecodedev/
+   - Community discussion on ForgeCode usage
@@ -1,45 +1,79 @@
 # GPT 5.4 with ForgeCode - Feedback Report

 **Model:** GPT 5.4  
+**Size:** [Not specified]  
 **Provider:** OpenAI  
 **Harness:** ForgeCode  
-**Source References:** DEV Community (Liran Baba), ForgeCode Blog  
-**Date Compiled:** April 9, 2026
+**Date Compiled:** April 9, 2026  
+**Source References:** DEV Community (Liran Baba), ForgeCode Blog

 ---

-## Benchmark Performance
+## Quick Reference

-### TermBench 2.0 (Self-Reported via ForgeCode)
- **Score:** 81.8% (tied for #1 with Opus 4.6)
- **Note:** Achieved through extensive harness optimizations, not raw model capability
+| Attribute | Value |
+|-----------|-------|
+| Model | GPT 5.4 |
+| Provider | OpenAI |
+| Context Window | 1M tokens |
+| Best For | Terminal execution, speed |
+| Cost | ~$10/M input, ~$30/M output |

 ---

-## Real-World Performance Feedback
+## Benchmark Results

-### Stability Issues
- **Assessment:** "Borderline unusable" for some tasks
- **Specific Issue:** 15-minute research task on small repo
-  - Tool calls repeatedly failing
-  - Agent stuck in retry loops
-  - Required manual kill
+### Terminal-Bench 2.0 (Harness-Specific)
+- **Score:** 81.8% (tied for #1)
+- **Harness:** ForgeCode
+- **Date:** March 2026
+- **Note:** Self-reported by ForgeCode; score reflects harness+model combination, not raw model capability

-> "I asked it to research the architecture of a small repo. Fifteen minutes. Kept going unstable, tool calls failing, the agent retrying and spinning. I killed it."
-
-### Tool Calling Reliability
- **Problem:** Persistent tool-call errors with GPT 5.4
- **ForgeCode Fixes Applied:**
-  1. Reordered JSON schema fields (`required` before `properties`)
-  2. Flattened nested schemas
-  3. Added explicit truncation reminders for partial file reads
- **Result:** These optimizations were benchmark-specific (described as "benchmaxxed")
+### SWE-Bench Pro
+- **Score:** 57.7% (Rank #3 overall)
+- **Behind:** Claude Mythos Preview (77.8%), GLM-5.1 (58.4%)
+- **Source:** llm-stats.com

 ---

-## Harness Optimizations for GPT 5.4
+## What Worked Well

-From ForgeCode's "Benchmarks Don't Matter" blog series:
+1. **Terminal Execution Speed**
+   - Fastest terminal execution among frontier models
+   - 47% token reduction with tool search
+   - Best price/performance ratio for terminal tasks
+
+2. **Benchmark Performance**
+   - High scores on Terminal-Bench with ForgeCode harness optimizations
+   - Strong reasoning capabilities on AIME 2025, HMMT, GPQA-Diamond
+
+---
+
+## Issues Encountered
+
+1. **Stability Problems** (Critical)
+   - **Description:** "Borderline unusable" for research tasks
+   - **Manifestation:** 15-minute research task on small repo failed repeatedly
+   - **Symptoms:** Tool calls failing, agent stuck in retry loops, required manual kill
+   - **Quote:** "I asked it to research the architecture of a small repo. Fifteen minutes. Kept going unstable, tool calls failing, the agent retrying and spinning. I killed it."
+
+2. **Tool Calling Reliability** (Major)
+   - **Description:** Persistent tool-call errors with GPT 5.4
+   - **ForgeCode Fixes Applied:**
+     - Reordered JSON schema fields (`required` before `properties`)
+     - Flattened nested schemas
+     - Added explicit truncation reminders for partial file reads
+   - **Note:** These optimizations were benchmark-specific ("benchmaxxed")
+
+3. **Long-Running Task Instability** (Major)
+   - **Description:** 15+ minute tasks became unstable
+   - **Impact:** Unpredictable failures requiring manual intervention
+
+---
+
+## Harness Optimizations
+
+ForgeCode applied specific optimizations for GPT 5.4:

 1. **Non-Interactive Mode:** System prompt rewritten to prohibit conversational branching
 2. **Tool Naming:** Renaming edit tool arguments to `old_string` and `new_string` (names appearing frequently in training data) measurably dropped tool-call error rates
@@ -50,19 +84,11 @@ From ForgeCode's "Benchmarks Don't Matter" blog series:

 ---

-## What Didn't Work Well
-
-1. **Research tasks:** Tool calling failures causing infinite loops
-2. **Long-running tasks:** 15+ minute tasks became unstable
-3. **Consistency:** Unpredictable failures requiring manual intervention
-
---
-
-## Comparison with Opus 4.6
+## Comparison with Claude Opus 4.6

 | Aspect | GPT 5.4 | Opus 4.6 |
 |--------|---------|----------|
-| TermBench 2.0 | 81.8% | 81.8% |
+| Terminal-Bench 2.0 (ForgeCode) | 81.8% | 81.8% |
 | Real-world stability | Poor | Excellent |
 | Tool calling reliability | Problematic | Reliable |
 | Research tasks | Unusable | Good |
@@ -73,5 +99,8 @@ From ForgeCode's "Benchmarks Don't Matter" blog series:

 ## Source References

-1. **DEV Community:** https://dev.to/liran_baba/forgecode-vs-claude-code-which-ai-coding-agent-actually-wins-36c
-2. **ForgeCode Blog:** https://forgecode.dev/blog/benchmarks-dont-matter/
+1. **DEV Community - ForgeCode vs Claude Code**: https://dev.to/liran_baba/forgecode-vs-claude-code-which-ai-coding-agent-actually-wins-36c
+   - Real-world performance comparison by Liran Baba
+
+2. **ForgeCode Blog - Benchmarks Don't Matter**: https://forgecode.dev/blog/benchmarks-dont-matter/
+   - Documentation of harness optimizations
@@ -1,6 +1,6 @@
-# Qwen 3.5 with ForgeCode - Feedback Report
+# Qwen Models with ForgeCode - Feedback Report

-**Model:** Qwen 3.5  
+**Models Covered:** Qwen 3.5, Qwen3  
 **Provider:** Alibaba Cloud (via local inference)  
 **Harness:** ForgeCode  
 **Source References:** GitHub Issue #2894, Reddit r/LocalLLaMA  
@@ -8,12 +8,24 @@

 ---

+## Model Reference Guide
+
+| Model Family | Available Sizes | Notes |
+|--------------|-----------------|-------|
+| **Qwen 3.5** | 0.8B, 2B, 4B, 9B (dense); 27B, 122B-A10B, 397B-A17B (MoE) | Released Feb 2026 |
+| **Qwen3** | 0.6B, 1.7B, 4B, 8B, 14B, 32B (dense); 30B-A3B, 235B-A22B (MoE) | Released April 2025 |
+| **Qwen2.5** | 0.5B, 1.5B, 3B, 7B, 14B, 32B, 72B + Coder variants | Earlier generation |
+
+> **Note:** References to "Qwen 3.5 14B" in community discussions likely mean Qwen3-14B or Qwen2.5-14B.
+
+---
+
 ## Known Issues

 ### Multiple System Messages Bug
 **GitHub Issue:** #2894 (Open as of April 8, 2026)

-**Problem:** Multiple system messages break models with strict chat templates (e.g., Qwen3.5)
+**Problem:** Multiple system messages break models with strict chat templates (e.g., Qwen3, Qwen 3.5)

 **Error Manifestation:**
 - Models with strict chat templates fail to parse message structure correctly
@@ -22,7 +34,7 @@

 **Impact:**
 - Affects local inference with llama.cpp, Ollama, and similar servers
- Qwen3.5 specifically mentioned as affected
+- Qwen3 and Qwen 3.5 specifically mentioned as affected

 **Workaround Status:** No official fix yet; issue under investigation

@@ -1,5 +1,7 @@
 # AGENTS.md

+**Last Updated:** April 9, 2026
+
 ## Research/Analysis Folder for hermes

 This is the research and analysis folder for the **hermes** coding harness.
@@ -0,0 +1,260 @@
+# Hermes Agent Repository Analysis: Local Model Suitability
+
+**Analysis Date:** 2026-04-09  
+**Based on:** Feedback from `feedback/localllm/`, `feedback/general/`, and source code in `repo/`
+
+---
+
+## Executive Summary
+
+Hermes Agent is **better suited for local models than alternatives** (OpenClaw), but has significant architectural challenges for smaller models (<14B parameters). The codebase shows intentional design for model-specific prompting, but the **fixed token overhead (~13.9K tokens)** and **tool schema complexity** create fundamental constraints for resource-constrained deployments.
+
+---
+
+## Strong Conclusions (Evidence-Based)
+
+### 1. Token Overhead is the Primary Local Model Killer
+
+**Evidence:** GitHub Issue #4379, feedback/general/terminal-bench-benchmarks.md
+
+| Component | Tokens | % of Overhead |
+|-----------|--------|---------------|
+| Tool definitions (31 tools) | ~8,759 | 63% |
+| System prompt | ~5,176 | 37% |
+| **Total Fixed Overhead** | **~13,935** | **100%** |
+
+**Impact on Local Models:**
+- A 4K context model has only ~20% of context available for actual task execution
+- Local models with 8K context are effectively reduced to 2-3K usable context
+- This is a **fundamental architectural constraint**, not a bug
+
+**Code Location:** `repo/toolsets.py`, `repo/model_tools.py` - tools are eagerly loaded into system prompt
+
+---
+
+### 2. Tool Call Argument Parsing Has Compatibility Bug with llama-server
+
+**Evidence:** feedback/localllm/qwen-models-feedback.md (Issue #1071)
+
+**Problem:** Line 8837 in `repo/run_agent.py`:
+```python
+if not args or not args.strip():  # <- crashes here
+```
+
+**Root Cause:** llama-server returns `tc.function.arguments` as a parsed `dict` instead of JSON string (OpenAI spec divergence). The code has partial handling for this at lines 8830-8831 but the `.strip()` check happens after the conversion attempt in some paths.
+
+**Fix Exists:** User-submitted fix confirmed working:
+```python
+if isinstance(args, (dict, list)):
+    tc.function.arguments = json.dumps(args)
+    continue  # Skip the strip() check after conversion
+```
+
+**Severity:** Critical - breaks llama-server/Ollama backends entirely
+
+---
+
+### 3. Smaller Models (4B-7B) Have Systematic Tool Calling Issues
+
+**Evidence:** feedback/localllm/qwen-models-feedback.md, feedback/localllm/general-local-llm-feedback.md
+
+**Specific Issues Reported:**
+- **Qwen 3.5 4B, Qwen 2.5 7B:** "tool calls work once then model forgets which tool to use"
+- **Gemma 4 27B:** Duplicates tool calls, gives up on complex challenges
+- **Pattern:** Tool use reliability correlates strongly with model size
+
+**Community Consensus:**
+| VRAM | Recommended Model | Tool Use Reliability |
+|------|------------------|---------------------|
+| 8GB | Qwen 3.5 4B | Inconsistent |
+| 16GB | Qwen 3.5 14B | Decent |
+| 24GB | Qwen 3.5 27B | Excellent (~25 t/s at 32K ctx) |
+
+**Code Relevance:** `repo/agent/prompt_builder.py` has `TOOL_USE_ENFORCEMENT_GUIDANCE` but it may not be sufficient for smaller models.
+
+---
+
+### 4. Context Length Configuration is Error-Prone
+
+**Evidence:** feedback/localllm/local-setup-issues.md
+
+**Common Error:** "Context exceeded your setting"
+
+**Root Cause:** Mismatch between:
+- Ollama's `num_ctx` setting
+- Hermes config `model.context_length`
+- Model's actual capability
+
+**Code Location:** `repo/agent/model_metadata.py` - has `query_ollama_num_ctx()` but users report confusion
+
+**Documentation Gap:** Users need to manually synchronize these values; no automatic detection/warning
+
+---
+
+### 5. Model-Specific Prompt Guidance Exists but May Be Insufficient
+
+**Evidence:** `repo/agent/prompt_builder.py` lines 173-276
+
+**Existing Model-Specific Guidance:**
+
+| Model Pattern | Guidance Applied |
+|--------------|------------------|
+| `gpt`, `codex`, `gemini`, `gemma`, `grok` | `TOOL_USE_ENFORCEMENT_GUIDANCE` |
+| `gpt-5`, `codex` | `DEVELOPER_ROLE_MODELS` (uses 'developer' role) |
+| OpenAI models | `OPENAI_MODEL_EXECUTION_GUIDANCE` (extensive) |
+| `gemini`, `gemma` | `GOOGLE_MODEL_OPERATIONAL_GUIDANCE` |
+
+**Assessment:** The prompts are well-crafted but **assume frontier-level comprehension**. Smaller models may be overwhelmed by the verbosity.
+
+---
+
+## Weaker Conclusions (Informed Conjecture)
+
+### 6. Skills System May Add Cognitive Load Beyond Token Count
+
+**Observation:** Skills are loaded as a structured index in the system prompt (~2,200 chars reported in memory stats).
+
+**Conjecture:** Beyond token cost, the **progressive disclosure** model (Level 0: names only, Level 1: full content when needed) may confuse smaller models. They see skill names but may not reliably understand when to load them via `skill_view()`.
+
+**Supporting Evidence:** No direct complaints about skills in local model feedback, but the pattern of "forgetting which tool to use" suggests similar issues with conditional tool usage.
+
+---
+
+### 7. Browser Tools are Expensive Dead Weight for Non-Browser Tasks
+
+**Evidence:** `repo/toolsets.py` - browser toolset adds ~1,258 tokens even when unused
+
+**Conjecture:** For local models on messaging platforms (Telegram, Discord, etc.), browser tools should be **lazily loaded** or disabled by default. The current eager inclusion hurts local model performance disproportionately.
+
+**Related Issue:** GitHub #4379 mentions "Platform-aware tool filtering" as a recommended optimization
+
+---
+
+### 8. Tool Schema Complexity May Exceed Small Model Parsing Capability
+
+**Observation:** Tools have rich JSON schemas with nested parameters, descriptions, and constraints.
+
+**Conjecture:** The schema complexity that works well for GPT-4/Claude may be **too verbose for 4B-7B models to parse reliably**. The models may see tool names but struggle to understand parameter requirements.
+
+**Supporting Evidence:** 
+- Qwen 4B "forgets which tool to use" after first call suggests schema comprehension issues
+- Gemma duplicates tool calls suggesting confusion about tool selection
+
+---
+
+### 9. The "hermes-cli" Toolset is Too Monolithic for Local Models
+
+**Observation:** `repo/toolsets.py` line 278-282 - `hermes-cli` includes all 31+ core tools via `_HERMES_CORE_TOOLS`
+
+**Conjecture:** There should be a **tiered toolset system** for local models:
+- `hermes-local-minimal`: 5-8 essential tools only
+- `hermes-local-standard`: 15-20 tools for 14B+ models
+- `hermes-cli`: Full toolset for frontier models
+
+---
+
+### 10. Context Compression Kicks In Too Late for Local Models
+
+**Observation:** `repo/agent/context_compressor.py` exists but triggers at high context pressure.
+
+**Conjecture:** For 8K context local models, compression should start **much earlier** (at 50% usage vs 80%) to preserve working memory for the actual task.
+
+---
+
+## Code Quality Assessment for Local Models
+
+### Good Patterns (Keep)
+
+| Pattern | Location | Why It Helps |
+|---------|----------|--------------|
+| Model-specific guidance | `prompt_builder.py:188-191` | Allows tailored prompting |
+| Tool argument coercion | `model_tools.py:372-408` | Handles string→type conversion |
+| Async bridging | `model_tools.py:44-126` | Prevents event loop issues |
+| Skills caching | `prompt_builder.py:370-438` | Reduces filesystem overhead |
+| Platform hints | `prompt_builder.py:285-358` | Adapts output format |
+
+### Problematic Patterns (Fix)
+
+| Pattern | Location | Issue |
+|---------|----------|-------|
+| Eager tool loading | `model_tools.py:132-184` | All tools loaded at startup |
+| `.strip()` on args | `run_agent.py:8837` | Crashes with dict args from llama-server |
+| Monolithic toolset | `toolsets.py:278` | No granularity for resource constraints |
+| Fixed skill prompt | `prompt_builder.py:730-744` | Always includes full skill guidance |
+
+---
+
+## Recommendations for Local Model Optimization
+
+### High Priority (Strong Evidence)
+
+1. **Fix llama-server argument parsing** (Issue #1071)
+   - Ensure dict/list args are converted before `.strip()` check
+   - Add type checking at all argument access points
+
+2. **Document minimum viable toolset**
+   - Create `hermes-local-minimal` toolset with: `terminal`, `file`, `read_file`, `write_file`, `patch`, `search_files`
+   - Reduces token overhead by ~60%
+
+3. **Add context length validation**
+   - Warn users when context setting mismatch detected
+   - Query Ollama's `num_ctx` and compare to Hermes config
+
+### Medium Priority (Informed Conjecture)
+
+4. **Implement lazy skill loading**
+   - Don't include skill index in system prompt
+   - Only load skill descriptions when explicitly referenced
+
+5. **Create model-size-aware defaults**
+   - Detect model size (if possible) or add `model_size: small|medium|large` config
+   - Adjust max_iterations, context compression thresholds, toolset accordingly
+
+6. **Add tool call validation retry**
+   - For models with known tool-calling issues, add client-side retry with nudging
+
+### Low Priority (Speculative)
+
+7. **Explore tool schema compression**
+   - Abbreviated parameter names for local models
+   - Reduced description verbosity
+
+8. **Consider tool-use training data**
+   - Partner with Unsloth/Qwen teams for tool-call fine-tuning on Hermes patterns
+
+---
+
+## Summary Table: Component Suitability for Local Models
+
+| Component | Rating | Notes |
+|-----------|--------|-------|
+| **Tool Architecture** | ⭐⭐⭐ | Well-designed but too monolithic |
+| **Prompt Engineering** | ⭐⭐⭐⭐ | Good model-specific guidance |
+| **Token Efficiency** | ⭐⭐ | ~14K overhead is prohibitive |
+| **Error Handling** | ⭐⭐⭐ | Good but missing llama-server edge case |
+| **Configuration** | ⭐⭐ | Context sync is manual/error-prone |
+| **Skills System** | ⭐⭐⭐⭐ | Excellent design but eager loading hurts |
+| **Documentation** | ⭐⭐⭐ | Local setup docs improving |
+
+---
+
+## Feedback Source References
+
+1. **Qwen Models:** `feedback/localllm/qwen-models-feedback.md`
+2. **Gemma Models:** `feedback/localllm/gemma-models-feedback.md`
+3. **General Local LLM:** `feedback/localllm/general-local-llm-feedback.md`
+4. **Local Setup Issues:** `feedback/localllm/local-setup-issues.md`
+5. **Bug Reports:** `feedback/general/bug-reports-and-issues.md`
+6. **Feature Feedback:** `feedback/general/feature-feedback.md`
+7. **Benchmarks:** `feedback/general/terminal-bench-benchmarks.md`
+
+---
+
+## Code References
+
+- **Prompt Building:** `repo/agent/prompt_builder.py`
+- **Tool Registry:** `repo/tools/registry.py`
+- **Tool Orchestration:** `repo/model_tools.py`
+- **Toolsets:** `repo/toolsets.py`
+- **Agent Loop:** `repo/run_agent.py` (lines 8800-8900 for tool validation)
+- **Skill Utilities:** `repo/agent/skill_utils.py`
@@ -1,12 +1,70 @@
 # Claude Sonnet Feedback for Hermes Agent

-**Source reference:** GitHub issues, community discussions, official docs
+**Model:** Claude Sonnet 4.5/4.6  
+**Provider:** Anthropic  
+**Harness:** Hermes  
+**Date Compiled:** April 9, 2026  
+**Source References:** GitHub issues, community discussions, official docs

 ---

-## Claude Sonnet 4.5/4.6 - Primary Recommendation
+## Quick Reference

-**Status:** Excellent performance, commonly used as default
+| Attribute | Value |
+|-----------|-------|
+| Model | Claude Sonnet 4.5/4.6 |
+| Provider | Anthropic |
+| Status | Primary recommendation for Hermes |
+| Best For | Complex reasoning, multi-step tasks |
+| Cost | ~$3-5/M input, ~$15-25/M output |
+
+---
+
+## Benchmark Results
+
+No specific Terminal-Bench or SWE-Bench results available for Hermes + Claude Sonnet combination.
+
+---
+
+## What Worked Well
+
+1. **Excellent Tool Calling Reliability**
+   - Strong performance on complex multi-step tasks
+   - Good context understanding
+
+2. **Performance vs OpenClaw**
+   - Source: https://www.buildmvpfast.com/blog/hermes-agent-v04-open-source-agent-infrastructure-2026
+   - > "One developer reported that a task taking OpenClaw 50+ tool calls and steps took Hermes 5 correct tool calls and finished 2.5 minutes faster."
+
+3. **Cost Efficiency for Complex Tasks**
+   - Fewer tool calls required compared to other agents
+   - More efficient execution path
+
+---
+
+## Issues Encountered
+
+1. **Token Overhead** (Major - Affects All Models)
+   - **Finding:** 73% of every API call is fixed overhead (~13.9K tokens)
+   - **Breakdown:**
+     | Component | Tokens | % of Request |
+     |-----------|--------|--------------|
+     | Tool definitions (31 tools) | 8,759 | 46.1% |
+     | System prompt (SOUL.md + skills) | 5,176 | 27.2% |
+     | Messages (conversation) | 3,000-8,775 | 26.7% avg |
+     | **Total per request** | **~17,000-23,000** | |
+   - **Impact:** This overhead is constant regardless of using Sonnet, Haiku, Llama, or any OpenRouter model
+
+2. **High Token Usage in Practice**
+   - **Quote:** "4 million tokens in 2 hours of light usage" — Reddit user who quit
+   - **High-token triggers:**
+     - Terminal tool spawning
+     - Browser automation with screenshots
+     - Complex code execution with large file reads
+
+---
+
+## Cost Analysis

 ### Token Usage Reality Check

@@ -33,30 +91,7 @@

 ---

-## Token Overhead Analysis (All Models)
-
-**Critical Finding:** 73% of every API call is fixed overhead (~13.9K tokens)
-
-| Component | Tokens | % of Request |
-|-----------|--------|--------------|
-| Tool definitions (31 tools) | 8,759 | 46.1% |
-| System prompt (SOUL.md + skills) | 5,176 | 27.2% |
-| Messages (conversation) | 3,000-8,775 | 26.7% avg |
-| **Total per request** | **~17,000-23,000** | |
-
-**Impact:** This overhead is constant regardless of using Sonnet, Haiku, Llama, or any OpenRouter model.
-
---
-
-## Performance Comparison
-
-**Source:** https://www.buildmvpfast.com/blog/hermes-agent-v04-open-source-agent-infrastructure-2026
-
-> "One developer reported that a task taking OpenClaw 50+ tool calls and steps took Hermes 5 correct tool calls and finished 2.5 minutes faster."
-
---
-
-## Best Practices for Cost Management
+## Best Practices

 ### 1. Use Cheaper Models for Routine Tasks

@@ -82,31 +117,7 @@ Start fresh for unrelated tasks:
 hermes --fresh
 ```

---
-
-## User Experience Feedback
-
-### Positive
-
- Excellent tool calling reliability
- Strong reasoning for complex multi-step tasks
- Good context understanding
-
-### Cost Concerns
-
-**Quote from Reddit user:**
-> "4 million tokens in 2 hours of light usage" — Reddit user who quit
-
-**High-token triggers:**
- Terminal tool spawning
- Browser automation with screenshots
- Complex code execution with large file reads
-
---
-
-## Configuration Tips
-
-### Auxiliary Vision Model
+### 4. Auxiliary Vision Model

 For vision tasks, consider using a cheaper model:
 ```yaml
@@ -125,6 +136,28 @@ auxiliary:

 ---

+## Configuration
+
+No special configuration required for Claude Sonnet with Hermes. Default settings work well.
+
+---
+
+## Source References
+
+1. **Hermes Agent Token Overhead Blog**: https://hermes-agent.ai/blog/hermes-agent-token-overhead
+   - Fixed overhead analysis and cost breakdown
+
+2. **Build MVP Fast - Hermes Agent v0.4**: https://www.buildmvpfast.com/blog/hermes-agent-v04-open-source-agent-infrastructure-2026
+   - Performance comparison with OpenClaw
+
+3. **GitHub Issue #4379**: Hermes repository
+   - Real-world usage example with token counts
+
+4. **Reddit r/LocalLLaMA**: Community discussions
+   - User experiences and cost concerns
+
+---
+
 ## Summary

 Claude Sonnet provides excellent performance with Hermes Agent but users should be aware of:
@@ -1,6 +1,22 @@
 # OpenAI GPT Models Feedback for Hermes Agent

-**Source reference:** Official docs, community discussions, blog posts
+**Models Covered:** GPT-4o, GPT-4o-mini, GPT-5 series, o1/o3, Codex  
+**Provider:** OpenAI  
+**Harness:** Hermes  
+**Date Compiled:** April 9, 2026  
+**Source References:** Official docs, community discussions, blog posts
+
+---
+
+## Quick Reference
+
+| Model | Best For | Cost Tier |
+|-------|----------|-----------|
+| GPT-5.4 | Complex reasoning, high accuracy | High |
+| GPT-4o | Balanced performance | Medium |
+| GPT-4o-mini | Routine tasks, cost efficiency | Low |
+| o1/o3 | Reasoning tasks | High |
+| Codex | Coding with vision | Medium (via ChatGPT Pro) |

 ---

@@ -1,10 +1,26 @@
 # Gemma Models Feedback for Hermes Agent

-**Source reference:** Reddit r/LocalLLaMA, HuggingFace blog, community discussions
+**Models Covered:** Gemma 4 (26B A4B)  
+**Provider:** Ollama, llama.cpp  
+**Harness:** Hermes  
+**Date Compiled:** April 9, 2026  
+**Source References:** Reddit r/LocalLLaMA, HuggingFace blog, community discussions

 ---

-## Gemma 4 Support
+## Quick Reference
+
+| Attribute | Value |
+|-----------|-------|
+| Model | Gemma 4 26B A4B |
+| Size | 26B parameters |
+| Quantization | Q8_0 recommended |
+| Best For | Conversational use, creative tasks |
+| Not Recommended For | Complex agentic tasks (per community feedback) |
+
+---
+
+## Gemma 4 Support Status

 **Status:** Day-0 ecosystem support confirmed

@@ -14,36 +30,79 @@

 ---

-## Gemma 4 vs Qwen 3.5 Comparison
+## Benchmark Results
+
+No specific benchmark results available for Hermes + Gemma 4 combination.
+
+---
+
+## What Worked Well
+
+1. **Ecosystem Support**
+   - Day-0 support confirmed by HuggingFace
+   - Works with Hermes, OpenClaw, pi, and OpenCode
+
+2. **Performance on Apple Silicon**
+   - Gemma 4 26B A4B Q8_0 on M2 Ultra achieves ~300 t/s
+   - Note: With speculative decoding caveats
+
+3. **Conversational Quality**
+   - "Gemma is pretty fun to talk to, reminds me of the early model whimsy."
+   - Good for creative writing tasks
+
+---
+
+## Issues Encountered
+
+1. **Tool Call Duplication** (Major)
+   - **Description:** Gemma keeps duplicating tool calls
+   - **Quote:** "Gemma keeps duplicating tool calls for some reason."
+   - **Source:** https://www.reddit.com/r/LocalLLaMA/comments/1scbpmo/so_qwen35_or_gemma_4/
+
+2. **Complex Task Completion** (Major)
+   - **Description:** Fails to complete complex challenges that Qwen can succeed at
+   - **Quote:** "Fixes for llama.cpp are happening in real-time so things may not be fair but so far Gemma is failing to complete the complex challenge which qwen can succeed at (24gb VRAM) it's just giving up and claiming it's succeeded when it hasn't."
+   - **Hardware:** 24GB VRAM
+
+3. **llama.cpp Maturity** (Minor)
+   - Support actively being fixed in real-time
+   - May improve with future updates
+
+---
+
+## Comparison: Gemma 4 vs Qwen 3.5

 **Source:** https://www.reddit.com/r/LocalLLaMA/comments/1scbpmo/so_qwen35_or_gemma_4/

-### Tool Use Issues
+| Aspect | Gemma 4 | Qwen 3.5 |
+|--------|---------|----------|
+| Tool use with novel tools | Duplicates calls | Works well |
+| Complex challenges | Gives up/fails | Succeeds |
+| Conversational | Fun, whimsical | - |
+| Agent reliability | Lower | Higher |

-> "Gemma keeps duplicating tool calls for some reason."
-
-> "Gemma is pretty fun to talk to, reminds me of the early model whimsy."
-
-> "Fixes for llama.cpp are happening in real-time so things may not be fair but so far Gemma is failing to complete the complex challenge which qwen can succeed at (24gb VRAM) it's just giving up and claiming it's succeeded when it hasn't."
+**Community Consensus:** For Hermes Agent specifically, Qwen 3.5 currently outperforms Gemma 4 for tool use and complex tasks.

 ---

-## Performance Notes
+## Recommendations

- Gemma 4 26B A4B Q8_0 on M2 Ultra achieves ~300 t/s (with speculative decoding caveats)
- llama.cpp support actively being fixed in real-time
- Better for conversational use than complex agentic tasks
+### Use Gemma 4 For:
+- Conversational interactions
+- Creative writing tasks
+- When llama.cpp optimizations mature

---
-
-## Recommendation
-
-For Hermes Agent specifically, community feedback suggests Qwen 3.5 currently outperforms Gemma 4 for:
+### Use Qwen 3.5 Instead For:
 - Tool use with novel tools
 - Complex multi-step tasks
 - Agent reliability

-Gemma 4 may be preferable for:
- Conversational interactions
- Creative writing tasks
- When llama.cpp optimizations mature
+---
+
+## Source References
+
+1. **HuggingFace Blog - Gemma 4**: https://huggingface.co/blog/gemma4
+   - Day-0 ecosystem support announcement
+
+2. **Reddit r/LocalLLaMA - Qwen vs Gemma**: https://www.reddit.com/r/LocalLLaMA/comments/1scbpmo/so_qwen35_or_gemma_4/
+   - Community comparison and tool use feedback
@@ -1,42 +1,91 @@
 # Qwen Models Feedback for Hermes Agent

-**Source reference:** Multiple Reddit r/LocalLLaMA posts, GitHub issues, community discussions
+**Models Covered:** Qwen 3.5 (4B, 27B), Qwen 2.5 (7B)  
+**Provider:** Ollama, Unsloth  
+**Harness:** Hermes  
+**Date Compiled:** April 9, 2026  
+**Source References:** Reddit r/LocalLLaMA, GitHub issues, community discussions

 ---

-## Model: Qwen 3.5 (Various Sizes)
+## Model Reference Guide

-### Qwen 3.5 27B - Highly Recommended
+| Model Family | Available Sizes | Type | Notes |
+|--------------|-----------------|------|-------|
+| **Qwen 3.5** | 0.8B, 2B, 4B, 9B | Dense | Released Feb 2026 |
+| **Qwen 3.5** | 27B, 122B-A10B, 397B-A17B | MoE | Released Feb 2026 |
+| **Qwen3** | 0.6B, 1.7B, 4B, 8B, 14B, 32B | Dense | Released April 2025 |
+| **Qwen3** | 30B-A3B, 235B-A22B | MoE | Released April 2025 |
+| **Qwen2.5** | 0.5B, 1.5B, 3B, 7B, 14B, 32B, 72B | Dense | + Coder variants |

-**Hardware:** Dual 3090s with UD_5XL quant from Unsloth  
-**Performance:** ~25 t/s at 32k context  
-**Source:** https://www.reddit.com/r/LocalLLaMA/comments/1ro9lph/anybody_who_tried_hermesagent/
+> **Note:** References to "Qwen 3.5 14B" in community discussions likely mean Qwen3-14B or Qwen2.5-14B.

-> "The go to model for intelligence on decent hardware is qwen 3.5 27B, if you have two 3090s, use the UD_5XL quant from unsloth - its amazing. You will get about 25 t/s with this one, at a contex size of 32k, which is perfect."
+---

-### Tool Calling Performance
+## Quick Reference

-**Issue:** Tool calls work once then model forgets which tool to use  
-**Models affected:** Qwen 3.5 4B, Qwen 2.5 7B  
-**Source:** https://www.reddit.com/r/LocalLLaMA/comments/1s4yy6o/best_model_for_hermesagent/
+| VRAM | Recommended Model | Notes |
+|------|------------------|-------|
+| 8GB | Qwen 3.5 4B | Tool calling may be inconsistent |
+| 24GB | Qwen 3.5 27B (Q4_K_M) | Excellent tool use, 25 t/s |
+| 48GB+ | Qwen 3.5 27B UD_5XL | Best quality, ~25 t/s at 32k ctx |

-> "I use ollama and qwen3.5:4b qwen2.5:7b and they all tool call once than they forget which one to use any recomendations for other models?"
+---

-**User hardware:** 8GB VRAM
+## Qwen 3.5 27B (MoE)

-### Qwen vs Gemma 4 Comparison
+### Configuration
+- **Hardware:** Dual 3090s recommended
+- **Quantization:** UD_5XL from Unsloth
+- **Performance:** ~25 t/s at 32k context
+- **Type:** MoE (Mixture-of-Experts)

-**Source:** https://www.reddit.com/r/LocalLLaMA/comments/1scbpmo/so_qwen35_or_gemma_4/
+### Benchmark Results
+No specific benchmark results available for Hermes + Qwen 3.5 combination.

-> "For me Qwen is working significantly better for tool use with novel tools (things unlike what you'd expect in OpenCode or Claude Code). Gemma keeps duplicating tool calls for some reason."
+### What Worked Well
+1. **Highly Recommended by Community**
+   - **Quote:** "The go to model for intelligence on decent hardware is qwen 3.5 27B, if you have two 3090s, use the UD_5XL quant from unsloth - its amazing. You will get about 25 t/s with this one, at a contex size of 32k, which is perfect."
+   - **Source:** https://www.reddit.com/r/LocalLLaMA/comments/1ro9lph/anybody_who_tried_hermesagent/

-> "Gemma is failing to complete the complex challenge which qwen can succeed at (24gb VRAM) it's just giving up and claiming it's succeeded when it hasn't."
+2. **Tool Use Performance**
+   - Works significantly better than Gemma for tool use with novel tools
+   - Better completion of complex challenges
+
+### Issues Encountered
+No major issues reported for 27B variant.
+
+---
+
+## Qwen 3.5 4B & Qwen 2.5 7B
+
+### Configuration
+- **Provider:** Ollama
+- **User Hardware:** 8GB VRAM
+
+### Benchmark Results
+No specific benchmark results available.
+
+### What Worked Well
+1. **Accessible for Consumer Hardware**
+   - Runs on 8GB VRAM systems
+   - Good entry point for local LLM usage
+
+### Issues Encountered
+1. **Tool Calling Consistency** (Major)
+   - **Description:** Tool calls work once then model forgets which tool to use
+   - **Models Affected:** Qwen 3.5 4B, Qwen 2.5 7B
+   - **Quote:** "I use ollama and qwen3.5:4b qwen2.5:7b and they all tool call once than they forget which one to use any recomendations for other models?"
+   - **Source:** https://www.reddit.com/r/LocalLLaMA/comments/1s4yy6o/best_model_for_hermesagent/

 ---

 ## llama-server (llama.cpp) Compatibility Issue

-**Issue #1071:** Critical bug with llama-server/Ollama backend
+### Issue #1071: Tool Call Argument Validation
+
+**Severity:** Critical  
+**Status:** User-submitted fix confirmed working

 **Error:** `'dict' object has no attribute 'strip'` during tool call argument validation

@@ -55,11 +104,9 @@ if isinstance(args, (dict, list)):
    tc.function.arguments = json.dumps(args)
 ```

-**Status:** User-submitted fix confirmed working
-
 ---

-## Best Practices for Local Models
+## Best Practices

 ### Context Length Configuration

@@ -69,17 +116,16 @@ if isinstance(args, (dict, list)):

 **Source:** https://hermes-agent.nousresearch.com/docs/reference/faq

-### Model Recommendations by VRAM
+### General Recommendations

-| VRAM | Recommended Model | Notes |
-|------|------------------|-------|
-| 8GB | Qwen 3.5 4B | Tool calling may be inconsistent |
-| 24GB | Qwen 3.5 27B (Q4_K_M) | Excellent tool use, 25 t/s |
-| 48GB+ | Qwen 3.5 27B UD_5XL | Best quality, ~25 t/s at 32k ctx |
+1. **Context exceeded errors** are common with default settings
+2. **Manually configure context length** to match model capabilities
+3. **Tool calling reliability** varies significantly by model size
+4. Use larger models (27B+) for best tool use performance

 ---

-## General Local Model Feedback
+## Community Feedback

 **Positive:**
 - "Hermes agent already works way way better than Open Claw and it actually works pretty well locally"
@@ -89,3 +135,34 @@ if isinstance(args, (dict, list)):
 - Context exceeded errors common with default settings
 - Need to manually configure context length to match model capabilities
 - Tool calling reliability varies significantly by model size
+
+---
+
+## Comparison: Qwen vs Gemma 4
+
+**Source:** https://www.reddit.com/r/LocalLLaMA/comments/1scbpmo/so_qwen35_or_gemma_4/
+
+> "For me Qwen is working significantly better for tool use with novel tools (things unlike what you'd expect in OpenCode or Claude Code). Gemma keeps duplicating tool calls for some reason."
+
+> "Gemma is failing to complete the complex challenge which qwen can succeed at (24gb VRAM) it's just giving up and claiming it's succeeded when it hasn't."
+
+**Conclusion:** Qwen preferred for tool use and complex challenges.
+
+---
+
+## Source References
+
+1. **Reddit r/LocalLLaMA - Hermes Agent Discussion**: https://www.reddit.com/r/LocalLLaMA/comments/1ro9lph/anybody_who_tried_hermesagent/
+   - Qwen 3.5 27B recommendation
+
+2. **Reddit r/LocalLLaMA - Best Model for Hermes**: https://www.reddit.com/r/LocalLLaMA/comments/1s4yy6o/best_model_for_hermesagent/
+   - Tool calling issues with smaller models
+
+3. **Reddit r/LocalLLaMA - Qwen vs Gemma**: https://www.reddit.com/r/LocalLLaMA/comments/1scbpmo/so_qwen35_or_gemma_4/
+   - Model comparison for tool use
+
+4. **Hermes FAQ**: https://hermes-agent.nousresearch.com/docs/reference/faq
+   - Context length configuration guidance
+
+5. **GitHub Issue #1071**: Hermes repository
+   - llama-server compatibility fix
@@ -1,5 +1,7 @@
 # AGENTS.md

+**Last Updated:** April 9, 2026
+
 ## Research/Analysis Folder for opencode

 This is the research and analysis folder for the **opencode** coding harness.
@@ -0,0 +1,350 @@
+# OpenCode Repository Analysis: Local Model Compatibility
+
+**Date:** April 9, 2026  
+**Analysis Focus:** Prompts, Tools, Parsing, and Skills for Local/Smaller Models  
+**Source:** Code analysis of `opencode-ai/opencode` + Community feedback from GitHub, Reddit, Discord
+
+---
+
+## Executive Summary
+
+OpenCode is a Go-based coding agent with heavy optimization for frontier models (Claude, GPT-4o). The codebase shows **strong architectural decisions** for general use but has **specific pain points for local models** that manifest as JSON parsing errors, tool calling failures, and context truncation issues.
+
+**Verdict:** Works well with local models 27B+ (Qwen3.5 27B, Gemma 4 26B) with configuration adjustments. Smaller models (7B-14B) struggle due to prompt complexity and tool count.
+
+---
+
+## 1. PROMPTS Analysis
+
+### 1.1 Prompt Structure
+
+OpenCode uses **provider-specific prompts** with significant differences between Anthropic and OpenAI formats:
+
+| File | Purpose | Lines | Strength for Local Models |
+|------|---------|-------|---------------------------|
+| `prompt/coder.go` | Main coding agent | ~220 | ⚠️ VERBOSE - Complex instructions |
+| `prompt/task.go` | Sub-agent (search) | ~17 | ✅ GOOD - Minimal, focused |
+| `prompt/summarizer.go` | Session summary | ~16 | ✅ GOOD - Simple directive |
+| `prompt/title.go` | Session titling | ~13 | ✅ GOOD - Single task |
+
+### 1.2 The Coder Prompt (Critical Analysis)
+
+**Location:** `internal/llm/prompt/coder.go`
+
+The `baseAnthropicCoderPrompt` is **excessively verbose** (~170 lines of instructions). Key sections:
+
+```go
+// Sections that add token overhead:
+- Tone and style (lines 86-93): ~400 tokens of verbosity constraints
+- Examples section (lines 94-135): Multiple <example> blocks
+- Proactiveness guidelines (lines 137-142)
+- Following conventions (lines 144-149)
+- Code style rules (lines 151-152)
+- Doing tasks workflow (lines 155-162)
+- Tool usage policy (lines 163-166)
+```
+
+**Strong Conclusion (Verified):** 
+- The prompt is designed for models with strong instruction-following (Claude 3.5+, GPT-4o)
+- Local models 14B and smaller struggle to retain all constraints
+- Community feedback confirms: "Qwen 3 14b fails" while "Qwen 3.5 27b works well"
+
+**Weak Conclusion (Inference):**
+- The verbosity may cause "instruction dilution" where smaller models fixate on early/late instructions and miss middle constraints
+- The example-heavy format (6+ examples) may be over-optimizing for frontier models
+
+### 1.3 What Works Well
+
+✅ **Provider-aware prompting** - Different prompts for Anthropic vs OpenAI endpoints  
+✅ **Environment injection** - Dynamic context (working dir, git status, platform, date)  
+✅ **Project-specific context** - Auto-loading from `OpenCode.md` or configured paths  
+✅ **LSP integration hints** - Conditional diagnostics info only when LSP available  
+
+### 1.4 Problems for Local Models
+
+❌ **Excessive constraints** - "You MUST..." appears 8+ times, creating conflicting priorities  
+❌ **Nested conditionals** - "If X then Y unless Z in which case..." structure  
+❌ **Implicit dependencies** - Assumes model can track multiple tool calls across turns  
+
+### 1.5 Community Evidence
+
+> "Local models are more for vibe coding. Not really set for agentic coding. Unless you can host minimax2.5 to actually be worthwhile." — Reddit r/opencodeCLI
+
+> "Qwen 3 14b - fails with hallucinations" vs "Qwen 3.5 27b Q3_XXS - 5.0% migration error, clear winner for local use" — Rost Glukhov benchmark
+
+---
+
+## 2. TOOLS Analysis
+
+### 2.1 Tool Inventory
+
+**Coder Agent Tools:** 11 core tools
+
+| Tool | Description Length | Params | Risk for Local Models |
+|------|-------------------|--------|----------------------|
+| `bash` | ~200 lines | 2 | ⚠️ HIGH - Complex bash description with git/PR instructions |
+| `edit` | ~90 lines | 3 | ⚠️ MEDIUM - Requires precise string matching |
+| `write` | ~60 lines | 2 | ✅ LOW - Straightforward |
+| `view` | ~70 lines | 3 | ✅ LOW - Well-documented |
+| `glob` | ~40 lines | 1 | ✅ LOW - Simple |
+| `grep` | ~80 lines | 4 | ⚠️ MEDIUM - Regex/literal_text nuance |
+| `ls` | ~30 lines | 2 | ✅ LOW - Simple |
+| `fetch` | ~40 lines | 1 | ✅ LOW - Simple |
+| `patch` | ~50 lines | 2 | ⚠️ MEDIUM - Requires understanding diff format |
+| `sourcegraph` | ~30 lines | 1 | ✅ LOW - Simple |
+| `diagnostics` | ~20 lines | 1 | ✅ LOW - Simple |
+| `agent` | ~40 lines | 1 | ⚠️ MEDIUM - Meta-cognitive (sub-agent) |
+
+### 2.2 Tool Description Problems
+
+**CRITICAL ISSUE: Bash Tool Description**
+
+Location: `internal/llm/tools/bash.go` lines 57-203
+
+The bash tool description is **excessively long** (~3500 characters) and includes:
+- Directory verification steps
+- Security check procedures  
+- Command execution flow
+- Output processing rules
+- Git commit workflow (lines 97-151)
+- PR creation workflow (lines 153-199)
+
+**Strong Conclusion (Verified):**
+- Community reports "invalid tool call message with wrong tool name" errors
+- GitHub Issue #13982: GLM-5 "screwing up the JSON parsing" specifically on read tool
+
+**Weak Conclusion (Inference):**
+- The tool descriptions may exceed effective context window for 8K-16K models when combined with prompts
+- Local models may "lose track" of which tool they're calling due to description overload
+
+### 2.3 Tool Calling Issues (Community Verified)
+
+GitHub Issue #4428 (36 comments): "Why is opencode not working with local llms via Ollama?"
+
+> "After many issues with Ollama (mostly that all models default to a very small context window and you have to modify them or find versions with bigger context window settings, and tool call formatting issues), after installing LM Studio I was able to consistently use qwen/qwen3-30b-a3b-2507 with tools"
+
+GitHub Issue #13982: "[bug] GLM 5 keeps screwing up the json parsing of read tool calling"
+
+> "The AI keeps screwing up the JSON formatting for the tool calling. Sometimes I even get 'Method Not Allowed' errors that stops the build dead in the tracks."
+
+---
+
+## 3. PARSING Analysis
+
+### 3.1 Tool Call Parsing Strategy
+
+**Location:** `internal/llm/provider/openai.go`
+
+OpenCode relies on **native function calling** via the OpenAI SDK:
+
+```go
+func (o *openaiClient) toolCalls(completion openai.ChatCompletion) []message.ToolCall {
+    // Extracts tool calls from API response
+    // Assumes provider returns well-formed JSON
+}
+```
+
+**Strong Conclusion (Verified):**
+- Uses standard OpenAI function calling format (works with llama.cpp, vLLM, Ollama)
+- No custom JSON parsing for tool arguments (relies on SDK/provider)
+
+### 3.2 The Problem: Local Model Output
+
+**Weak Conclusion (Inference from patterns):**
+
+Local models often produce:
+1. **Malformed JSON** - Trailing commas, unescaped quotes, missing braces
+2. **Partial tool calls** - Starting JSON but not completing before max_tokens
+3. **Invalid tool names** - Hallucinating tools that don't exist
+4. **Parameter type mismatches** - Sending strings where numbers expected
+
+**The codebase has NO resilience for:**
+- JSON repair/relaxation
+- Partial tool call streaming recovery
+- Tool name fuzzy matching
+- Parameter coercion
+
+### 3.3 Context Window Truncation
+
+**Strong Conclusion (Verified):**
+
+GitHub Issue #1212: "Fetched documentation exceeds context window limit"
+
+> "When opencode pulls documentation from websites, the resulting response can sometimes exceed the context length of the current model in use (currently Claude Sonnet 4 for me). It's impossible to continue this session in this case."
+
+**Config gap:** `internal/llm/models/local.go` sets:
+```go
+ContextWindow: cmp.Or(model.LoadedContextLength, 4096),  // Falls back to 4K!
+```
+
+Community fix (Medium article): Must increase Ollama context from 4K to 32K for reasonable performance.
+
+---
+
+## 4. SKILLS / SUB-AGENTS Analysis
+
+### 4.1 Agent Tool Architecture
+
+**Location:** `internal/llm/agent/agent-tool.go`
+
+```go
+const AgentToolName = "agent"
+
+// Description emphasizes:
+// - Parallel execution (good)
+// - Stateless operation
+// - Read-only (no bash/edit/write)
+```
+
+**Strong Conclusion (Verified):**
+- Sub-agents use **TaskPrompt** (minimal) vs **CoderPrompt** (verbose)
+- Task agent only gets: Glob, Grep, LS, Sourcegraph, View
+- This is actually **good design** - search tasks don't need editing tools
+
+### 4.2 KV Cache Invalidation Issue
+
+**Strong Conclusion (Verified):**
+
+Reddit r/LocalLLaMA:
+
+> "I tried opencode, it also works fine with qwen models but the kv cache was invalidated when working with gpt 120B model."
+
+> "When a new sub agent is spun, the kv cache from parent is not reused so for the sub agent model processed the whole prompt again."
+
+This is an **architectural limitation** - each agent spawns a new session/context.
+
+### 4.3 Sub-Agent Loop Risk
+
+**Weak Conclusion (Inference):**
+
+The agent tool description says:
+> "The agent's outputs should generally be trusted"
+
+For local models, this trust may be misplaced:
+- Sub-agent may return incomplete search results
+- No verification loop in parent agent
+- Can lead to cascading errors
+
+---
+
+## 5. LOCAL MODEL CONFIGURATION
+
+### 5.1 Auto-Discovery (Good)
+
+**Location:** `internal/llm/models/local.go`
+
+```go
+// Automatically discovers models from:
+// - v1/models endpoint (OpenAI compatible)
+// - api/v0/models endpoint (LM Studio)
+// Sets defaults for all agents
+```
+
+✅ **Works well** - No manual model registration needed for local endpoints
+
+### 5.2 Context Window Defaults (Bad)
+
+```go
+ContextWindow: cmp.Or(model.LoadedContextLength, 4096),
+```
+
+❌ **4K fallback is too small** for the verbose prompts + tool descriptions
+
+Community workaround:
+```json
+// ~/.config/opencode/opencode.json
+{
+  "provider": {
+    "ollama": {
+      "models": {
+        "qwen3:32b": {
+          "contextLength": 32768  // Manually override
+        }
+      }
+    }
+  }
+}
+```
+
+---
+
+## 6. RECOMMENDATIONS
+
+### 6.1 Strong Recommendations (Based on Verified Feedback)
+
+1. **Use 27B+ models minimum** for reliable tool calling
+   - Qwen 3.5 27B (Q3_XXS or Q4_K_XL)
+   - Gemma 4 26B (IQ4_XS)
+   - Avoid 14B and smaller for complex tasks
+
+2. **Set context window to 32K minimum** for local models
+   - Default 4K is insufficient for OpenCode's verbose prompts
+
+3. **Use LM Studio or llama.cpp over Ollama** for better tool calling
+   - Community reports more consistent results
+   - May relate to chat template handling
+
+4. **Correct chat templates required**
+   - Default Qwen3.5 template causes 500 errors
+   - Use corrected template from community gist
+
+### 6.2 Weak Recommendations (Inferred from Analysis)
+
+1. **Prompt compression** would benefit local models:
+   - Remove redundant examples from CoderPrompt
+   - Shorten tool descriptions (especially bash)
+   - Consider "instruction hierarchy" formatting
+
+2. **Tool description tiering**:
+   - "Essential" tools for small models
+   - "Extended" tools for large models
+
+3. **JSON resilience layer**:
+   - Partial JSON repair
+   - Tool name fuzzy matching
+   - Parameter type coercion
+
+### 6.3 Architecture Observations
+
+**Good for Local Models:**
+- Stateless sub-agents prevent context overflow
+- Minimal TaskPrompt for search operations
+- Auto-discovery of local endpoints
+- Provider abstraction allows local/remote mixing
+
+**Challenging for Local Models:**
+- No prompt compression/tiering
+- No tool subset selection
+- No JSON repair for malformed calls
+- No KV cache sharing between agents
+
+---
+
+## 7. BENCHMARK DATA
+
+From Rost Glukhov's testing (March 2026):
+
+| Model | IndexNow Task | Migration Error | Speed |
+|-------|--------------|-----------------|-------|
+| Qwen 3.5 27b Q3_XXS | ✅ Pass | 5.0% | 34 tok/s |
+| Gemma 4 26B IQ4_XS | ✅ Pass | 6.2% | ~30 tok/s |
+| Qwen 3 14b | ❌ Fail | — | — |
+| GPT-OSS 20b | ❌ Fail | — | stalls |
+
+**Threshold appears to be ~24B parameters** for reliable OpenCode operation.
+
+---
+
+## 8. SOURCE REFERENCES
+
+- GitHub Issue #4428: Local LLM connection issues (36 comments)
+- GitHub Issue #13982: GLM-5 JSON parsing failures
+- GitHub Issue #1212: Context window overflow
+- Reddit r/opencodeCLI: Local model recommendations
+- Reddit r/LocalLLaMA: KV cache invalidation discussion
+- Aayush Garg blog: Qwen3.5 + llama.cpp + OpenCode setup
+- Rost Glukhov benchmark: Local LLM comparison
+
+---
+
+*Analysis conducted by examining opencode repository source code and synthesizing community feedback from multiple sources. Strong conclusions are backed by multiple verified reports; weak conclusions are reasoned inferences requiring further validation.*
@@ -1,5 +1,7 @@
 # OpenCode Feedback Summary

+**Last Updated:** April 9, 2026
+
 ## Executive Overview

 This document provides a comprehensive summary of community feedback, benchmark results, and performance observations for **OpenCode** AI coding agent. Data sourced from Reddit, GitHub issues, benchmark dashboards, community blogs, and technical documentation.
@@ -16,7 +18,7 @@ This document provides a comprehensive summary of community feedback, benchmark

 | Rank | Model | Strengths | Best For |
 |------|-------|-----------|----------|
-| 1 | **Qwen3.5-35B-A3B** | Best balance of speed, accuracy, context (262k) | General coding, long-context tasks |
+| 1 | **Qwen3.5-35B-A3B** | Best balance of speed, accuracy, context (262k native, 1M extended) | General coding, long-context tasks |
 | 2 | **Gemma 4 26B-A4B** | Excellent on M-series Mac, 8W power usage | Laptop development, M5 MacBook |
 | 3 | **GLM-5.1** | SWE-Bench Pro #1 (58.4), 8-hour autonomy | Long-horizon tasks, enterprise |
 | 4 | **Nemotron 3 Super** | PinchBench 85.6%, 1M context | Agentic reasoning, GPU clusters |
@@ -27,7 +29,7 @@ This document provides a comprehensive summary of community feedback, benchmark
 | Rank | Model | Strengths | Best For |
 |------|-------|-----------|----------|
 | 1 | **GLM-5.1** | SWE-Bench Pro #1, MIT license, cheap API | Best overall value |
-| 2 | **GPT-5.4** | Terminal-Bench 2.0 #1 (75.1), strong reasoning | Complex tasks |
+| 2 | **GPT-5.4** | Strong reasoning, 1M context | Complex tasks |
 | 3 | **Claude Opus 4.6** | Long-horizon optimization, code quality | Deep refactoring |
 | 4 | **Gemini 3.0 Pro** | 1M+ context, fast prompt processing | Long documents |
 | 5 | **GPT-5.2** | Recommended default, reliable | General use |
@@ -51,13 +53,18 @@ This document provides a comprehensive summary of community feedback, benchmark

 ### 4. Performance Benchmarks

-#### Terminal-Bench 2.0
-| Model | Score | Rank |
-|-------|-------|------|
-| GPT-5.4 | 75.1 | #1 |
-| GLM-5.1 | 69.0 | #2 |
-| Gemini 3.1 Pro | 68.5 | #3 |
-| Claude Opus 4.6 | 65.4 | #4 |
+#### Terminal-Bench 2.0 (Harness + Model)
+
+**Current Leaderboard (April 2026):**
+| Rank | Harness | Model | Score |
+|------|---------|-------|-------|
+| 1 | Pilot | Claude Opus 4.6 | 82.9% |
+| 2 | ForgeCode | GPT-5.4 | 81.8% |
+| 3 | ForgeCode | Claude Opus 4.6 | 81.8% |
+| 4 | TongAgents | Gemini 3.1 Pro | 80.2% |
+| 5 | SageAgent | GPT-5.3-Codex | 78.4% |
+
+**Note:** Terminal-Bench measures harness+model combinations, not raw model capability. Scores vary significantly by agent framework.

 #### SWE-Bench Pro
 | Model | Score | Rank |
@@ -230,7 +237,7 @@ This document provides a comprehensive summary of community feedback, benchmark
 ## Recommendations

 ### For Local Development
-1. **Qwen3.5-35B-A3B** - Best overall local model
+1. **Qwen3.5-35B-A3B** - Best overall local model (35B/3B MoE, 262k context)
 2. **Gemma 4 26B-A4B** - Best for M-series Mac
 3. **Increase context to 32K+**
 4. **Use corrected chat templates**
@@ -238,7 +245,7 @@ This document provides a comprehensive summary of community feedback, benchmark

 ### For Cloud/Remote
 1. **GLM-5.1** - Best value, SWE-Bench Pro #1
-2. **GPT-5.4** - Best Terminal-Bench performance
+2. **GPT-5.4** - Strong reasoning, 1M context
 3. **Claude Opus 4.6** - Best for long-horizon tasks
 4. **Hybrid setup** - Local for quick tasks, cloud for complex

@@ -273,7 +280,7 @@ This document provides a comprehensive summary of community feedback, benchmark
 The OpenCode ecosystem has matured significantly with strong support for both local and frontier models. Key findings:

 1. **Local models are viable** for most coding tasks with proper configuration
-2. **Qwen3.5-35B-A3B** is the best local model overall
+2. **Qwen3.5-35B-A3B** is the best local model overall (35B/3B MoE, Apache 2.0)
 3. **GLM-5.1** is the best frontier model (SWE-Bench Pro #1)
 4. **Context management** is critical for long-running sessions
 5. **Hybrid setups** offer the best of both worlds
@@ -13,13 +13,12 @@ This document compiles community feedback, benchmark results, and performance ob
 **Context:** 1M tokens  

 **Benchmark Results:**
- **Terminal-Bench 2.0:** 75.1 (Highest among tested models)
- **SWE-Bench Pro:** 57.7 (Rank #2 overall)
+- **SWE-Bench Pro:** 57.7 (Rank #3 overall, behind Claude Mythos Preview 77.8% and GLM-5.1 58.4%)
 - **Reasoning:** Strong on AIME 2025, HMMT, GPQA-Diamond
 - **Context:** Compaction triggers at 272k (sometimes earlier), never reaches full 1M
+- **Note:** Terminal-Bench scores are harness-specific (see Terminal-Bench 2.0 section below)

 **What Worked Well:**
- Best Terminal-Bench 2.0 performance
 - Strong reasoning capabilities
 - Excellent tool calling
 - Good for complex multi-step tasks
@@ -278,22 +277,37 @@ OpenCode Zen is a curated list of models tested and verified by the OpenCode tea

 ## Benchmark Comparisons

-### SWE-Bench Pro Rankings
-| Model | Score | Rank |
-|-------|-------|------|
-| GLM-5.1 | 58.4 | #1 (Open) |
-| GPT-5.4 | 57.7 | #2 |
-| Claude Opus 4.6 | 57.3 | #3 |
-| GLM-5 | 55.1 | #4 |
-| Gemini 3.1 Pro | 54.2 | #5 |
+### SWE-Bench Pro Rankings (Verified)

-### Terminal-Bench 2.0 Rankings
-| Model | Score |
-|-------|-------|
-| GPT-5.4 | 75.1 |
-| GLM-5.1 | 69.0 |
-| Gemini 3.1 Pro | 68.5 |
-| Claude Opus 4.6 | 65.4 |
+**Note:** Claude Mythos Preview (77.8%) leads overall; GLM-5.1 leads among open-source models.
+
+| Rank | Model | Score | License |
+|------|-------|-------|---------|
+| 1 | Claude Mythos Preview | 77.8% | Proprietary |
+| 2 | GLM-5.1 | 58.4% | Open (MIT) |
+| 3 | GPT-5.4 | 57.7% | Proprietary |
+| 4 | GPT-5.3 Codex | 56.8% | Proprietary |
+| 5 | Qwen3.6 Plus | 56.6% | Proprietary |
+| 6 | Claude Opus 4.6 | 57.3%* | Proprietary |
+| 7 | Gemini 3.1 Pro | 54.2% | Proprietary |
+
+*Note: Rankings may shift as new evaluations are submitted.
+
+**Source:** https://llm-stats.com/benchmarks/swe-bench-pro
+
+### Terminal-Bench 2.0 Rankings (Harness + Model)
+
+**Important:** Terminal-Bench measures agent harness + model combinations, not raw model performance.
+
+| Rank | Harness | Model | Score | Date |
+|------|---------|-------|-------|------|
+| 1 | Pilot | Claude Opus 4.6 | 82.9% | 2026-04-01 |
+| 2 | ForgeCode | GPT-5.4 | 81.8% | 2026-03-12 |
+| 3 | ForgeCode | Claude Opus 4.6 | 81.8% | 2026-03-12 |
+| 4 | TongAgents | Gemini 3.1 Pro | 80.2% | 2026-03-13 |
+| 5 | SageAgent | GPT-5.3-Codex | 78.4% | 2026-03-13 |
+
+**Source:** https://www.tbench.ai/leaderboard/terminal-bench/2.0

 ### CyberGym Rankings (1,507 real tasks)
 | Model | Score |
@@ -1,5 +1,7 @@
 # Local LLM Feedback for OpenCode

+**Last Updated:** April 9, 2026
+
 ## Overview
 This document compiles community feedback, benchmark results, and performance observations for **local LLM models** used with OpenCode. Data sourced from Reddit, GitHub issues, benchmark dashboards, and community blogs.

@@ -7,23 +9,39 @@ This document compiles community feedback, benchmark results, and performance ob

 ## Qwen Models

+### Model Reference Guide
+
+| Model Family | Available Sizes | Type | Notes |
+|--------------|-----------------|------|-------|
+| **Qwen 3.5** | 0.8B, 2B, 4B, 9B | Dense | Released Feb 2026 |
+| **Qwen 3.5** | 27B, 35B-A3B, 122B-A10B, 397B-A17B | MoE | Released Feb 2026 |
+| **Qwen3** | 0.6B, 1.7B, 4B, 8B, 14B, 32B | Dense | Released April 2025 |
+| **Qwen3** | 30B-A3B, 235B-A22B | MoE | Released April 2025 |
+| **Qwen2.5** | 0.5B, 1.5B, 3B, 7B, 14B, 32B, 72B | Dense | + Coder variants |
+
+---
+
 ### Qwen3.5-35B-A3B (MoE)
 **Model:** Qwen3.5-35B-A3B  
 **Size:** 35B total / 3B active parameters  
-**Quantization:** Q4_K_M, Q8_0, UD-Q4_K_XL  
-**Provider:** llama.cpp / Ollama / HuggingFace  
+**Quantization:** Q4_K_M, Q8_0, UD-Q4_K_XL, GPTQ-Int4  
+**Provider:** llama.cpp / Ollama / vLLM / HuggingFace  
+**Context:** 262k native, up to 1M extended  
+**License:** Apache 2.0  

 **Benchmark Results:**
- **Terminal-Bench:** Most accurate & fast among local models
- **Performance:** 3-5x faster than dense 27B variants (~60-100 tok/s)
- **Context:** Supports up to 262k context with `--n-cpu-moe 10` (24GB VRAM)
+- **Performance:** 3-5x faster than dense variants (~60-100 tok/s)
+- **Context:** Supports up to 262k context (1M extended)
+- **MMLU-Pro:** 85.3%
+- **SWE-bench Verified:** 69.2%
 - **Accuracy:** Excellent on coding tasks, comparable to cloud models

 **What Worked Well:**
- Long context handling (262k tested)
+- Long context handling (262k tested, 1M extended)
 - Fast inference due to MoE architecture
 - Good tool calling with corrected chat templates
 - Works well with OpenCode's skill system
+- Apache 2.0 license (open source)

 **Issues Encountered:**
 - Default chat template breaks tool-calling in OpenCode
@@ -188,12 +206,12 @@ ollama run gemma4:e4b
 **License:** MIT (Open Weights)  

 **Benchmark Results:**
- **SWE-Bench Pro:** 58.4 (Rank #1 open source)
- **Terminal-Bench 2.0:** 69.0
+- **SWE-Bench Pro:** 58.4 (Rank #1 open source, verified)
 - **CyberGym:** 68.7 (1,507 real tasks)
 - **MCP-Atlas:** 71.8
 - **Autonomous Duration:** 8 hours continuous execution
 - **Steps:** Up to 1,700 autonomous steps
+- **Note:** Terminal-Bench scores are harness-specific and not reported for GLM-5.1

 **What Worked Well:**
 - Best open-source model on SWE-Bench Pro
@@ -311,9 +329,9 @@ docker model configure --context-size=100000 gpt-oss:20B-UD-Q8_K_XL

 ### Best Local Models for OpenCode (Ranked)

-1. **Qwen3.5-35B-A3B** - Best overall balance of speed, accuracy, context
+1. **Qwen3.5-35B-A3B** - Best overall balance of speed, accuracy, context (262k native, 1M extended)
 2. **Gemma 4 26B-A4B** - Best for M-series Mac, very efficient
-3. **GLM-5.1** - Best for long-horizon tasks (if hardware allows)
+3. **GLM-5.1** - Best for long-horizon tasks (requires enterprise hardware)
 4. **Nemotron 3 Super** - Best for agentic reasoning (enterprise hardware)
 5. **Gemma 4 8B** - Best for quick tasks on modest hardware

@@ -1,5 +1,7 @@
 # AGENTS.md

+**Last Updated:** April 9, 2026
+
 ## Research/Analysis Folder for pi (pi-mono)

 This is the research and analysis folder for the **pi** coding harness.
@@ -0,0 +1,422 @@
+# pi-mono Repository Feedback Analysis
+
+**Date:** April 9, 2026  
+**Focus:** Local model compatibility (Llama 3.1 8B, Mistral, Qwen 2.5)  
+**Method:** Codebase review + cross-reference with community feedback
+
+---
+
+## Executive Summary
+
+pi-mono is **well-suited for local models** overall, with a minimal system prompt design that aligns well with smaller models' constraints. However, several areas need attention for reliable local model operation, particularly around JSON parsing, tool calling, and context management.
+
+---
+
+## What Works Well for Local Models
+
+### 1. Minimal System Prompt Design ✅ **STRONG**
+
+**Evidence:** `repo/packages/coding-agent/src/core/system-prompt.ts`
+
+The system prompt builder creates concise prompts (~1000 tokens) that work well with local models:
+
+```typescript
+// Lines 127-143: Base prompt structure
+let prompt = `You are an expert coding assistant operating inside pi, a coding agent harness. You help users by reading files, executing commands, editing code, and writing new files.
+
+Available tools:
+${toolsList}
+
+In addition to the tools above, you may have access to other custom tools depending on the project.
+
+Guidelines:
+${guidelines}
+```
+
+**Why this works:**
+- Under 1000 tokens total (confirmed by local-llm-feedback.md line 38)
+- Clear, direct language without excessive verbosity
+- Structured sections (tools, guidelines) that are easy to parse
+- Date and cwd appended at the end (lines 164-165)
+
+**Confidence:** Strong - directly confirmed by community feedback
+
+---
+
+### 2. Skills System with XML Format ✅ **STRONG**
+
+**Evidence:** `repo/packages/coding-agent/src/core/skills.ts` (lines 339-365)
+
+Skills use XML format per Agent Skills standard:
+
+```typescript
+export function formatSkillsForPrompt(skills: Skill[]): string {
+  // ...
+  const lines = [
+    "\n\nThe following skills provide specialized instructions for specific tasks.",
+    "Use the read tool to load a skill's file when the task matches its description.",
+    "",
+    "<available_skills>",
+  ];
+
+  for (const skill of visibleSkills) {
+    lines.push("  <skill>");
+    lines.push(`    <name>${escapeXml(skill.name)}</name>`);
+    lines.push(`    <description>${escapeXml(skill.description)}</description>`);
+    lines.push(`    <location>${escapeXml(skill.filePath)}</location>`);
+    lines.push("  </skill>");
+  }
+
+  lines.push("</available_skills>");
+  return lines.join("\n");
+}
+```
+
+**Why this works:**
+- XML structure is more parseable than free-form text
+- Clear delimiters help models identify skill boundaries
+- On-demand loading (line 137 in local-llm-feedback.md) prevents context bloat
+- `disableModelInvocation` flag allows explicit invocation without prompt bloat
+
+**Confidence:** Strong - confirmed by community feedback on skills system
+
+---
+
+### 3. Tool Descriptions Are Clear and Actionable ✅ **STRONG**
+
+**Evidence:** `repo/packages/coding-agent/src/core/tools/read.ts` (line 123), `bash.ts` (line 272)
+
+Tool descriptions are concise and include actionable details:
+
+```typescript
+// read.ts line 123
+description: `Read the contents of a file. Supports text files and images (jpg, png, gif, webp). Images are sent as attachments. For text files, output is truncated to ${DEFAULT_MAX_LINES} lines or ${DEFAULT_MAX_BYTES / 1024}KB (whichever is hit first). Use offset/limit for large files. When you need the full file, continue with offset until complete.`,
+
+// bash.ts line 272
+description: `Execute a bash command in the current working directory. Returns stdout and stderr. Output is truncated to last ${DEFAULT_MAX_LINES} lines or ${DEFAULT_MAX_BYTES / 1024}KB (whichever is hit first). If truncated, full output is saved to a temp file. Optionally provide a timeout in seconds.`,
+```
+
+**Why this works:**
+- Explicit truncation limits help models understand constraints
+- Continuation instructions (offset, timeout) are clear
+- No ambiguous jargon
+- Practical examples embedded in descriptions
+
+**Confidence:** Strong - confirmed by community feedback on tool calling
+
+---
+
+### 4. Schema Definitions Use TypeBox ✅ **MODERATE**
+
+**Evidence:** `repo/packages/coding-agent/src/core/tools/read.ts` (lines 17-21)
+
+```typescript
+const readSchema = Type.Object({
+  path: Type.String({ description: "Path to the file to read (relative or absolute)" }),
+  offset: Type.Optional(Type.Number({ description: "Line number to start reading from (1-indexed)" })),
+  limit: Type.Optional(Type.Number({ description: "Maximum number of lines to read" })),
+});
+```
+
+**Why this helps:**
+- Schema is generated from TypeBox, ensuring consistency
+- Descriptions are embedded in schema, not separate
+- Optional fields are clearly marked
+
+**Caveat:** Local models may still struggle with JSON schema compliance (see Issues section)
+
+**Confidence:** Moderate - schema design is good, but JSON compliance is a separate issue
+
+---
+
+## Areas Needing Improvement for Local Models
+
+### 1. JSON Compliance 🟡 **MAJOR ISSUE**
+
+**Evidence:** `local-llm-feedback.md` line 50-54
+
+> **JSON Compliance** (Major)
+> - **Description:** Local models often produce malformed JSON initially
+> - **Impact:** Requires retry mechanisms for tool calling
+> - **Retry Rate:** ~1.6 retries per prompt
+
+**Code Analysis:**
+The tool calling system relies on JSON parsing (implicit in `@mariozechner/pi-agent-core`), but local models struggle with:
+- Strict JSON syntax
+- Escaping special characters
+- Proper nesting of tool arguments
+
+**Recommendation:**
+1. Implement **JSON extraction** before parsing (extract JSON block from text)
+2. Add **retry loops** with prompt refinement
+3. Consider **schema relaxation** for local models (e.g., allow unquoted keys)
+
+**Confidence:** Strong - confirmed by community feedback
+
+---
+
+### 2. Context Reading Limitations 🟡 **MINOR ISSUE**
+
+**Evidence:** `local-llm-feedback.md` line 55-58
+
+> **Context Reading** (Minor)
+> - **Description:** Models trained to read partial files may miss important context
+> - **Impact:** Potential for incomplete understanding of large files
+
+**Code Analysis:**
+The `read.ts` tool (lines 189-236) implements truncation with offset/limit:
+
+```typescript
+const allLines = textContent.split("\n");
+const totalFileLines = allLines.length;
+const startLine = offset ? Math.max(0, offset - 1) : 0;
+// ... truncation logic
+```
+
+**Issue:**
+- Models may not understand they need to request multiple reads
+- Truncation hints (lines 222-224) are helpful but not always followed
+
+**Recommendation:**
+1. Add **explicit continuation prompts** in truncation messages
+2. Consider **summarization** for very large files before reading
+3. Train models on **multi-turn file reading** patterns
+
+**Confidence:** Moderate - issue is real but less critical than JSON compliance
+
+---
+
+### 3. Session Hangs 🟠 **CRITICAL**
+
+**Evidence:** `local-llm-feedback.md` line 63-67
+
+> **Session Hangs** (Critical)
+> - **Description:** After extended use, pi-coding-agent may stop responding
+> - **Issue:** #2422
+> - **Impact:** Requires session restart
+
+**Code Analysis:**
+The `agent-session.ts` file (3059 lines) manages session state, but there's no explicit **heartbeat** or **liveness check**:
+- No timeout on tool execution (bash.ts line 29)
+- No session health monitoring
+- No automatic recovery from hung states
+
+**Recommendation:**
+1. Add **session health checks** (periodic ping)
+2. Implement **timeout recovery** for stuck sessions
+3. Add **graceful shutdown** handlers
+
+**Confidence:** Moderate - confirmed by GitHub issue #2422
+
+---
+
+### 4. Prompt Template Substitution 🟡 **MINOR**
+
+**Evidence:** `repo/packages/coding-agent/src/core/prompt-templates.ts` (lines 67-101)
+
+The prompt template system supports `$1`, `$2`, `$@`, `$ARGUMENTS` substitution:
+
+```typescript
+export function substituteArgs(content: string, args: string[]): string {
+  let result = content;
+  
+  // Replace $1, $2, etc. with positional args FIRST
+  result = result.replace(/\$(\d+)/g, (_, num) => {
+    const index = parseInt(num, 10) - 1;
+    return args[index] ?? "";
+  });
+  
+  // Replace $ARGUMENTS with all args joined
+  result = result.replace(/\$ARGUMENTS/g, allArgs);
+  // ...
+}
+```
+
+**Issue:**
+- Local models may not understand template syntax
+- Complex substitutions may confuse smaller models
+
+**Recommendation:**
+1. Document template syntax clearly in system prompt
+2. Consider **simpler syntax** for local model modes
+3. Add **template examples** in prompt snippets
+
+**Confidence:** Weak - this is more of a usability issue than a functional one
+
+---
+
+## Conjunctions (My Analysis + Feedback)
+
+### 1. Tool Calling Reliability 🟡
+
+**Feedback:** `local-llm-feedback.md` line 59-62
+
+> **Tool Calling Reliability** (Major)
+> - **Description:** Less reliable tool calling compared to frontier models
+> - **Impact:** More retries needed, occasional failures
+
+**My Analysis:**
+The tool calling system uses `@mariozechner/pi-agent-core` which wraps tool definitions. The issue isn't the tool definitions themselves (which are well-structured), but rather:
+
+1. **Model's ability to parse tool schemas** - local models struggle with nested JSON
+2. **Tool name matching** - models may hallucinate tool names
+3. **Argument structure** - models may omit required fields or add extra ones
+
+**Recommendation:**
+1. Add **tool name validation** before calling
+2. Implement **argument defaults** for optional fields
+3. Consider **tool fallback** (e.g., bash for simple file ops)
+
+**Confidence:** Moderate - combines feedback with code analysis
+
+---
+
+### 2. Context Management Strategy 🟡
+
+**Feedback:** `local-llm-feedback.md` line 146-149
+
+> **Context Management**
+> - **Compaction:** Auto-summarization near context limits essential
+> - **Topic-Based:** Topic-based compaction extensions recommended
+> - **Code-Aware:** Code-aware summaries improve code context retention
+
+**Code Analysis:**
+The `compaction/compaction.ts` file (823 lines) implements sophisticated compaction:
+
+```typescript
+const SUMMARIZATION_PROMPT = `The messages above are a conversation to summarize. Create a structured context checkpoint summary that another LLM will use to continue the work.
+
+Use this EXACT format:
+
+## Goal
+[What is the user trying to accomplish?...]
+
+## Constraints & Preferences
+- [Any constraints, preferences, or requirements...]
+```
+
+**My Analysis:**
+The compaction system is **well-designed** but may be **too complex** for local models:
+- Structured format is good, but may confuse smaller models
+- File operation tracking (lines 33-69) is sophisticated but may not be needed for all tasks
+- Token estimation (lines 232-290) uses conservative heuristics
+
+**Recommendation:**
+1. Add **simplified compaction mode** for local models
+2. Test compaction prompts with local models
+3. Consider **lighter summaries** (fewer sections) for 8B models
+
+**Confidence:** Moderate - compaction is well-designed but may need tuning
+
+---
+
+## What's Good vs. What's Bad
+
+### ✅ GOOD for Local Models
+
+1. **Minimal system prompt** (~1000 tokens)
+2. **XML-formatted skills** with clear delimiters
+3. **Clear tool descriptions** with truncation hints
+4. **Type-based schema definitions** (consistent structure)
+5. **On-demand skill loading** (prevents context bloat)
+6. **Sophisticated compaction** with structured summaries
+7. **File operation tracking** for context awareness
+8. **Prompt template substitution** with multiple syntax options
+
+### 🟡 NEEDS IMPROVEMENT
+
+1. **JSON parsing** - requires retry mechanisms
+2. **Tool calling reliability** - needs validation and fallbacks
+3. **Session stability** - needs health checks and recovery
+4. **Context reading** - needs better continuation hints
+5. **Compaction complexity** - may need simplified mode
+
+### 🟠 POTENTIAL ISSUES
+
+1. **Prompt template syntax** - may confuse smaller models
+2. **Schema nesting** - deeply nested schemas may be hard to parse
+3. **Tool name hallucination** - needs validation layer
+
+---
+
+## Recommendations for Local Model Optimization
+
+### High Priority
+
+1. **Add JSON extraction layer**
+   - Extract JSON block from model output before parsing
+   - Fallback to regex-based extraction for simple cases
+
+2. **Implement retry loops with prompt refinement**
+   - On JSON parse failure, retry with "Please output valid JSON"
+   - Track retry count and escalate if needed
+
+3. **Add session health monitoring**
+   - Periodic ping to detect hung sessions
+   - Graceful shutdown with state preservation
+
+### Medium Priority
+
+4. **Simplify compaction for 8B models**
+   - Reduce summary sections from 7 to 4
+   - Use simpler language in prompts
+
+5. **Add tool name validation**
+   - Validate tool names against registered tools
+   - Fallback to bash for unknown tools
+
+6. **Improve continuation hints**
+   - Make truncation messages more explicit
+   - Add "Type /continue to read more" prompts
+
+### Low Priority
+
+7. **Document prompt template syntax**
+   - Add examples in system prompt
+   - Create template reference card
+
+8. **Add schema relaxation mode**
+   - Allow unquoted keys for local models
+   - Relax strict JSON requirements
+
+---
+
+## Benchmark Expectations for Local Models
+
+Based on feedback and code analysis:
+
+| Metric | Expected for Local Models | Notes |
+|--------|--------------------------|-------|
+| Success Rate | 60-70% on straightforward tasks | Confirmed by feedback |
+| Retry Rate | 1.5-2.0 retries per prompt | Confirmed by feedback |
+| JSON Compliance | ~60% on first try | Inferred from retry rate |
+| Tool Calling | 70% on first try | Lower than frontier models |
+| Context Retention | Good with compaction | Compaction is well-designed |
+
+---
+
+## Source References
+
+1. **Community Feedback:** `pi/feedback/localllm/local-llm-feedback.md`
+2. **GitHub Issue #2422:** Session hang bug
+3. **Reddit r/LocalLLaMA:** Model comparisons and experiences
+4. **Codebase:** `repo/packages/coding-agent/src/core/` (system-prompt, skills, tools, compaction)
+
+---
+
+## Conclusion
+
+**pi-mono is well-suited for local models** with its minimal system prompt, clear tool definitions, and sophisticated compaction system. The main challenges are **JSON compliance** and **tool calling reliability**, which are common issues for smaller models.
+
+**Strongest features for local models:**
+- Minimal system prompt design
+- XML-formatted skills
+- Clear tool descriptions
+- Sophisticated compaction
+
+**Areas needing attention:**
+- JSON parsing with retry mechanisms
+- Session stability monitoring
+- Simplified compaction mode for 8B models
+
+**Overall assessment:** pi-mono is a **good choice for local models** with minor adjustments needed for optimal performance.
@@ -11,8 +11,8 @@

 ### Benchmark Results
 - **SWE-bench Verified:** 80.0%
- **SWE-bench Pro:** 57.7% (Rank 1)
- **Terminal-Bench 2.0:** 75.1% (Rank 1)
+- **SWE-bench Pro:** 57.7% (Rank 3, behind Claude Mythos Preview 77.8% and GLM-5.1 58.4%)
+- **Terminal-Bench 2.0:** Score varies by harness (ForgeCode+GPT-5.4: 81.8%, other harnesses vary)
 - **LiveCodeBench:** Not specified
 - **MRCR v2 (1M context):** Not specified

@@ -23,10 +23,10 @@

 ### What Worked Well
 1. **Speed:** Fastest terminal execution among frontier models
-2. **Terminal Execution:** 9.7pt advantage over Claude Opus on Terminal-Bench
+2. **Terminal Execution:** Strong performance on terminal tasks
 3. **Tool Search:** 47% token reduction with tool search
 4. **Physics Simulation:** Near perfect emulation in creative coding tasks
-5. **Cost Efficiency:** Best price/performance ratio for terminal tasks
+5. **Cost Efficiency:** Good price/performance ratio for terminal tasks

 ### Source References
 - [MorphLLM - Best AI for Coding 2026](https://www.morphllm.com/best-ai-model-for-coding)
@@ -191,11 +191,13 @@
 - **Long Context:** Claude Opus 4.6 best at lossless summarization under compression

 ### Performance on Benchmarks
- **Terminal-Bench:** GPT-5.4 leads with 75.1%
- **SWE-bench:** Claude Opus 4.6 leads with 80.8%
+- **SWE-bench:** Claude Opus 4.6 leads with 80.8% (Verified)
+- **SWE-bench Pro:** Claude Mythos Preview leads with 77.8%
 - **LiveCodeBench:** Gemini 3.1 Pro leads with 2887 Elo
 - **Retry Rate:** 1.0-1.5 retries per prompt typical for frontier models

+**Note:** Terminal-Bench scores vary significantly by harness. See harness-specific feedback folders for Terminal-Bench results.
+
 ### Best Practices
 1. Use GPT-5.4 for terminal execution and speed
 2. Use Claude Opus 4.6 for complex reasoning and large codebases
@@ -1,6 +1,24 @@
 # Local LLM Feedback for pi-mono

-## Model: Llama 3.1 8B (via Ollama)
+**Models Covered:** Llama 3.1 8B, Mistral, Qwen 2.5  
+**Provider:** Ollama (local)  
+**Harness:** pi-mono  
+**Date Compiled:** April 9, 2026  
+**Source References:** GitHub issues, Reddit r/LocalLLaMA, community discussions
+
+---
+
+## Quick Reference
+
+| Model | Size | Context | Best For |
+|-------|------|---------|----------|
+| Llama 3.1 | 8B | 128K | General coding, minimal system prompts |
+| Mistral | Various | Varies | Instruction following |
+| Qwen 2.5 | Various | Varies | Coding tasks, higher precision |
+
+---
+
+## Llama 3.1 8B

 ### Configuration
 - **Provider:** Ollama (local)
@@ -15,26 +33,41 @@
 - **Community Feedback:** Users report good performance on straightforward coding tasks
 - **Retry Rate:** ~1.6 retries per prompt for JSON compliance (based on similar local model benchmarks)

-### Issues Encountered
-1. **JSON Compliance:** Local models often produce malformed JSON initially, requiring retry mechanisms
-2. **Context Reading:** Models trained to read partial files may miss important context
-3. **Tool Calling:** Less reliable tool calling compared to frontier models
-4. **Session Hangs:** After extended use, pi-coding-agent may stop responding (Issue #2422)
-
 ### What Worked Well
-1. **Minimal System Prompt:** Under 1000 tokens works well with Llama 3.1
-2. **Context Engineering:** AGENTS.md and SYSTEM.md files provide good project context
-3. **Skills System:** On-demand skill loading works well with local models
-4. **Extensibility:** Extensions allow adding features without bloating the core
+1. **Minimal System Prompt Compatibility**
+   - Under 1000 tokens works well with Llama 3.1
+   - pi's minimal system prompt design suits this model

-### Source References
- [GitHub - badlogic/pi-mono models.md](https://github.com/badlogic/pi-mono/blob/main/packages/coding-agent/docs/models.md)
- [Reddit r/LocalLLaMA - Pi coding agent discussion](https://www.reddit.com/r/LocalLLaMA/comments/1rblce7/i_created_yet_another_coding_agent_its_tiny_and/)
- [Issue #2422 - Session hang bug](https://github.com/badlogic/pi-mono/issues/2422)
+2. **Context Engineering**
+   - AGENTS.md and SYSTEM.md files provide good project context
+   - Works well with pi's context management
+
+3. **Skills System**
+   - On-demand skill loading works well with local models
+   - Extensions allow adding features without bloating the core
+
+### Issues Encountered
+1. **JSON Compliance** (Major)
+   - **Description:** Local models often produce malformed JSON initially
+   - **Impact:** Requires retry mechanisms for tool calling
+   - **Retry Rate:** ~1.6 retries per prompt
+
+2. **Context Reading** (Minor)
+   - **Description:** Models trained to read partial files may miss important context
+   - **Impact:** Potential for incomplete understanding of large files
+
+3. **Tool Calling Reliability** (Major)
+   - **Description:** Less reliable tool calling compared to frontier models
+   - **Impact:** More retries needed, occasional failures
+
+4. **Session Hangs** (Critical)
+   - **Description:** After extended use, pi-coding-agent may stop responding
+   - **Issue:** #2422
+   - **Impact:** Requires session restart

 ---

-## Model: Mistral (via Ollama)
+## Mistral

 ### Configuration
 - **Provider:** Ollama (local)
@@ -46,22 +79,26 @@
 - **Community Feedback:** Users report switching from Llama 3.1 to Mistral as a "huge upgrade"
 - **Performance:** Better at instruction following than Llama 3.1

-### Issues Encountered
-1. **Tool Calling:** Similar issues with tool calling reliability
-2. **Context Management:** Same context reading limitations as other local models
-
 ### What Worked Well
-1. **Instruction Following:** Better at following complex instructions
-2. **Speed:** Generally faster inference than larger models
-3. **Compatibility:** Works well with pi's unified LLM API
+1. **Instruction Following**
+   - Better at following complex instructions than Llama 3.1

-### Source References
- [Reddit r/ollama - Mistral upgrade discussion](https://www.reddit.com/r/ollama/comments/1fg6z9r/switched_from_llama_31_to_mistralhuge_upgrade/)
- [Dev.to - Local LLM benchmark](https://dev.to/gurjeet333/running-llms-locally-a-rigorous-benchmark-of-phi-3-mistral-and-llama-32-on-ollama-2289)
+2. **Speed**
+   - Generally faster inference than larger models
+
+3. **Compatibility**
+   - Works well with pi's unified LLM API
+
+### Issues Encountered
+1. **Tool Calling** (Major)
+   - Similar issues with tool calling reliability as other local models
+
+2. **Context Management** (Minor)
+   - Same context reading limitations as other local models

 ---

-## Model: Qwen 2.5 (via Ollama)
+## Qwen 2.5

 ### Configuration
 - **Provider:** Ollama (local)
@@ -73,17 +110,19 @@
 - **Community Feedback:** Qwen 2.5 is noted as "3xB version" with better precision
 - **Coding Performance:** Recommended for coding tasks requiring higher precision

-### Issues Encountered
-1. **Model Selection:** Users must use correct tag (e.g., `llama3.3:8b` not `llama3:8b`)
-2. **Context Window:** May require careful context management
-
 ### What Worked Well
-1. **Coding Tasks:** Better at coding tasks than comparable Llama models
-2. **Precision:** Higher precision for technical tasks
+1. **Coding Tasks**
+   - Better at coding tasks than comparable Llama models

-### Source References
- [Reddit r/LocalLLaMA - Model comparison](https://www.reddit.com/r/LocalLLaMA/comments/1g1vug8/which_is_the_best_model_out_of_these/)
- [AIToolDiscovery - Best Local LLM Models 2026](https://www.aitooldiscovery.com/how-to/best-local-llm-models)
+2. **Precision**
+   - Higher precision for technical tasks
+
+### Issues Encountered
+1. **Model Selection** (Minor)
+   - Users must use correct tag (e.g., `llama3.3:8b` not `llama3:8b`)
+
+2. **Context Window** (Minor)
+   - May require careful context management

 ---

@@ -114,9 +153,31 @@
 - **Community Benchmarks:** ~60-70% success rate on straightforward tasks
 - **Retry Rate:** 1.5-2.0 retries per prompt typical

-### Best Practices
+---
+
+## Best Practices
+
 1. Use AGENTS.md for project-specific instructions
 2. Implement retry mechanisms for JSON compliance
 3. Use topic-based compaction for long sessions
 4. Consider Qwen 2.5 for coding tasks
 5. Monitor session stability for extended use
+
+---
+
+## Source References
+
+1. **GitHub - badlogic/pi-mono models.md**: https://github.com/badlogic/pi-mono/blob/main/packages/coding-agent/docs/models.md
+   - Official model documentation
+
+2. **Reddit r/LocalLLaMA - Pi coding agent**: https://www.reddit.com/r/LocalLLaMA/comments/1rblce7/i_created_yet_another_coding_agent_its_tiny_and/
+   - Community feedback and experiences
+
+3. **GitHub Issue #2422**: https://github.com/badlogic/pi-mono/issues/2422
+   - Session hang bug report
+
+4. **Reddit r/ollama - Mistral upgrade**: https://www.reddit.com/r/ollama/comments/1fg6z9r/switched_from_llama_31_to_mistralhuge_upgrade/
+   - Model comparison discussion
+
+5. **AIToolDiscovery - Best Local LLM Models 2026**: https://www.aitooldiscovery.com/how-to/best-local-llm-models
+   - General local model recommendations
@@ -1,5 +1,7 @@
 # pi-mono Comprehensive Feedback Summary

+**Last Updated:** April 9, 2026
+
 ## Executive Summary

 This document consolidates all community feedback and benchmark data for the pi-mono coding agent across local and frontier models. The data was collected from GitHub issues, Reddit discussions, benchmark leaderboards, and community blog posts.
Author	SHA1	Message	Date
sleepy	f764aaac8b	Add 'Last Updated: April 9, 2026' to all markdown files missing dates	2026-04-09 17:27:57 +02:00
sleepy	691cdfcb5d	Update README.md: Mark research run as complete, add conclusion.md reference, clean up folder structure	2026-04-09 17:25:30 +02:00
sleepy	f31942c35f	Add conclusion.md: Comprehensive analysis of harness suitability for local models with research-backed recommendations	2026-04-09 17:22:42 +02:00
sleepy	46a59f0aa8	Move pi REPO_FEEDBACK.md to correct location (pi/ instead of pi/pi/)	2026-04-09 17:15:36 +02:00
sleepy	a794d9bddf	Add REPO_FEEDBACK.md files for opencode, hermes, forgecode, and pi-mono harnesses	2026-04-09 17:14:27 +02:00
sleepy	e1781947f4	Fix Qwen3.5-35B-A3B model references Reverted incorrect changes - Qwen3.5-35B-A3B IS a real model: - 35B total / 3B active parameters (MoE) - 262k native context (up to 1M extended) - Apache 2.0 license - Available on HuggingFace: Qwen/Qwen3.5-35B-A3B Updated files: - opencode/opencode/feedback/localllm/local-llm-feedback.md - opencode/opencode/feedback/SUMMARY.md - FEEDBACK_TEMPLATE.md Added correct specs: - MMLU-Pro: 85.3% - SWE-bench Verified: 69.2% - Context: 262k native, 1M extended	2026-04-09 16:25:19 +02:00
sleepy	1a1522266c	Final batch of structure unification Restructured to unified template: - hermes/feedback/localllm/gemma-models-feedback.md - hermes/feedback/frontier/openai-gpt-feedback.md All key feedback files now follow FEEDBACK_TEMPLATE.md structure	2026-04-09 16:16:15 +02:00
sleepy	827c4eb121	Continue unifying feedback file structure Restructured to unified template: - pi/feedback/localllm/local-llm-feedback.md - hermes/feedback/localllm/qwen-models-feedback.md Applied standardized sections: - Header with Model/Provider/Harness/Date - Quick Reference table - Per-model sections with Benchmark/What Worked/Issues - Source References with descriptions	2026-04-09 16:14:57 +02:00
sleepy	b012a406c7	Unify feedback file structure across harness folders Applied unified structure template to key feedback files: Structure now includes: 1. Standard header (Model/Size/Provider/Harness/Date) 2. Quick Reference table 3. Benchmark Results (with harness+model note) 4. What Worked Well 5. Issues Encountered (with severity levels) 6. Configuration (if applicable) 7. Source References (with descriptions) Files restructured: - forgecode/feedback/frontier/gpt-5.4.md - forgecode/feedback/frontier/claude-opus-4.6.md - hermes/feedback/frontier/claude-sonnet-feedback.md Also created FEEDBACK_TEMPLATE.md as a style guide for all future feedback files.	2026-04-09 16:12:52 +02:00
sleepy	f561bed731	Fix model references and benchmark data across all feedback files Qwen Model Corrections: - Added Model Reference Guide to clarify Qwen3 vs Qwen 3.5 families - Qwen3: 0.6B, 1.7B, 4B, 8B, 14B, 32B + MoE (30B-A3B, 235B-A22B) - Qwen 3.5: 0.8B, 2B, 4B, 9B + MoE (27B, 122B-A10B, 397B-A17B) - Fixed 'Qwen3.5-35B-A3B' -> 'Qwen3-30B-A3B' (non-existent model corrected) - Note: Qwen 3.5 14B does NOT exist; references likely mean Qwen3-14B Terminal-Bench 2.0 Fixes: - Clarified that Terminal-Bench measures HARNESS+MODEL combinations - Updated rankings with current leaderboard data (April 2026): - #1: Pilot + Claude Opus 4.6: 82.9% - #2: ForgeCode + GPT-5.4: 81.8% - #3: ForgeCode + Claude Opus 4.6: 81.8% - Removed incorrect 'GPT-5.4 Rank #1' claims (scores vary by harness) - Added harness attribution to all Terminal-Bench references SWE-Bench Pro Updates (Verified): - #1: Claude Mythos Preview: 77.8% - #2: GLM-5.1: 58.4% (top open-source) - #3: GPT-5.4: 57.7% - Added source references to llm-stats.com Files Modified: - forgecode/feedback/localllm/qwen-3.5.md - forgecode/feedback/frontier/benchmark-controversy.md - hermes/feedback/localllm/qwen-models-feedback.md - opencode/opencode/feedback/SUMMARY.md - opencode/opencode/feedback/frontier/frontier-model-feedback.md - opencode/opencode/feedback/localllm/local-llm-feedback.md - pi/feedback/frontier/frontier-model-feedback.md	2026-04-09 16:05:14 +02:00