sleepy/ternary
1
0
Fork
You've already forked ternary
 0
Code Issues 17 Pull requests Projects Releases Packages Wiki Activity Actions

EngramModule has double residual — one in Engram, one in TransformerBlock #16

New issue
Open
opened 2026-05-09 19:22:08 +02:00 by sleepy · 0 comments
sleepy commented 2026-05-09 19:22:08 +02:00
Owner
Copy link

Problem

EngramModule.forward() (engram.py:156) returns h + y (a residual connection). Then in model.py:130-133, the TransformerBlock also adds a residual:

# model.py: Engram injection
s = self.engrams[engram_key](s, compressed_ids)  # returns h + y

# model.py: block forward
attn_out, kv = self.attn(self.norm1(h_in), mask, past_kv)
x = x + attn_out  # second residual

So the Engram output gets two residual additions — one from Engram itself and one from TransformerBlock. The input h bypasses both the Engram processing and the block's attention/FFN.

Impact

  • Not necessarily a bug, but architecturally unusual
  • May affect gradient flow — the signal from h reaches the next layer through two independent paths (Engram residual + block residual)
  • Could lead to different optimization dynamics than intended

Action needed

Decide on intent:

  • If Engram is meant to be a preprocessing step: remove its internal residual (return y instead of h + y)
  • If Engram is meant to be a residual module: keep it, but document the double-residual design

Files

  • tergent/engram.py:156
  • tergent/model.py:130-133
## Problem `EngramModule.forward()` (engram.py:156) returns `h + y` (a residual connection). Then in `model.py:130-133`, the TransformerBlock also adds a residual: ```python # model.py: Engram injection s = self.engrams[engram_key](s, compressed_ids) # returns h + y # model.py: block forward attn_out, kv = self.attn(self.norm1(h_in), mask, past_kv) x = x + attn_out # second residual ``` So the Engram output gets **two** residual additions — one from Engram itself and one from TransformerBlock. The input `h` bypasses both the Engram processing and the block's attention/FFN. ## Impact - Not necessarily a bug, but architecturally unusual - May affect gradient flow — the signal from `h` reaches the next layer through two independent paths (Engram residual + block residual) - Could lead to different optimization dynamics than intended ## Action needed Decide on intent: - If Engram is meant to be a **preprocessing** step: remove its internal residual (`return y` instead of `h + y`) - If Engram is meant to be a **residual module**: keep it, but document the double-residual design ## Files - `tergent/engram.py:156` - `tergent/model.py:130-133`
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
fix/1-integrate-fixes
fix/5-sweep-json-serialization
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
sleepy/ternary#16
No description provided.