Transformer parts and their geometric replacements
In one paragraph. A transformer is built from a handful of standard parts: a tokenizer, an embedding table, a position signal, attention, feed-forward layers, normalization, an output layer, a loss and an optimizer. UOR-R4 tries to replace each part with a geometric or exact mechanism that avoids softmax and matrix multiplication. This page puts each transformer part beside the project's replacement and says how far that replacement has got. The short version: most parts have a candidate, and several candidates are built and tested. Two parts have no candidate at all: position and normalization. No two replacements have yet been trained together into a model that beats simple word-count tables (audit synthesis, section 1).
Each transformer part beside its replacement. Colour gives the status. The two dashed boxes are the gaps.
Terms are in the glossary. Labels follow docs/formal_vocabulary.md: proof, measurement or hypothesis.
How to read the status
Every mechanism has exactly one of three statuses, set on the mechanism board:
- ACTIVE: on the current model path,
crates/uor-r4-discrete. - READY: built and tested, but never trained into a model.
- PARKED: the forms tried so far failed or broke a rule. The mechanism stays open, and its page lists what would reopen it.
There is no "retired" or "dead" status. A failed run closes one configuration, never a mechanism (charter, section 4.1). Today the board has 10 ACTIVE, 4 READY and 13 PARKED mechanisms.
A passing test shows that code does what it claims. It is not a trained capability, and a test marked NOT_RUN is not a pass.
Part by part
| Transformer part | What it does | UOR-R4 replacement (status) | Where it stands |
|---|---|---|---|
| Tokenizer | Cuts text into numbered pieces | tokenizer (ACTIVE) | 4,096 pieces, pinned in every corpus and artifact. No other vocabulary size has been compared |
| Embedding table | Gives each token a starting vector | prime-zeta codebook (ACTIVE), address cells (ACTIVE) | PrimeZeta gives 4,096 of 4,096 tokens distinct vectors (measurement, record). Address cells raised zero-training reach from 127 to 1,351 of 3,847 rows; training then eroded it to 551, below the control's 857 (measurement, pre-flight, dose) |
| Position signal | Tells the model the order of tokens | None | Gap 1. See below |
| Attention: what to read | Picks which earlier tokens matter | route read, flock read, relation read, admission (all ACTIVE); Hopf observation (PARKED) | Flock passes its 6 of the 25 shared GPU construction checks (11 Route, 6 Flock, 8 driver) (measurement, record). No read's own value has been measured: Route received 20 learning proposals in 16,404 batches (attention state) |
| Attention: mixing what was read | Combines the chosen tokens | flock read (ACTIVE) | Rotates a few nearby earlier tokens into one frame and moves toward their consensus |
| Copy heads | Copies a word seen earlier | relation read (ACTIVE), pointer key (PARKED) | Relation passes 9 of 9 checks and trained twice; both runs stopped before a verdict |
| Context memory (key-value cache) | Stores every earlier position | exact memory, VSA, read binding (all PARKED) | Using an exact address as the read has never been built |
| Durable memory outside the model | Not a transformer part | object store (READY) | 20 integration tests at delivery (60 after later work) and a conformance pass. Nothing uses it yet |
| Feed-forward layer | Processes each token on its own; most of the multiplication | SpiralCore/FBS projection, Hamilton gate (READY); E8 orbit memory (PARKED) | None has ever been trained. The current model has no feed-forward layer |
| Gates and activation functions | Squash signals with sigmoid, tanh and the like | Hamilton gate, recurrence gates (READY) | Turn the gate into a rotation, or use tables of powers of two. 18 GPU construction checks pass; never trained |
| Recurrent mixing (in hybrids) | Carries a running state forward | recurrence gates (READY), geometric stack (PARKED) | The old stack trained, but used softmax and matrix multiplication |
| Matrix arithmetic | Multiplies by learned grids | D11 integer serving (ACTIVE), SpiralCore/FBS (READY); spin weights, ternary/XOR (PARKED) | Integer engine's audits are green, but no model built under today's rules has been exported into it |
| Normalization | Rescales signals between layers | None | Gap 2. See below |
| Output head (softmax over all words) | Scores every word and picks one | admission, discrete learning (ACTIVE) | Best trained result: 65 and 463 of 14,287 correct, against 1,226 and 6,300 for two count tables (measurement, K16 outcome) |
| Loss (cross-entropy) | Measures how wrong a guess was | rank-margin hinge (ACTIVE); read binding (PARKED) | Asks only whether the right word beats its strongest rivals by a gap. It gives no probability, so no bits-per-byte score |
| Optimizer (backpropagation) | Adjusts the learned numbers | discrete learning (ACTIVE); carrier credit, legal construction (PARKED) | Tries every legal move for one entry and keeps it only if the score strictly improves. A full run accepts about 130 to 410 moves |
| Runtime | Runs the finished model | D11 integer serving (ACTIVE); R4G1 graph (PARKED) | Runs in a browser with no multiply instruction in 7 of 7 serving functions (measurement, record) |
Sources: the mechanism board slot map and the audit's stitching map (synthesis, section 1), which cite each record. The move counts are from the K16 outcome (130, 129, 407 and 410 accepted moves).
One status is disputed. The board keeps D11 integer serving as ACTIVE, as its own page says, but the audit would call it READY, because no lawful model has been exported through it yet (mechanism board, contradictions).
The two gaps
Gap 1: position. A transformer adds a signal that says where each token sits in the sequence. No UOR-R4 mechanism is assigned to this job, and no lawful evidence exists for one. The current model keeps the most recent tokens in order at the leaves of a small tree, so order is present in its layout, but nothing is learned about it (the model page). The nearest result is about carrying a running state by quaternion rotation. It beat no rotation by 0.071 and 0.077 nats (a nat is a unit of prediction error; lower is better), but in a model that still used softmax attention, a sigmoid gate and dense layers, with 332,928 fewer parameters in the comparison arm (transport record). So "rotation can carry order" is a hypothesis.
Gap 2: normalization. A transformer rescales its signals between layers so they stay in a workable range. UOR-R4 has no designed mechanism for this. Snapping each block back to one of its fixed lattice points might play that role, but no record says so or tests it (mechanism board). This is also a hypothesis.
What "replaces" does not mean yet
Having a replacement on paper is not the same as having a working model:
- No replacement has matched its transformer part on quality. Every attention-style read that learned well in the older decimal experiments had help from softmax during training, and the best softmax-free read was rejected (softmax-free read; audit synthesis (
docs/labs/repo-renormalization-2026-10-19/audit/synthesis.md), section 1). - No two parts have been trained together except on the current discrete path. That path has no feed-forward layer and no position mechanism, and it scores below a table of word counts (measurement, K16 outcome).
- Several other things are not built: using an exact address as the read, any consumer of the object store, an exporter that turns a trained model into a D11 file, and an output step that avoids scanning the whole vocabulary (audit synthesis, section 1).
The project does not compare its models against transformers. It compares them against count tables and earlier geometric models, and grades them with fixed, deterministic checks (charter, sections 3.10 and 4.4).
Where it stands
| Measure | Value | Source |
|---|---|---|
| Mechanisms | 27: 10 ACTIVE, 4 READY, 13 PARKED | mechanism board |
| Transformer parts with no mechanism | 2: position, normalization | mechanism board slot map |
| READY parts never trained in | Hamilton gate, recurrence gates, SpiralCore/FBS (decimal form), object store | mechanism board |
| Parts trained together | Only the discrete path: tokenizer, codebook or cells, reads, admission, hinge, discrete learning | the model page |
| Best trained result vs count tables | 65 and 463 vs 1,226 and 6,300, of 14,287 | K16 outcome, count tables |
Go deeper
- What UOR-R4 is and where we are.
- The model, end to end: the current pipeline stage by stage.
- Mechanism board: one page per mechanism, with reopen conditions for PARKED ones.
- Geometry: the mathematics behind the replacements.
- Open questions and the history of how each part was tried.