Geometric stack (layered quaternion-recurrence language model)

Status: PARKED. The version that was trained leaned on arithmetic the project later banned (softmax reads, sigmoid and SiLU gates, dense matrix products), and it never met the first milestone's bars. Its quaternion "spinner" idea was kept. A rules-compliant rebuild exists only as a small, untrained extract. See what would reopen it.

Labels used below: proof means derived from definitions or code, measurement means a recorded run, hypothesis means an untested idea. See docs/formal_vocabulary.md.

In plain words

This was the project's main trained language model from late September to mid-October 2026, between 7 and 214 million parameters. It reads text through a stack of layers. Most layers carry a running memory made of four numbers, a "spinner" called a quaternion. At every word the spinner is turned a little and the old memory fades. A few layers look back over earlier words, and every layer ends in a small feed-forward block. It was trained on GPUs, then rounded to whole numbers so a laptop could run it. It produced chat-style replies, though only 43 of 232 reply checks passed and it was served through the integer engine on pod CPUs. It used arithmetic the project later banned, and it never reached its quality bars.

What it replaces in a transformer

The whole network. The recurrence layers do the job that attention does in a transformer, carrying context forward. The read layers and feed-forward blocks, though, are the transformer's own parts. The project's own audit in October found that the served model was a transformer-shaped dense stack with a quaternion-mixed residual. The geometry was a thin layer inside it (see the figure).

Left, where one token's arithmetic went in the 29-million-parameter stack. Middle, what one recurrence lane does. Right, which parts of the trained form the later rules allow.

Its chapter in the project's history

• 26 to 30 August (planning a decoder). Before building a stack, the project checked whether ordinary attention could live inside the geometry. A rotation-frame version of softmax attention matched a donor model's logits to within 1.05e-5, but it carried no meaning of its own. Trace distillation added nothing (#1002, #1011).
• 30 August. A 7.16-million-parameter softmax language model in R4/Spin frames showed learned attention matters: loss per token 2.1274 with it, 4.8048 without. A single frozen continuation to 150 million tokens ended at 1.5728, short of its 1.50 bar (#1014, #1017).
• 25 September. A 1.68-million-parameter recurrent learner with quaternion transport lost narrowly to a matched ordinary one (2.110 against 2.085) (#1389).
• 27 to 29 September. A capacity-matched layered stack reached parity with a transformer control, and the owner made multiplier-free serving the one contract for every lab (D11, #1437, #1443). The stack became the main line.
• 6 to 10 October. The first chat stack was trained, exported and served end to end (#1893). Matching the training data mattered more than adding parameters (#1913). Memory and reply targets were missed.
• 11 October. An arithmetic audit showed the quaternion rotations were 0.05 percent of the arithmetic (#2186).
• 12 to 13 October. The bans on softmax, matrix multiplication and squashing functions arrived (D23, D25, D29, D30). Thirty-three kernels carrying banned content were moved out of every compiled module, and a 24-kernel lawful extract was kept. The discrete path (crates/uor-r4-discrete) then became the live model.

How it works

Layer pattern (definition). Layers repeat in a pattern such as rrarra. Each r is a recurrence layer and each a a read layer. Every layer is followed by a feed-forward block, inside a residual stream with RMS normalisation.

Recurrence (definition). Lanes of four channels hold a quaternion state \(h_t\). Per token,

\[h_t = \lambda_t\,(u_t \otimes h_{t-1}) + \sqrt{1-\lambda_t^{2}}\;c_t ,\]

where \(\otimes\) is the Hamilton product, \(u_t\) is a unit quaternion computed from the input (it starts at the identity), \(\lambda_t\) is a gated decay with timescales spread from 2 to about 1,000 tokens, and \(c_t\) is a width-4 causal convolution of the input. Multiplying by a unit quaternion is a rotation of 4-space, so \(u_t\) turns the memory without changing its length.

Reads and feed-forward. Read layers attend over earlier positions with a dot-product, Euclidean or Lorentz hyperbolic-distance score. Feed-forward blocks are SwiGLU with dense projections. Optional parts were a copy-pointer head, a product-key memory that replaced a feed-forward block or the last read, and flock or starling rank reads.

Training and serving. Trained in float on GPU at 7M, 19.9M, 29M and 214M parameters, then exported to 4-bit grouped weights and served by the integer engine described in lut-serving.

Where the arithmetic went (measurement, an operation count of the deployed 29M stack, width 576, 10 layers). Of 30,270,720 multiply-adds per token, the feed-forward blocks took 39.6 percent, recurrence projections 31.0 percent, reads 16.6 percent (40.5 percent at 4,096 tokens of context) and the output head 7.4 percent. The quaternion rotations took 16,128 multiplies, or 0.05 percent. The prime addresses, zeta phases and exact store took none, because they were not in the trained base (geometry audit).

Rule check.

What has been tried

"Development" means a panel used while fitting. A "control" is a matched comparison model; transformer controls were allowed when these runs were made and are not allowed now (D28).

Reading these honestly. The stack reached parity with a transformer control at small scale, and it was served bit-for-bit through an integer engine. The quaternion transport kept its place in the core on a development measurement. But the pieces that did the work were dense ones, the recurrence was never shown to help in a model built from allowed parts, and no run met the memory or quality bars. A loss win was not a conversation.

What has never been tested

• A stack built only from allowed parts (Hamilton-gate feed-forward, power-of-two recurrence, rank or flock reads), trained at the scale where this one reached 0.877 bits per byte. The 24-kernel extract is qualified at small finite checks only.
• Product-key memory layers with fixed H4 and E8 codebooks (cycle 5; held, never run).
• Trained-in icosian or E8 read addresses with their length carried.
• The Lorentz read at chat scale. The chat base trained with the flat Euclidean read.
• A larger chat data set than the 82-million-token split (none was reachable).
• A fair energy measurement. The only figures come from a truncated capture and a table-emulation test that cost 4.3 times more.
• A step-matched control for the 22 of 40 reading that the half-length memory run produced.

What would reopen it

1. A new pre-registered sheet that rebuilds the stack from allowed parts only, in its own scanned module, with the owner's release (D37). It needs a TEST FITNESS: FIT review first.
2. Report the export gap on the same evaluator as the float score, and aim at the served bar (at most 0.90 bits per byte on the chat held-out stream).
3. At least 2 seeds or 200 fresh rows (D22), compared only with earlier native rungs and count-table references (D28).
4. Replace the dense hotspots (feed-forward 39.6%, recurrence projections 31.0%) with addressed or exact-geometry reads, and show that tokens per second move on the product path.
5. Optional: the quaternion adapter in the post-dose plan (optimization path), conditional and measured against the ceiling.

Also known as

rrarra / rrarrarr / rrarrarrar stack; cycle-4 stack (geometric_s1); chat-29m-B; the 19.9M chat stack; 214M pretrain geo-214m-e448de86; R4/Spin softmax reference (#1014, #1017); joint recurrent learner (the 1.68M precursor); StackArch::Geometric; geometric_stack_native (the lawful extract).

Code

• geometric_stack.rs: the historical float stack (36,976 lines). Inactive source.
• geometric_stack_native.rs and geometric_stack_native_cuda.rs: the qualified extract. Only the allowed projection, recurrence and shared writer run (24 prebuilt kernels).
• cuda_stack.cu: the historical kernel source. Thirty-three banned kernels are excluded from every compiled module.
• stack_memory.rs: the product-key memory. stack_export.rs: the exporter, compiled out.
• joint_model.rs: the September precursor. chat-grade.rs: the chat and memory scorer.
• geometric-stack-native.rs: the current example entry.
• session.rs: the integer engine that served these stacks.
• Evidence: served stacks, module split, cycle 4.

Related mechanisms

lut-serving: the engine that served these stacks bit-for-bit. spin-weights: icosian-pair weights, the allowed form of the stack's quaternion maps. recurrence-gates: the allowed form of the gated recurrence. hamilton-gate: the rotation gate that replaces the SwiGLU block. flock-read and route-read: softmax-free reads for the attention layers. exact-memory and e8-orbit-memory: stores meant to replace the dense feed-forward block and read. read-binding and pointer-key: the supervised read and copy head added here. spiralcore-fbs: the projection the extract runs. rank-margin: the allowed loss. native-geometric: the parallel from-scratch engine. r4g1-graph: the earlier approach this stack replaced. discrete-learning: the live model that took over.