ATOMIC: Adaptive Token Operator for Memory-Efficient Inference on Consumer CPUs

A Unified Linear Recurrent Attention Architecture with Constant Footprint and Multi-Step Reasoning Structure.

Authors: UnikAI Lab (www.unikai.in · unikailab@proton.me) · GitHub Repository ↗

Abstract

Standard autoregressive Transformer architectures based on scaled dot-product Softmax Attention exhibit quadratic computational complexity O(T²) during sequence processing and require an unboundedly growing Key-Value (KV) cache of size O(T) during autoregressive generation. On consumer Central Processing Units (CPUs), which are strictly bottlenecked by main memory DRAM bandwidth (40–90 GB/s) rather than peak compute throughput, the expanding KV cache induces devastating L1/L2/L3 cache thrashing, degrading generation throughput as context length scales.

In this work, we introduce ATOMIC (Adaptive Token Operator for Memory-efficient Inference on CPUs), a linear recurrent language model architecture specifically co-designed for modern consumer x86-64 and ARM CPU memory hierarchies. ATOMIC eliminates the KV cache entirely by synthesizing three hardware-aligned components:

• Gated Linear Recurrent Attention (GLRA): Maps multi-head attention into an exact outer-product recurrent state matrix with data-dependent retention gates.
• Zero-GEMM Learnable Token Shifting: Injects directional 1D local sequential context via learned elementwise interpolation with strictly zero matrix multiplications.
• Exact Mathematical Duality: Parallel associative prefix scanning O(T) for multi-threaded training, numerically equivalent (max deviation < 10⁻⁴) to a constant-time O(1) recurrent inference step.

I. The Consumer CPU Memory Wall

Unlike high-end datacenter GPUs (e.g. NVIDIA H100 with 3.35 TB/s HBM3 bandwidth), modern consumer CPUs operate with dual-channel DDR4/DDR5 system memory providing an effective bandwidth of only 40 to 90 GB/s.

During autoregressive generation, each new token has an arithmetic intensity of approximately 1 FLOP per byte transferred. Consequently, generation is entirely memory-bandwidth bound:

• At short context (< 1,000 tokens): The standard Transformer KV cache is small enough to fit within CPU L1/L2 on-chip caches, yielding high initial generation rates.
• At long context (> 2,000 tokens): The KV cache swells past on-chip cache capacity, spilling into DRAM. Every newly generated token forces the CPU to stream tens of megabytes across saturated DDR memory channels. Generation speed drops drastically.

II. The ATOMIC Architecture

ATOMIC replaces dynamic KV buffer allocation with a fixed-size recurrent state matrix S_t ∈ ℝ^{d_k × d_v}:

Token x_t ─> [Zero-GEMM Token Shift] ─> Projections (Q_t, K_t, V_t, Gate_t)
                   │
                   ▼
       Recurrent State Update:
       S_t = α_t ⊙ S_{t-1} + K_tᵀ V_t   (Fixed 192 KB per stream)
                   │
                   ▼
       Output = (Q_t S_t) ⊙ OutputGate ─> [SwiGLU FFN] ─> Next Token Logits

Because the state update consists solely of an outer product and elementwise gating, state size is strictly invariant to prompt length. At 4,096 tokens, ATOMIC occupies 192.0 KB fixed memory—a 256× compression compared to the 49.15 MB required by standard Transformer KV caches.

III. Empirical CPU Benchmarks (Intel Core i7-12700H)

Evaluated on an Intel Core i7-12700H (14 cores, 20 threads, FP32, single socket, averaged across 3 independent trials on the 5.8M micro configuration):

IV. Proof-of-Concept Reasoning Model Trained on CPU

To verify learning dynamics, we trained atomic-reasoning-prod (3.31M parameters) completely from scratch on a single consumer CPU in 12.6 minutes (757s, 3 epochs).

• Convergence: Training loss decreased from 4.25 to 0.57; validation perplexity converged to 1.77.
• Chain-of-Thought Scratchpad Formatting: The model learned consistent syntactic formatting (98.9% validity) emitting structured <think>...</think> traces followed by <answer>...</answer> blocks.
• Open Limitations: While formatting is mastered, exact mathematical answer accuracy remains early-stage (2.2% exact match). Furthermore, synthetic Multi-Query Associative Recall (MQAR) remains an open challenge at sub-10M parameter budgets (2.5% vs 5.0%), indicating that scaling parameter capacity and pretraining data will be required for complex multi-hop deduction.

V. Quickstart & Installation

ATOMIC is fully packaged and installable directly via pip:

# Install directly from GitHub:
pip install git+https://github.com/tgakathunderr/Atomic.git

# Or with serving & ONNX export dependencies:
pip install "atomic-lm[all] @ git+https://github.com/tgakathunderr/Atomic.git"

Python usage:

import atomic

# Initialize model preset
config = atomic.AtomicConfig.micro()
model = atomic.AtomicForCausalLM(config)
tokenizer = atomic.AtomicTokenizer.from_pretrained("models/atomic-reasoning-prod/tokenizer.json")

# Generate with constant recurrent state
output = atomic.generate(model, tokenizer, "Question: 45 + 55?\n<think>\n")
print(output)

VI. Citation

@article{atomic2026,
  title={ATOMIC: Adaptive Token Operator for Memory-Efficient Inference on Consumer CPUs},
  author={UnikAI Lab},
  year={2026},
  journal={UnikAI Lab Research Technical Reports},
  url={https://github.com/tgakathunderr/Atomic}
}