BILM — Biologically Inspired Language Model

BILM is the applied sequence modeling track of the UnikAI research program. Derived from the BIM unified substrate, it isolates the mathematical efficiency of biological sequence memory to address the sequence learning and memory retention limitations of the standard Transformer architecture.

BILM is designed as a CPU-native, online learning sequence engine that trains on streaming bytes while mitigating catastrophic forgetting through sparse activation patterns.

Limitations of the Transformer Architecture

Traditional Transformer architectures rely on static, dense floating-point weight matrices optimized offline via backpropagation. This design introduces three core challenges in online environments:

• Catastrophic Forgetting: Dense neural networks struggle with online, continuous learning. Sequential training on new distributions tends to overwrite previously learned features.
• Computational Complexity: The O(N²) attention mechanisms and massive parameter counts require significant compute infrastructure during both training and inference.
• Fixed Context Windows: Transformers cannot update their long-term parameters during inference, relying entirely on the local context window for short-term recall.

Architecture and Key Mechanisms

BILM isolates the core sequence prediction mechanisms from the biological framework to focus on computational efficiency and online adaptation:

• Deterministic Byte-Level Codec: Bypasses traditional Byte-Pair Encoding (BPE) tokenizers. Instead, it maps raw bytes directly to fixed Sparse Distributed Representations (SDRs).
• 0.39% Sparse Activation: Utilizes 16,384-dimensional representations with exactly 64 active bits. This high level of sparsity reduces overlap, minimizing interference between distinct sequences.
• Hierarchical Predictive Cortex: A multi-layered predictive sequence memory utilizing localized learning rules (such as Hebbian updates and localized weight pruning) to learn associations without backpropagation.
• Neuromodulated Plasticity: Uses rolling prediction error (surprisal) to scale the rate of synaptic changes, prioritizing resource allocation for novel or unexpected inputs.
• Auto-Associative Attractor: A recurrent network structure that binds co-active representations for fast retrieval via pattern completion from partial or noisy cues.

Evaluation and Performance

In empirical evaluations, BILM demonstrates stable sequence learning on streaming inputs. Because BILM utilizes a sparse distributed output representation (SDR) rather than a dense softmax probability distribution, traditional Bits-Per-Character (BPC) cross-entropy metrics are not directly applicable. Instead, we evaluate the model using Next-Byte prediction accuracy.

On the standard WikiText-2 dataset, BILM achieves a Next-Byte prediction accuracy of 26.89% after single-pass sequential training on only 100,000 tokens, demonstrating high sample efficiency.

Under continuous sequential training across shifting domains (Domain A: natural language, Domain B: source code), the model maintains its existing synaptic states with 0.0% degradation (perfect retention) of Domain A accuracy when evaluated on a strictly held-out natural language set.

The core engine runs entirely on a single CPU core using JIT-compiled Numba kernels. During training, it operates at ~18 to 31 TPS (Tokens Per Second) with a peak RAM footprint of ~635 MB (which includes the ~268 MB weight matrix of the CA3 Hippocampus attractor network).