Phase-Associative Memory: Sequence Modeling in Complex Hilbert Space

Unlocking Next-Gen AI: Complex-Valued Models for Language Understanding

Experiments probing natural language processing by both humans and LLMs suggest that the meaning of a semantic expression is indeterminate prior to the act of interpretation rather than being specifiable simply as the sum of its parts (i.e. compositionality). This observer-dependent act dynamically actualizes meaning under genuine contextuality more consistent with quantum logical mechanisms than with classical Boolean approaches that assume separability, motivating an approach to language modeling that utilizes a Hilbert space formalism. In this work, we introduce Phase-Associative Memory (PAM)—a complex-valued sequence model whose state St ∈ Cdxd accumulates outer products of complex token embeddings retrieved through the conjugate inner product Re(K | Q)/√d and evaluate it against a structurally matched real-valued ablation. Both architectures train stably across a 5M-100M parameter sweep on WikiText-103 under identical conditions; PAM sits at higher absolute loss at every measured scale but improves more rapidly with parameter count, with power-law exponents of -0.15 vs. -0.12 in loss and -0.65 vs. -0.49 in perplexity that narrow the gap between the two architectures monotonically. Further investigation of complex-valued sequence modeling at larger scales could reveal that the loss plateau characteristic of real-valued state-of-the-art language models (e.g. transformers) is reachable with PAM-style architectures with an order of magnitude fewer parameters than the current frontier (~1T), implying that similar capabilities are achievable at sizes runnable on consumer-grade hardware.