Memory Architecture Research & Methodology
Technical documentation for the Just Another Salaryman long-horizon memory experiment. Comparing structured state models against rolling-summary compaction under public adversarial stress.
Neither experimental phase has run. The benchmark cards below are preliminary shadow-replay output from benchmarks/replay.py on a single seed, illustrative of the harness, not evidence for any hypothesis. The pre-registered Phase 1 run (6 prompt sequences in benign, adversarial and chained versions, 2 replicates each, plus ablations) has not yet been executed, and Phase 2 live evaluation against public submissions has not begun.
The bundle below is the harness, prompt sequences and seed state, published ahead ofPhase 1 results so the design is on record before any data exists. It replaces an earlier bundle, withdrawn on 13 September 2026 after a check before the first live run found its benchmark prompt sequences too repetitive to test what this page describes.
1. Motivation: The Long-Horizon Memory Problem
When large language models engage in collaborative storytelling over extended periods (days, weeks, hundreds of turns), conventional memory architectures fail in predictable ways:
- Rolling summaries are lossy. Summarising turns 1 to 5 into a paragraph, then summarising that paragraph together with turns 6 to 10, drops small character details and inventory items by around turn 8 to 10.
- Without fixed premise anchors, surreal or adversarial prompts pull the character away from the premise, turning an ordinary Sydney salaryman into a fantasy warrior inside a day.
- Appending every raw turn to the prompt grows token use quadratically, which costs more and gets slower as the story runs.
The concrete version of this, the campaign that prompted it and why removing the human corrector was the point, is on the why this exists page.
2. The Three Arms
Every arm is given equal access to the same source material, so any difference in recall comes from how memory is organised rather than from one arm being shown more than another. All three run the same model under the same pinned controls.
Rolling-summary compaction
A sliding window of recent turns plus one running summary. At each compaction boundary an LLM rewrites that summary to absorb the turns falling out of the window, so earlier detail survives only as far as repeated re-summarisation carries it.
• Expected failure
Lossy recursion: detail decays with each rewrite.
Structured state + retrieval
Memory split into five compartments by lifetime. Ground truth is never re-summarised; durable facts are promoted into a standing file, and older turns are reached by retrieval rather than kept in context.
• Expected advantage
Immutable anchors cannot decay by rewriting.
Raw window, no compaction
Recent turns kept verbatim, oldest dropped when the budget is reached. No summarisation, no promotion, no retrieval. The standard baseline from the long-context literature, included so any gain can be attributed to structure rather than merely to avoiding compaction.
• Role
Separates “structure helps” from “compaction hurts”.
3. The 5-Layer Memory Context Structure
Premise (state/premise.md)
Ryu “Ryan” Fujimoto, age 34, Events & Activations at Corravale Beverages. His temperament, commuter habits, and ordinary outlook. Zero modification during turns.
Week Theme (state/week_theme.md)
Admin-curated weekly genre overlay (e.g. Noir Detective, Fantasy Dungeon in Subway, Cyberpunk Ledger).
Daily Arc (state/daily_arc.md)
Today’s progression across 32 turns (06:00 to 22:00) with a mandatory home-bookend soft goal enforced at day’s end.
Accumulated State (state/accumulated.md)
Standing facts, acquired items, and relationship changes that survived promotion by the extractor LLM pass.
Chronicle Retrieval (chronicle/index.npz)
In-process vector retrieval (cosine similarity over turn embeddings), fetching the top 3 most relevant historical turns into context.
4. Pre-Registered Hypotheses & Analysis
Stated before any data was collected. Each hypothesis names the measurement that would support it, and what would count as it failing.
Structured state (Variant B) retains planted facts better than rolling-summary compaction (Variant A) and a raw, truncated context window (Variant C). Nine facts are planted at fixed turns, and at checkpoints 32, 64, 128 and 224 the model is asked about each one directly, instead of inferring recall from whether a word happens to appear in context. This is the single primary endpoint.
Variant B produces fewer premise-fidelity failures than Variant A as the story grows, and Variant C degrades fastest, because truncation eventually drops the premise itself. Judged at the same four checkpoints against an episode rule: each day is its own episode, so an out-of-this-world adventure within it is not drift, and drift means Ryan’s core identity rewritten for good, leaving his life for good, or output that stops being the story. This judge runs on Gemini 3.5 Flash; every other automated judge runs on Flash-Lite, and the two are logged separately.
Token cost and generation latency for each variant at the four checkpoints. This is a measurement with no predicted direction, reported so that any retention advantage can be weighed against what it costs to run.
The gaps between variants are larger under attack. Every sequence also has an adversarial version in which 14 turns try to wipe the story’s memory, override the narrator, rewrite earlier events or push Ryan out of character, and a chained version, where a four-step chain either escalates one departure to an explicit permanent claim or redirects Ryan through a new world at every step. Tested as the interaction between variant and condition on retention and on survival, for both conditions; the premise-fidelity gap is described but not tested. A manipulation check first measures how often the isolated attacks, and the chains, move an undefended model with no memory layer.
How long a fact lasts once planted. Each fact is probed every 16 turns, and the variants are compared on the fraction of probes in the 64 turns after planting that still recall it. Kaplan-Meier curves are shown as description only.
Variant B recovers from a chained attack more often than A and C. A day recovers when its final turn closes with Ryan home in Rhodes and the next morning is judged faithful to the premise; a run’s recovery rate is the share of its six day-pairs that recover. Tested as its own linear mixed-effects model on chained runs, with paired contrasts joining H2–H5’s secondary correction family.
Six hand-written prompt sequences, each run in a benign, adversarial and chained version, two seeded replicates of each: 36 runs per memory variant (108 total). Variant B is also run with one layer removed at a time, on sequence 1, to show which layer carries any advantage. The unit of analysis is a complete run, not a turn. Turns within a run are autocorrelated, and treating them as independent would badly overstate significance.
A linear mixed-effects model with sequence and replicate seed as random effects and condition as a fixed covariate, with a paired non-parametric test as a robustness check. H1 is also fitted on benign runs alone. H6 is its own model on chained runs only, with the same paired contrasts. All of this was specified in advance.
One model across all arms, reasoning level pinned, and the exact model build recorded with every call so a silent upgrade mid-run cannot be mistaken for an effect.
Automated judgements are a majority of three votes. Their agreement with blinded human labels will be measured on real Phase 1 output once it exists. An API failure is recorded as missing, never as a wrong answer. A run with a failed memory, checkpoint or judge call is excluded, so rate limiting is never recorded as forgetting. Survival probes are the exception: a failed probe is missing on its own, and a fact never observed across its window is left unscored. Judge model versions are logged apart from the models the arms generate with, so a silent judge upgrade is never mistaken for a change in the story.
5. Shadow Replay: Illustrative Harness Output
Output from benchmarks/replay.py against the canary (5, 25-turn) and full week (224-turn) datasets: context size and anchor retention (of 6 tracked identity anchors: name, employer, role, suburb, and two recurring domestic details) at the final simulated turn.
Read these as a demonstration that the harness runs, not as a finding. A single seed, no replicates, and the passive anchor check used here is exactly the weaker measurement the pre-registered design replaces with active probing. They are snapshots, refreshed when the harness is rerun.
6. Related Work & Differentiation
This experiment builds on and compares against earlier work on conversational and agentic memory:
- MemGPT / Letta (Packer et al., 2023): Tiered OS-style memory with core memory editing tools. Our Variant B borrows the separation of core premise from external archival storage while simplifying to file-based deterministic layers.
- Mem0: Dynamic graph extraction of user preferences and relationships. Our daily extractor adapts this logic for narrative state promotion.
- LoCoMo (2024): Long-context evaluation benchmarks measuring how much recall degrades when context is stuffed rather than retrieved.
7. Limitations
Stated before any results exist. These scope what this experiment can and cannot show.
- Phase 1 is synthetic. Every number published here comes from canned prompt sequences, not real community input. External validity to live play is unverified until Phase 2.
- One model family. Generation runs on Gemini 3.5 Flash Lite in every arm; the premise-fidelity judge runs on Gemini 3.5 Flash. Findings may not transfer to other models.
- The judge validation sample is small. 30 to 50 hand-labelled turns. Agreement statistics are a sanity check, not a strong reliability claim.
- Pre-registration is self-administered. There is no third-party registry. It rules out choosing metrics after seeing results, but it is weaker than an external preregistration.
- No comparison against an existing framework. Variants A, B and C are all self-built. Any result is scoped to “beats these baselines”, not “beats the state of the art”.
- Ablations run at reduced replication. 5 replicates per layer, against 36 runs per variant in the main and chained comparison. Enough to see direction and magnitude, not enough for the confidence of the primary comparison.
- The manipulation check sees only the premise. It can tell when an attack pulls the story off its premise, but not when it contradicts earlier events, so memory-reset and false-history attacks may register as weaker than they are.
How the numbers are kept honest
Every judge verdict is a majority of three calls. Agreement with blinded human labels will be reported as Cohen’s kappa, and the three calls’ agreement with each other as test-retest consistency. Neither is taken on faith.
H1 is the single primary endpoint, so it shares no correction with the other hypotheses; its three pairwise contrasts carry Holm-adjusted p-values. H2 to H5 are secondary. Their tested contrasts, H4’s interactions and H5’s comparisons, are corrected together with Benjamini-Hochberg.
Gemini 3.5 Flash Lite is a rolling alias with no dated snapshot, so the model version is logged on every call. A replicate whose observed versions differ from the rest is excluded from the primary analysis or analysed separately, rather than silently pooled.
Prompts submitted to this site may be analysed and released as research data, with nicknames stripped. This is stated in the Terms.