Skip to content

Audit: Studies & Research (April 2026)

A critical self-review of the AFT research corpus — strengths, gaps, and priority order for closing the remaining evidence. Intended both as an internal compass and as reviewer-anticipation prep for the arXiv submission.

1. Snapshot

AFT is shipped as a five-layer emotional memory library with a theory stack (philosophy → neuroscience → computational models → design principles), 9 families of empirical studies, a pre-registration + 11 addenda (B/D/E/F/G/H/I/J/K/Hd2-PowerTopUp/M), and a canonical claim matrix. The public claim ceiling, per 10_scientific_quality_bar.md, is currently "strong theory-driven prototype with early controlled evidence + one honest external negative." Of the seven tracked claims in claim_validation_matrix.json, three have controlled/early evidence, one carries strong intra-theory evidence, one is implemented only, and one is not_established; none carry ecological validation yet.

Quality gate status (2026-05-13, updated from 2026-05-06): - Gate 1 (external benchmark): CLOSED — LoCoMo H1/H2 FAIL, honest negative result documented. Add. J Pareto sweep (Hj1 FAIL 2026-05-06): no AFT config ≥ naive_rag on any category; per-task base_weights tuning line closed. - Gate 2 (human eval): OPEN — kit shipped (benchmarks/human_eval/), zero rater ratings collected. - Gate 3 (architecture attribution): CLOSED — Hd1 PASS (Addendum D), aft_noAppraisal Δ=+0.23 vs naive_cosine. - Add. G — Hg1 LLM dual-path (CLOSED 2026-05-07, FAIL): Pre-registered study on LLM dual-path with real appraisal engine. Hg1 FAIL — LLM dual-path degrades realistic-recall top1 (Δ=−0.03 to −0.05 across embedders). Synchronous-keyword destructive override the dominant pattern. See benchmarks/preregistration_addendum_g_closure.md. - Add. K — Hk1 DailyDialog ecological replication (CLOSED 2026-05-07, FAIL Branch B): N=120 synthetic personas, 396 affect-conditioned queries, multilingual-e5-small. AFT top1=0.212 vs naive_cosine=0.220 (Δ=−0.008, p_holm=1.000, d=−0.015). AFT advantage is regime-specific (curated affective benchmarks); does not extend to naturalistic short-turn dialogue. Exploratory: affective_trajectory Δ=+0.103 (N=39, underpowered). See benchmarks/preregistration_addendum_k_dailydialog_closure.md. - Hd2-PowerTopUp (CLOSED 2026-05-07, FAIL-FAIL Branch C): Power top-up to N=120 for Italian and Spanish on multilingual-e5-small. IT Δ=+0.058 p=0.276; ES Δ=0.000 p=1.000 — both FAIL at declared power. Cross-language evidence scoped to Spanish-SBERT N=80 (exploratory positive only). See benchmarks/preregistration_addendum_hd2_powertopup_closure.md. - Add. M — Hm1 French slice (CLOSED 2026-05-16, PASS Branch A): Pre-registered FR realistic_recall_v2 (me5, N=120, 30 hand-authored native scenarios, 2-session). AFT top1=0.31 vs naive_cosine=0.12 (Δ=+0.18 [0.11, 0.26], p<0.0001, g=0.424). Cross-language evidence is now two-positive (Spanish-SBERT exploratory + French-me5 confirmatory). See benchmarks/preregistration_addendum_m_fr_closure.md. - Add. I — Hi3 confirmatory (CLOSED 2026-05-06, PASS): N=500 confirms cross-embedder amplification on semantic_confound (Δ=+0.090, d=0.257, Holm p=0.0234). Hi3_recency PASS; Hi3_arc FAIL. Mechanism (Hi2: link-type/strength differentiation) remains exploratory. See benchmarks/preregistration_addendum_i_closure.md.

2. The corpus at a glance

Research documents → role in the theory-to-evidence pyramid

# File Role
01 01_foundations.md Why emotion is a memory dimension (Aristotle → enactivism)
02 02_neuroscience.md Dual-path encoding, arousal consolidation, appraisal, reconsolidation
03 03_computational_models.md 20+ prior models; identifies the gap AFT targets
04 04_state_of_art.md LLM memory systems; positions AFT against Mem0/MemGPT/Letta
05 05_design_principles.md 5 layers + 9 principles; pseudocode for core operations
06 06_bibliography.md ~120 references across disciplines
07 07_related_work.md 29-system comparative table
08 08_limitations.md Honest account of what is not yet covered
09 09_current_evidence.md Live evidence matrix; study ladder
10 10_scientific_quality_bar.md Pre-committed gates before upgrading the claim

Benchmark families → evidence level

Family Path Evidence level
Fidelity benchmarks/fidelity/ 1 — theory fidelity (126 cases, 20 phenomena)
Comparative benchmarks/comparative/ 2 — controlled retrieval (SBERT + hash)
Appraisal quality benchmarks/appraisal_quality/ 3 — appraisal, LLM-dependent
Appraisal confound benchmarks/appraisal_confound/ 2–4 — architecture attribution (Gate 3)
Realistic replay benchmarks/realistic/ 4 — multi-session (N=100, v1.4)
Ablation benchmarks/ablation/ 4 — layer isolation
LoCoMo benchmarks/locomo/ 4–5 — external conversational QA (Gate 1)
Human eval benchmarks/human_eval/ 5 — human ecological (Gate 2)
Performance benchmarks/perf/ — (non-scientific; throughput/latency)

3. Strengths

  • Theory-fidelity density. 126 cases across 20 phenomena in benchmarks/fidelity/ directly validate Russell mood-congruence, Yerkes- Dodson arousal modulation, ACT-R power-law decay, spacing effect, spreading activation, and state-dependent recall.
  • Pre-registered evidence programme. preregistration.md (S1 LoCoMo, S2 realistic v2) + preregistration_addendum_v2.md (Addenda A/B/C) commit hypotheses, effect-size targets, and Holm correction before results.
  • Claim-ceiling discipline. claim_validation_matrix.json freezes the "allowed public wording" per claim and ties each to evidence, limitations, and the next study — forcing paper and README to match evidence, not ambition.
  • Reproducibility primitives. benchmarks/common/statistics.py (bootstrap, McNemar, Holm), per-run results.protocol.json, deterministic seeds, dual embedders (hash + sbert-bge). The .dryrun.* convention in benchmarks/locomo/ gitignores partial runs so they cannot be cited as evidence.
  • Architecture attribution pre-scaffolded. benchmarks/appraisal_confound/ anticipates the single most obvious reviewer objection ("AFT wins because the LLM appraisal leaks affect into the query") and wires a no-LLM keyword- appraisal condition to test it. Runner is implemented and awaiting execution.
  • One per-challenge result survives Holm. On the realistic replay semantic_confound subset (N=30), AFT top1 = 0.73 vs naive_cosine = 0.47, Δ = +0.27 [0.10, 0.43], p_adj = 0.006 — the first per-slice AFT advantage that survives multiple-testing correction.

4. Gaps and risks

Each gap is named, evidence-backed, and ranked by how much it constrains the public claim ceiling.

G1 — No human-ecological evidence (blocks Gate 2). benchmarks/human_eval/ has packets, protocol, and a rater-instruction file, but no ratings.jsonl. Claim models_human_emotional_memory stays at not_established. Nothing in the corpus shows that humans perceive AFT output as more coherent or useful than a cosine baseline.

G2 — LoCoMo run completed; Gate 1 did NOT pass (negative result, now committed). benchmarks/locomo/results.json and results.md committed 2026-04-27. Full 10-conversation run (1986 QA pairs), both aft and naive_rag:

System N F1 BLEU-1 Judge Acc
aft 1540 0.168 0.169 0.279
naive_rag 1540 0.271 0.263 0.441

Hypothesis tests (pre-reg S1, Holm-Bonferroni n_bootstrap=2000): - H1 (F1): Δ = −0.101, p_one = 1.0 → FAIL - H2 (judge_acc): Δ = −0.159, p_one = 1.0 → FAIL - Gate 1: FAIL

Interpretation: AFT retrieval is optimised for affective relevance, not factual QA. On LoCoMo — where the gold answer is an explicit factual statement from a specific turn — mood-congruent and appraisal-weighted retrieval actively mis-prioritises emotionally salient but factually irrelevant turns. This is an honest negative result. The "external benchmark advantage" wording remains prohibited; 10_scientific_quality_bar.md status updated accordingly. Claim locomo_external_qa_negative added to claim_validation_matrix.json.

Add. J Pareto sweep — CLOSED 2026-05-06. Hj1 FAIL. Pre-registered base_weights tuning study (10 configs × 200-QA stratified subsample, seed=42). No AFT config achieves F1 ≥ naive_rag on any category. Best: W2 (sem=0.50, mood=0.30, no resonance) aggregate F1=0.1765 vs naive_rag=0.2092. Per-task base_weights tuning line closed. See benchmarks/preregistration_addendum_j_closure.md.

G3 — CLOSED 2026-04-27. Appraisal confound resolved. Confirmatory run (seed=1, Addendum D Hd1) committed to benchmarks/appraisal_confound/results.confirmatory.{json,md}.

Results (SBERT, N=100, seed=1):

System top1
aft_noAppraisal 0.780
naive_cosine 0.550
aft_keyword 0.160

Hd1 PASS: aft_noAppraisal (0.78) > naive_cosine (0.55), Δ=+0.23 [+0.12, +0.34], d=0.515 — effect replicates with new seed. AFT architecture (preset affect, no appraisal) reliably outperforms naive cosine.

Gate 3 interpretation: the realistic-replay advantage is attributable to the AFT retrieval architecture, not to LLM-injected appraisal signals. aft_keyword (0.16) confirms that keyword appraisal degrades preset affect (same destructive-override finding as G3 discovery and Addendum F).

Initial study (2026-04-26, seed=42, SBERT, N=100): benchmarks/appraisal_confound/results.json — Ha2 FAIL (aft_keyword vs naive_cosine, Δ=−0.39), Hb2 FAIL (|aft_keyword − aft_noAppraisal|=0.62). Pre-registered hypotheses both failed because keyword appraisal destructively overrides hand-curated preset affect. The descriptive architecture-attribution comparison (aft_noAppraisal vs naive_cosine, Δ≈+0.23 [+0.12, +0.33]) was the signal that motivated Hd1 as the confirmatory re-pre-registration.

G4 — ✅ CLOSED (2026-04-27): Statistical power confirmed at N=200. Realistic replay v2 (50 scenarios, 200 queries, 5 challenge types × 40). SBERT bge-small: AFT top1=0.53 vs naive_cosine=0.33, Δ=+0.205 [0.150,0.265], p_bootstrap<0.001, d=0.49. Per challenge type (SBERT):

Challenge N AFT top1 naive top1 Δ [95% CI]
semantic_confound 40 0.72 0.47 +0.25
affective_arc 40 0.42 0.15 +0.27
momentum_alignment 40 0.60 0.33 +0.27
same_topic_distractor 40 0.75 0.62 +0.13
recency_confound 40 0.15 0.05 +0.10

Aggregate bootstrap CI excludes 0 on all metrics. G4 is resolved.

G5 — ✅ CLOSED (2026-04-27): Cross-embedder advantage confirmed. v2 cross-embedder results: SBERT Δ=+0.205 (d=0.49); e5-small-v2 Δ=+0.155 (d=0.31). Both embedder classes yield consistent positive advantage with bootstrap CI excluding 0. The earlier hash-embedder inversion at v1.4 was a hash-embedder artefact — dense-embedding-conditional scope remains honest, but the claim now holds on two meaningfully different dense embedder classes. Results: benchmarks/realistic/results.v2.{sbert,e5}.{json,md}.

G6 — Language coverage. ✓ CLOSED (2026-05-02) Italian multilingual slice executed on two embedder configurations: SBERT all-MiniLM-L6-v2 (EN-centric) and intfloat/multilingual-e5-small. AFT hit@k advantage is significant on both (p=0.0005 SBERT, p=0.001 me5); delta is consistent at ≈+0.15–0.16. Cross-embedder robustness documented in 12_multilingual_followup.md and benchmarks/preregistration_addendum_h.md. Language claims now extend to "Italian and English conversational text". See also Addendum M FR (Hm1 PASS, 2026-05-16): French me5 N=120 confirms cross-language effect.

G7 — Dimensional coverage. ✓ CLOSED 2026-05-05 CoreAffect is now 3D PAD (valence × arousal × dominance), shipped in v0.8.2 (commit 8b9ddbe). AppraisalVector.coping_potential populates CoreAffect.dominance; MoodField uses dominance directly (no heuristic). Default dominance = 0.5 when no appraisal engine is active. No study yet tests whether the dominance dimension adds discriminative power over the 2D baseline — the omission of a dominance-specific study is now a deliberate choice, disclosed in §Affect model scope of the paper.

G8 — Paper narrative risk. ✓ CLOSED 2026-05-05 The ceiling-effect framing and the controlled-vs-realistic distinction are now explicitly stated in paper/main.tex §Comparative Benchmark → Results (around "The tie at $N=20$ reflects a dense-embedder ceiling effect…"). Both the controlled tie (0.80 vs 0.80, Δ=0.00, N=20) and the realistic win (0.53 vs 0.33, Δ=+0.21, N=200) are explained together; reviewers are not left to reconstruct the argument.

G9 — Under-benchmarked implemented mechanisms. ✓ CLOSED 2026-04-26 Dual-path encoding and APE-gated reconsolidation ablations executed (pre-reg v3, Addendum E; benchmarks/ablation/results.sbert.{json,md,protocol.json}).

Results (SBERT, N=100, seed=0):

Variant top1 Δ vs full Holm p_adj He verdict
full 0.70 — — baseline
no_reconsolidation 0.70 0.00 1.000 He2 FAIL
dual_path 0.35 −0.35 0.000 He1 FAIL

He2 (no_reconsolidation): Δ=0.00, p_adj=1.000 — reconsolidation is implemented per theory (Nader & Schiller 2000) but does not measurably affect retrieval on this benchmark. The benchmark may not exercise the conditions that trigger reconsolidation windows (high APE at retrieval).

He1 (dual_path): Δ=−0.35, p_adj≈0 — as expected (caveat documented in pre-reg v3): the KeywordAppraisalEngine degrades preset affect at the slow-path elaborate step (same destructive-override finding from G3/Addendum A). He1 measures "no appraisal" vs "deferred bad appraisal" — not the more informative "does deferral mitigate synchronous keyword destruction?", which requires a synchronous-keyword baseline. Addendum F CLOSED (2026-04-27) — see below.

Addendum F — Hf1 CLOSED 2026-04-27 (benchmarks/preregistration_addendum_f.md)

aft_keyword_synchronous variant added: synchronous KeywordAppraisalEngine at encode time, no elaborate(). Tests Hf1: dual_path.top1 > aft_keyword_synchronous.top1.

Results (SBERT, N=100, seed=0):

Variant top1 Δ vs full Hf1
dual_path 0.350 −0.35 —
aft_keyword_synchronous 0.070 −0.63 —
Δ_Hf1 (dual_path − sync) — +0.280 PASS

Hf1 PASS: Deferring keyword appraisal to the slow path reduces destructive override (0.35 vs 0.07). The LeDoux dual-path architecture provides a measurable benefit relative to synchronous keyword appraisal, even though both are inferior to no appraisal (preset affect only). Note: aft_keyword_synchronous actual=0.07 is slightly below the pre-registered expected range (0.10–0.20); destructive override was more severe on this call path than anticipated.

5. Theory–evidence coherence check

Layer Tested by Coverage
Core affect (Russell) fidelity, comparative, realistic strong
Affective momentum fidelity/test_momentum.py, ablation (no_momentum) direct
Mood field (Heidegger) fidelity/test_mood_congruent.py, fidelity/test_mood_adaptive.py, ablation (no_mood) direct
Appraisal (Scherer) appraisal_quality/, ablation (no_appraisal), appraisal_confound scaffolded, thin (LLM-dependent)
Resonance (spreading + Hebbian) fidelity/test_spreading_activation.py, ablation (no_resonance) direct
Dual-path encoding (LeDoux) ablation dual_path (He1 FAIL; expected) + Addendum F Hf1 PASS (dual_path > synchronous) closed
APE-gated reconsolidation ablation no_reconsolidation (He2 FAIL; benchmark doesn't exercise reconsolidation triggers) closed (null result)

All seven layers benchmarked. Dual-path shows null vs full (expected: keyword appraisal destructive) but Hf1 PASS confirms deferred appraisal outperforms synchronous — the LeDoux dual-path architecture provides measurable benefit at the appraisal-timing level. Reconsolidation shows null result on this dataset (triggers not exercised).

Ordered by evidence-per-cost and by which gap most constrains the claim ceiling. Each item is framed as "do X before claiming Y."

  1. Execute appraisal_confound (Gate 3). ✓ DONE 2026-04-27. Hd1 PASS: aft_noAppraisal (0.78) > naive_cosine (0.55), Δ=+0.23 with new seed. "Architecture drives the advantage" is now claimable.
  2. ~~Execute LoCoMo S1 (Gate 1).~~ Completed 2026-04-27. Gate 1 NOT passed. AFT F1=0.168 vs naive_rag=0.271. External-benchmark advantage wording remains prohibited. See benchmarks/locomo/results.json.
  3. Ship the human-eval pilot (Gate 2). Even a minimal N=5 raters × 10 scenarios lifts claim models_human_emotional_memory out of not_established. Addendum C already specifies Krippendorff α ≥ 0.67 as the publishability criterion.
  4. ~~Expand realistic replay to v2, N≥200, with cross-embedder slice.~~ Completed 2026-04-27. G4 + G5 both closed. SBERT Δ=+0.205 d=0.49, e5-small-v2 Δ=+0.155 d=0.31. Per-challenge advantage confirmed at N=40/type.
  5. Add a dedicated ablation for dual-path encoding and APE-reconsolidation. ✓ Done 2026-04-26 (G9 CLOSED). He1/He2 both FAIL on realistic_recall_v1.4 — null results are documented with honest interpretation. Paper's layer-by-layer contribution table is now complete.
  6. ~~Multilingual slice (Addendum B).~~ ✓ Done 2026-05-02 (Addendum H). Italian SBERT + me5-small both executed. hit@k SIG on both embedders (p=0.0005 / p=0.001). Cross-embedder robustness confirmed. Language claims updated to "Italian and English conversational text". Addendum M (2026-05-16) extends to French (Hm1 PASS, me5 N=120, Δ=+0.18 p<0.0001 g=0.424).

7. Reviewer anticipation (paper-specific)

Five questions a skeptical arXiv reader will ask, with the strongest currently-defensible answer and the honest remaining gap.

Q1 — "Does the advantage come from your architecture or from your LLM appraisal leaking affect into the query?" → Current answer: Closed (Gate 3, G3, 2026-04-27). aft_noAppraisal (pure preset affect, no appraisal engine) achieves top1=0.78 vs naive_cosine 0.55, Δ=+0.23 [+0.12, +0.34], d=0.52, seed=1 confirmatory. The advantage holds without any LLM involvement — it is architectural.

Q2 — "Does AFT beat cosine on real human conversation, not just on a benchmark you built?" → Current answer: No — LoCoMo S1 ran on 1986 QA pairs and AFT underperformed naive_rag (F1 0.168 vs 0.271, Gate 1 FAIL). This is an honest negative result: AFT's affective weighting does not help on factual open-domain conversational QA. The advantage remains restricted to controlled and realistic affect-oriented replay benchmarks. Gap (updated): Gate 1 is closed with a negative result. The claim ceiling is unchanged; LoCoMo advantage wording is still prohibited.

Q3 — "Do humans actually prefer AFT's output?" → Current answer: unknown — human_eval packet is ready, no ratings collected. Gap: Gate 2 (G1).

Q4 — "Why does AFT tie cosine on your controlled probe but beat it on your realistic probe?" → Current answer: the controlled probe hits a dense-embedder ceiling at N=20 (both systems at 0.80); the N=200 realistic probe discriminates (Δ=+0.21, d=0.49, SBERT; Δ=+0.16, d=0.31, e5-small-v2). Gap: ~~stated once in 09_current_evidence.md but not framed explicitly in paper/main.tex (G8)~~ — G8 CLOSED 2026-05-05: explicitly framed in paper/main.tex §Comparative Benchmark → Results.

Q5 — "Does this work outside English, on a sparse embedder, with the dominance dimension?" → Current answer (updated 2026-05-05): - Multilingual: Italian (G6, Hd2_IT PASS, Δ=+0.163, p=0.012) and Spanish (Hd2_ES SBERT PASS, Δ=+0.138, p=0.045) confirmed; me5-small ES borderline (Δ=+0.113, p=0.110). French (Hm1, Addendum M, 2026-05-16) confirms me5 at N=120 (Δ=+0.18, p<0.0001, g=0.424). Scope extended to "English + Italian + Spanish + French + dense embedding". - Sparse embedder (G5): still scoped out; hash-embedder gap Δ≈+0.06 ns. Documented in §Language and embedding scope of the paper. - Dominance (G7): shipped in v0.8.2 — CoreAffect is now 3D PAD. No study yet isolates dominance's discriminative contribution. Remaining gap: G5 (sparse embedder), dominance ablation study (no pre-reg yet). G6 and G7 are closed.