Held-out benchmarks

Issue #7 asks for circuits that were not used to choose the model, and for a sensitivity check that is not a retune. This note freezes that partition on 2026-10-11, before any held-out simulation. The lock is validation/benchmarks.yaml (version 1). Changing a held-out case, a check, or a sensitivity corner requires a new version. Criterion v4 stays the baseline: flymsg validate is unchanged, and N1 is not redrawn.

flymsg benchmarks prints the frozen confirmatory case. flymsg benchmarks --run scores that case and the reported case at the scored corner. It does not edit the manifest, criterion v4, or N1.

Result (2026-10-11)

The scored run fails. w_syn stayed 0.192 and th_jump stayed 0. Seeds 0–2, control seed 0. The manifest was not edited after the numbers. Both sensitivity corners fail the same two checks. Neither replaces this result.

The specific check needs the class-matched control at or below 94.1 Hz for DM1_lPN. The control is other olfactory neurons, the same count, excluding the targets and their strong inputs. DM1_lPN does fire. A same-sized olfactory drive fires it too, and the activity does not die out. That is the finding.

ORN_V is the same pattern on the reported case. It does not rescue DM1, and it does not score CO2 avoidance.

These rates are the model under a 100 Hz Poisson input for 300 ms. They are not a match to an odor recording. The first-spike times are the model's medians, not a claim about the odor puffs in Kazama and Wilson.

The numbers are from one run on 2026-10-11. The scorer was not committed yet, so runs/benchmarks/manifest.json records git 51b5305 with a dirty tree. The result itself is commit b3e1859. The csv and the manifest stay in runs/, outside git. Dataset file hashes are in that manifest.

Sensitivity (2026-10-11)

flymsg benchmarks --corners ran the two frozen rows and did not rerun the scored corner. Seeds 0–2, control seed 0, same two cases. The manifest hash is unchanged.

Specific fails in every row: the control stays above the bar (99.7 Hz at the Shiu weight, 61.2 Hz with adaptation). Stability fails in every row. Confirmatory verdict at both corners: FAIL. The scored failure survives both perturbations.

ORN_V stays reported. At the Shiu weight, V_ilPN is 287.8 Hz against 252.2 Hz and V_l2PN is 131.1 Hz against 104.4 Hz, with 7883 neurons still firing. With adaptation, V_ilPN is 165.6 Hz against 125.6 Hz and V_l2PN is 75.6 Hz against 51.1 Hz, with 2685 neurons still firing. Positive passes. Specific and stability fail. It does not decide.

The corner run started from commit 490c32b with this command still uncommitted, so its manifest records a dirty tree. The csv is runs/benchmarks/corners/report.csv.

Why activity outlasts the stimulus (exploratory, 2026-10-12)

Not a scored result and not a retune; scripts/orn_persistence.py reproduces it with flymsg.tracing. The scored corner (w_syn 0.192, th_jump 0, 100 Hz for 300 ms, seed 0) leaves 4,907 neurons firing from 400 to 600 ms. Most are Kenyon cells of the mushroom body: 495 KCg-m, 213 KCab-m, 152 KCab-c and 69 KCab-s (28–32 Hz late), with PAM dopaminergic neurons, the local neuron lLN1_bc (303 Hz), HRN_VP1d and VM5d_adPN among the rest.

Recording one neuron of each leading type shows the drive does not decay after the stimulus: the synaptic drive g of a KCg-m neuron is 13.0 mV during the stimulus and 13.5 mV from 400 to 600 ms, and 67–79% of the excitatory synapses onto the recorded Kenyon cells come from neurons that are themselves still firing. Across all 4,064 Kenyon cells, 73.4% of the excitatory synapses they receive come from other Kenyon cells. In a point neuron every one of those synapses reaches the soma, so the Kenyon cells excite each other into a loop that outlasts the input. The two APL neurons send 196,200 inhibitory synapses onto Kenyon cells, but here APL is a spiking point neuron.

This points at a model limit, not a fitted fix. In the fly, Kenyon cell–Kenyon cell synapses are mostly axo-axonal in the lobes, and APL gives graded, non-spiking feedback inhibition that keeps Kenyon cell activity sparse (Lin et al., Nat. Neurosci. 17, 559–568, 2014, doi:10.1038/nn.3660). Neither has a parameter in this LIF. A model change would need its own pre-registered test on cases it was not chosen on.

Pulse vs sine song across neuropils (lock, 2026-10-14, #36)

The first held-out benchmark of M4 (#36): per central-brain neuropil of male flies, does the connectome predict the sign of the preference for pulse over sine song that Pacheco et al. 2021 imaged (doi:10.1038/s41593-020-00743-y; data doi:10.34770/gv6w-5351, CC BY 4.0)? validation/song.yaml v1, locked before any run and before any response value of the data was read.

• Input, temporal only. Johnston's organ JO-A, JO-B and JO-D on both sides (146 neurons) get the same input in three conditions that differ only in time: pulse (16 ms on, 20 ms off, 10 s, the paper's stimulus), sine (constant) and an aperiodic train (the same pulses at random places, at least two refractory periods apart). Kicks pass one refractory period of dead time and are subsampled per neuron to the same count in every condition, so each JO neuron gets the same number of kicks (its spikes are reported). The mapping "JO-A ≈ pulse, JO-B ≈ sine" is not supported by its sources (Kamikouchi et al. 2009, Matsuo et al. 2014, Ohashi et al. 2023), and the model has no carrier frequency, so the claim is the envelope-driven part of selectivity only; white noise is not scored.
• Model readout. Per neuropil, spike counts over the 12 s window (train and 2 s after, the paper's) weighted by each neuron's synaptic connections there (MaleCNS primary_post, left and right pooled); the driven JO neurons are left out. Index (pulse − sine)/(pulse + sine) per seed; the prediction is the sign of the seed median. Rates 50, 100 and 200 Hz (nominal; the delivered rate is reported); the verdict at 100 Hz.
• Data target. The paper's response magnitude per male ROI (baseline −4 to −0.5 s, 80th or 20th percentile of the baseline-subtracted signal from onset to 2 s after offset), the same index, the median per fly, and a sign per neuropil when at least four flies and two thirds of them agree. AMMC, SAD, WED and GNG are excluded (receptor-near), as are the five tracts without a MaleCNS region.
• Bar. Balanced accuracy (a constant prediction scores 0.5) against a permutation of the model's signs and against 99 degree-preserving shuffled connectomes, p ≤ 0.05 each; pass also needs the periodic train to beat the aperiodic one, otherwise pass_burst_only. The real model, every shuffle and the aperiodic train are scored on the same neuropils by the same rule.
• No-go, model only. If fewer than half of the scorable neuropils resolve a sign (every seed agreeing), or the resolved ones all prefer the same song, the run stops before the data file is opened and is recorded as uninformative.
• Secondary, reported. JO-A and JO-B weighted by their measured response ratio (Yamakoshi et al. 2025, doi:10.1016/j.isci.2025.113232, data 10.17632/29hpr3nrrv): females, a pan-JO driver with two ROIs, a 167 Hz carrier and a 100 Hz tone for the sine.

flymsg song prints the lock; --run runs the model, its no-go, then the data.

Result (2026-10-15, run at 2c86f71, the lock commit, seeds 360–364): uninformative, the no-go.

• The no-go. The run took 1,045 simulations: three conditions at three rates on the measured connectome and on 99 shuffles. It was resumed once from its checkpoint, with 6 workers. At the decision rate of 100 Hz, only 14 of the 32 scorable neuropils resolve a sign over the seeds (0.44, below the locked 0.5), so the run stopped before the data file was opened. The Pacheco et al. 2021 responses were never read.
• The model barely tells the songs apart. The median |index| over the neuropils is 0.046, two neuropils are silent, and the resolved ones split 7 pulse-preferring and 7 sine-preferring.
• The reason the lock gave. This is consistent with the no-go's stated reason, that LIF neurons with no intrinsic band-pass may smooth the 27.8 Hz pulse envelope away. The run does not test that mechanism.
• Reported, not scored. The JO-A/B response ratios of Yamakoshi et al. are 3.0 (pulse) and 1.7 (sine).
• What a sharper test needs. A model with frequency selectivity in the JO pathway: a measured JO band-pass or a carrier-tuned input. Neither is in the model.

results/song/v1.

Looming to giant fiber timing (lock, 2026-10-15, #34)

The first scored test of the looming transducer (gate G3 of the sensorimotor plan, #34): does the connectome's giant fiber (DNp01) peak at the right time before a looming disc's collision? Dombrovski et al. 2025 (doi:10.1038/s41586-025-09037-4) deposited the giant fiber's whole-cell potential to discs at r/v 10, 20, 40 and 80 ms (Zenodo 10.5281/zenodo.14983850, CC BY 4.0). validation/gf_looming.yaml v1 is locked before the files are downloaded, except one trace and the spreadsheet an agent had opened during the search, and the structure of the Fig. 1 file (recorded in the lock).

• Stimulus. A dark disc from 10° to 63°, then held, at the four r/v: the protocol of Jang et al. 2023 (doi:10.1242/jeb.244790), which the 2025 recordings cite. Its use for them is inferred, since they do not restate the sizes.
• Drive, engineered. LC4 (126 neurons) fires in proportion to the disc's angular velocity, LPLC2 (185) in proportion to its angular size, both sides, each capped at one spike per refractory period. These forms follow Ache et al. 2019's description; the gains are scanned on a 4 × 4 grid (15 cells with input), because no LC4 or LPLC2 rate to looming is measured.
• Model readout. Each giant fiber's potential at every step, filtered as the data; its peak time is the first sample at 95% of its maximum in the window from onset to 150 ms after the expansion stops, and the run's is the mean of the two. Only cells where both giant fibers respond and stay below threshold are scored, since the recordings are graded. Per cell the median over five seeds and a fit of the peak's time before collision as α · r/v − δ; α fixes the angular size the response follows (θ = 2 atan(1/α)).
• Data target. Per control fly (10, two trials each), the trial mean, the baseline of the 1 s before onset and a Savitzky–Golay filter (Jang et al. 2023's), the same peak rule, window and fit; a bootstrap interval of the median α over flies.
• Bar. pass when every scored cell's α lies inside the flies' interval, partial when some do, fail when none does. A trace that only follows the stimulus (its size or angular velocity, raw or low-passed by the synaptic and membrane time constants) gives α between about 1.73 and 2.01 under the same readout, and a peak at onset 11.43, whatever the circuit; if the flies' interval overlaps that band or holds the onset slope, the run is uninformative. Each scored cell's distance from the band is reported. δ, the implied angle and the peaks near the expansion's end are reported. Amplitudes are never compared.
• No-go, model only. If fewer than 4 of the 15 cells respond (one synapse's jump in both giant fibers, every seed and r/v) and stay graded, the run stops before any file is opened.
• Seen before the lock. During the search an agent opened one control file and the deposited spreadsheet and saw the structure of the Fig. 1 file; the lock records what was seen. The scored slope was not computed.

flymsg gf-looming prints the lock; --run runs the model, its no-go, then the data.

Result (2026-10-15, run at 0579ad8, seeds 340–344): uninformative, the no-go. Every one of the 15 cells moves both giant fibers by more than one synapse's jump, and in every one of the 300 runs at least one giant fiber fires inside the window, so no cell stays graded and none is scored; the recordings were not opened. The model's giant fiber is reached at the lowest gains already: LC4 alone at 2 Hz per rad/s (a peak rate of about 110 Hz at r/v 10 ms) or LPLC2 alone at 50 Hz per rad. Its peaks come early, a fifth to a quarter of the way through the expansion (median 18 ms after onset at r/v 10, 186 ms at r/v 80), so the unscored slopes (α 5.2 to 10.5, angles 11° to 22°) sit near the onset slope 11.43 rather than the follower band. The recordings are scored as graded potentials (Jang et al. 2023 and Dombrovski et al. 2025 report peak depolarizations); in this model one looming disc through LC4 or LPLC2 is enough to fire the giant fiber. A timing test of the transducer needs either gains low enough to keep the model's giant fiber graded, which the grid does not reach, or a readout of the drive into it; either is a new version. results/gf_looming/v1.

Version 2 (locked 2026-10-15). A model-only probe at seed 9000 found the giant fiber firing even at gains 4 to 32 times below the grid (except LC4 alone at r/v 80 ms), so lower gains do not reach the graded regime. Version 2 makes the two giant fibers passive: a threshold of 10 V (sim.Membrane, every other value the default) that no potential of the run reaches, so their potential integrates the drive as a recorded potential below threshold; a spike refuses the run. Everything else is version 1's (grid, filter, peak rule, follower band, data rule), with seeds 350–354. The compensation is named in the lock: the giant fibers' own spikes no longer reach their targets, which the readout does not use. A passive integrator follows its drive, so the stimulus-follower band is the direct path's own prediction: only a cell's distance from it is the circuit's. The rate cap and dead time compress the drive late in the expansion and can shift α; this is a named limit.

Result (2026-10-15, run at 21ce984, seeds 350–354): uninformative.

• Model. All 15 cells respond and stay graded, so all are scored. α is 1.39 to 2.13 (median 1.85). Ten cells lie inside the stimulus-follower band (1.73 to 2.01), and the other five sit within 0.35 of it. A median of 45% of a cell's peaks come within 20 ms of the expansion's end. The passive giant fiber thus peaks as its drive does, and the timing carries almost nothing beyond the direct path.
• Data, opened only after the model and its no-go. Across the ten control flies the median α is 3.32. Three flies have α ≈ 12.6, at the onset slope (11.43), so the 95% interval of the median α runs from 2.95 to 12.58 and holds that slope.
• Verdict. By the lock's rule this is uninformative: the flies' timing cannot separate a circuit from a peak locked to the onset. Outside the verdict, the model's α also lies below the flies' interval in every cell.

results/gf_looming/v2.

What a version 3 needs (entry protocol 3, 2026-10-16; not a lock). The onset slope above is computed analytically (gf_looming.stimulus_locked); the drive offers only the angular velocity and size forms (gf_looming.rates), so no arm feeds the model an input that follows only the onset. A version 3 needs that arm as an input next to the transducer, a measured LC4 or LPLC2 spike rate with trial variability on MaleCNS bodyIds in place of the engineered forms, and a holdout partition written before any data. The ten control flies were opened by versions 1 and 2, so they cannot be that holdout.

Inventory

Counts are bodies with that cell type in MaleCNS v1.0 (165,122 traced neurons, CC-BY 4.0). No new recording dataset is downloaded. A recording claim is about spikes or synaptic drive. A behavioral proxy is about what the fly does, and is not turned into a firing rate.

The v4 cases, the silencing tests T1–T4, N1 and the null families are calibration or already-reported results. They are not held-out. LPLC2 is named in the looming case's own evidence sentence. LB3a and LB3d were compared by connectivity when the sugar set was chosen, and MN9 is the calibration readout. The alternate LC4 routes were enumerated by path. GF → DLMn, vPR6, vMS11 and pCd were already inspected. MDN has no single recorded motor target in the manifest. T4/T5 needed a moving bar, and on 2026-10-11 this simulator's stimulus was a Poisson rate on a whole type (column stimuli came later, eye). DN1pA acts over hours, outside a 600 ms run.

In the lock, every row this table marks exploratory, excluded or not scored sits in partition.exploratory. DN1pA is the entry clock neurons. DNp02 and DNp11 are the entries LC4 via DNp02 and LC4 via DNp11. Those role words say why a row is outside the scored held-out case. They do not add a second partition.

No row in this inventory is promoted to a held-out behavior. The promotion would choose the test after the olfactory failure and the silence failure. A later case still needs a new version and a source that did not select the model. This note does not add that source.

The M4 v2 proposal is the prospective path for a new source. Its Baker identity audit extends anatomical coverage without opening responses or creating a new partition. Remaining scientific eligibility and publication conditions are recorded there. Neither document changes this frozen v1 manifest, promotes an exploratory row, or supplies a scored result.

Confirmatory test

One case: stimulate ORN_DM1 and score DM1_lPN. Couto, Alenius and Dickson (2005) place Or42b ORNs in DM1. Kazama and Wilson (2008) show that ORNs of a glomerulus drive that glomerulus's PN through an identified excitatory synapse. The claim is that recruitment, not an odor preference and not a published firing rate. Kazama and Wilson used odor puffs. This stimulus is 100 Hz Poisson for 300 ms, so a rate in hertz is reported and is not matched to those papers.

The case passes only when all three hold, at seeds 0–2, control seed 0, w_syn 0.192, th_jump 0. The numbers are criterion v4:

1. positive. The best DM1_lPN fires at least 3 spikes in the 300 ms window in at least 2 of 3 seeds.
2. specific. Its rate is at least 3 times the class-matched control plus 2 Hz.
3. stability. At most 100 neurons still fire from 400 ms to 600 ms.

Order and dose are not endpoints. There is one confirmatory case, so the three checks are simply all required. Adding a case later is a new version.

ORN_V → V_ilPN and V_l2PN is the same kind of claim for the CO2 receptor population (Suh et al. 2004; Jones et al. 2007). Both targets are reported. The case does not decide, and it does not score CO2 avoidance.

Sensitivity

Parameter uncertainty and stimulus-seed variation are separate. The seeds above are the stimulus variation. The parameter corners are three published points, not a search:

The scored claim uses only the default. Another corner that flips a pass into a failure, or a failure into a pass, is reported. No corner replaces the default.

Edge confidence is not varied: the build uses one table, synapses at confidence ≥ 0.5. The sign shuffle is the null-family protocol, not this one. Transmitter labels stay the release's consensus_nt.

What a result may say

A pass says that, under this stimulus and these three checks, DM1_lPN is recruited by ORN_DM1 beyond a class-matched control, and the network does not run away. It does not say that the model smells, that the rate matches an odor recording, or that w_syn should move. A failure is a finding. The scored run is that failure: positive passes, and specific and stability do not. Both sensitivity corners fail the same way, and the scored failure survives them. The record is above. w_syn did not move. The v4 battery remains the baseline comparison.

References

• Ache, J. M. et al. Current Biology 29, 1073–1081 (2019). doi:10.1016/j.cub.2019.01.079
• Allen, M. J. & Murphey, R. K. European Journal of Neuroscience 26, 439–445 (2007). doi:10.1111/j.1460-9568.2007.05686.x
• Bidaye, S. S. et al. Neuronal control of Drosophila walking direction. Science 344, 97–101 (2014). doi:10.1126/science.1249964
• Blagburn, J. M. et al. Journal of Comparative Neurology 404, 449–458 (1999). PMID 9987990
• Couto, A., Alenius, M. & Dickson, B. J. Molecular, anatomical, and functional organization of the Drosophila olfactory system. Current Biology 15, 1535–1547 (2005). doi:10.1016/j.cub.2005.07.034
• Jones, W. D. et al. Two chemosensory receptors together mediate carbon dioxide detection in Drosophila. Nature 445, 86–90 (2007). doi:10.1038/nature05466
• Kazama, H. & Wilson, R. I. Homeostatic matching and nonlinear amplification at identified central synapses. Neuron 58, 401–413 (2008). doi:10.1016/j.neuron.2008.02.030
• Maisak, M. S. et al. A directional tuning map of Drosophila elementary motion detectors. Nature 500, 212–216 (2013). doi:10.1038/nature12320
• Suh, G. S. B. et al. A single population of olfactory sensory neurons mediates an innate avoidance behaviour in Drosophila. Nature 431, 854–859 (2004). doi:10.1038/nature02980
• Tastekin, I. et al. bioRxiv (2025). doi:10.1101/2025.08.25.671814
• von Reyn, C. R. et al. Nature Neuroscience 17, 962–970 (2014). doi:10.1038/nn.3741