M4 version-2 held-out benchmark proposal

Status: proposal, 2026-10-17. Not a lock; nothing here is scored.

Scope: M4 (#36), S3. This proposes eligibility for a later manifest, not a new experimental result.

Why a v2 is needed

The v1 manifest has no unused scored case left. Its only scored held-out case, ORN_DM1 → DM1_lPN, ran and failed specificity and stability. ORN_V is reported only and does not decide. No exploratory row in the benchmark inventory is promoted. A new source must not have selected the model. The completion plan requires a pass against untouched animal observations; a no-go does not close M4.

Eligibility rules

All must hold, with the rules written before any candidate response data are opened:

• Independent animal observations, unused to build or tune flymsg.
• Cells identified in MaleCNS through a cited cross-match. Name similarity, resemblance and an authored mapping do not establish identity.
• Sex compatible with the male connectome, or an explicit non-dimorphic annotation for the matched type. An empty annotation is not evidence of non-dimorphism.
• An admissible measured model input under the entry protocols. An auditory case requires protocol 6's measured band-pass or carrier-tuned input on exactly MaleCNS JO-A, JO-B and JO-D. That input does not yet exist.
• Prediction, observable, metric, null, simpler-model baseline and pass/fail rule fixed and hashed before opening responses. S0 must qualify the observable; required S1/S2 interfaces must be evidenced. Freeze trial handling, thresholds, seeds and no-go/indeterminate rules too.
• A licence permitting the intended analysis.

Candidates assessed

These are feasibility findings from the supplied entry-protocol search and type match analysis, without opening response files. No candidate is eligible now.

Baker's route is Table S3 root IDs → flywire_annotations v3.2.0, materialization 783 → exact MaleCNS flywireType tokens. Annotation sources: Schlegel et al. 2024 and Berg et al. 2026. The CAVE coordinate/lineage audit extends complete named-type coverage from 2 to 12 of 28 classes: 11 retain unresolved sex compatibility and AVLP_pr36 remains excluded for dimorphism. Fourteen classes are partial and two (AVLP_pr18 and AVLP_pr24) are absent from Table S3. All 404 coordinate anchors agree with both source root-ID lineages at materialization 783; 332 reconstructions have named MaleCNS routes. Complete coverage may be one-to-many, not a single modern type. The older name/synonym search does not override the cited route. No physiological response data were opened and no class is benchmark eligible.

Baker licence basis and publication scope

For the proposed noncommercial scientific benchmark, CC BY-NC-ND 4.0 permits local analysis, including private transformations. ND is not an automatic bar to computing benchmark metrics. This is a scoped licence basis, not permission for commercial use or unrestricted publication of derived data.

Metadata checked on 2026-10-17, without downloading or opening response files:

• Dataset: Calcium responses to acoustic stimuli in Drosophila melanogaster, McKellar et al. (2022), including Christa Baker; DOI 10.34770/7zem-x425, version 1.
• The DataCite registration metadata identifies cc-by-nc-nd-4.0 and links its legal code. The Princeton landing page returned a browser challenge during this check; file-level notices have not been inspected. Recheck applicable notices at acquisition after the scientific lock; conflicting or additional terms require resolution before analysis.
• Licence section 2(a)(1) permits reproduction and production of adapted material for noncommercial purposes, but not sharing adapted material. Section 4 also covers applicable database rights. The CC data-mining FAQ explicitly confirms private extraction, reuse and adaptation under all six version-4.0 licences, subject to NC and output restrictions.

The intended workflow is:

• Keep source responses and transformed response arrays in a local archive, outside public repositories, CI artifacts and release bundles.
• Publish independently authored code, the prospective protocol, source DOI, version, provenance hashes and instructions for obtaining the source directly. Source data are not relicensed under the software licence.
• Publish aggregate scores and conclusions only after checking that the actual output does not reproduce protected source material, constitute an adaptation, or reuse database contents in a way requiring additional permission. Aggregation alone does not establish that an output is publishable. Experimental curves, normalized traces and response tables require their own review before sharing.
• Credit the dataset creators and Baker et al.'s paper, link the deposit and CC BY-NC-ND 4.0, and describe the analysis. Do not imply author endorsement.
• Reassess the licence basis if the purpose or published outputs change. NonCommercial concerns the purpose of the use, not simply whether data are redistributed or the code is open source. A use primarily directed towards commercial advantage requires separate permission or another documented legal basis, even locally.

Within this scope, no separate permission is needed merely because the local analysis transforms data. Output review remains a publication gate. This does not make Baker scientifically eligible or authorize opening held-out responses before the prospective lock.

Blockers and owner decisions

• Obtain protocol 6's JO input under the measured-input design and accepted ADR. The audited calcium/population sources do not yet supply an eligible absolute JO-A/B/D input. No auditory case can be scored until source, identity/sex and numerical gates pass.
• Resolve the remaining missing FlyWire labels and MaleCNS identities listed in the completed CAVE audit. Token access and the root-ID update are verified; partial classes remain partial.
• Apply the documented noncommercial licence scope above; verify source notices at acquisition and review actual outputs before publication. Obtain separate permission or document another legal basis for uses outside that scope.
• Resolve female recordings versus the male connectome with cited sex evidence. Acceptance alone cannot replace a non-dimorphic annotation.

Next, if unblocked

Write the calibration/exploratory/held-out partition first. Lock validation/benchmarks.yaml version 2 with a new content hash and the complete prospective rules before opening responses. Preserve v1's failure and reported roles. Then open the eligible data once under that lock; archive provenance, controls and all outcomes. Baker, Deutsch and Pacheco response data remain unopened here.