STOA on GitHub results notes

STOA recovery measurement

Headline

When the plan names a part it is right 94% of the time (75 of 80), a figure that may be inflated because 41.2% of Tier A authored part names share a word with their material's name, which the plan authors could see, and it names 35% of judged parts (80 of 229). The cross-tab of named or described, by leak or no leak, by verdict has not been computed; until it is, read the 94% with that confound attached.

The other 149 judged pairs the plan described (shape, colour, position) instead of naming; 28 of those were still ruled same-or-grain, and 121 were ruled wrong by judges whose rubric had no category for declining to name. They are recorded as abstained: the pipeline was told to describe rather than guess when the evidence does not support a name, so scoring them as errors would misreport the method. That the rubric had no such category is itself a finding.

Structure: over 49 models and 1,755 parts the macro mean best IoU is 0.728 and the macro recovery rate (parts with IoU ≥ 0.5) is 75.8%. Macro is the headline: the pooled figures (0.931, 94.8%) are dominated by the two largest scenes (IridescenceDielectricSpheres and IridescenceMetallicSpheres), which are 39.4% of the parts and average 1.000. Macro mean best IoU by tier: A 0.646, B 0.673, C 0.971.

Naming, all Tier A parts: 49.9% macro (39.3% pooled) were matched and ruled same-or-grain. That counts abstentions against the method; read it with the precision and coverage above.

What the material boundary already gives you

1,281 of 1,755 parts (73.0%) are exactly one material's mesh, recovered by the importer before SEG runs (Tier A: 179 of 262, 68.3%). On the 474 parts it does not hand over (22 models) the macro mean best IoU is 0.736 and the recovery rate 77.8%; pooled 0.889, 92.6%. That macro IoU is essentially identical to the 0.728 overall, so the handover was not inflating the score. In Tier A the residual is 83 parts: 3 same, 47 grain, 2 wrong, 19 abstained, 12 below the threshold.

Denominators

• Structure: 49 models, 1,755 parts (a part is a mesh instance).
• Naming: 21 Tier A models, 262 instances, 282 distinct names. 229 judged pairs (IoU ≥ 0.5), 33 below threshold, 0 unmatched, 0 left unnamed by SEG.
• Index-bearing: 149 of 262 instances (56.9%) carry an instance index; authored names were compared with one trailing index group removed.

The failure set

This is an output, not a defect list. Nothing was fixed during the run. Counts are computed; the descriptions are curated from the plan authors' logs in plans/materials.

• Granularity mismatch. 21 models recovered at least three parts for every original (part-count ratio): AntiqueCamera 28.0, MaterialsVariantsShoe 25.0, MetalRoughSpheres 21.2, DragonAttenuation 21.0, IridescentDishWithOlives 13.0, SheenChair 12.8, FlightHelmet 12.7, EnvironmentTest 11.0, GlamVelvetSofa 10.3, CesiumMilkTruck 9.0, TransmissionRoughnessTest 7.7, VertexColorTest 6.5, Lantern 5.3, StainedGlassLamp 5.0, IridescenceLamp 4.7, SheenDamask 4.6, ToyCar 4.3, TextureSettingsTest 3.6, IridescenceSuzanne 3.0, MosquitoInAmber 3.0, SheenHighHeel 3.0. Where an author grouped parts and the plan cut finer, IoU falls although nothing is geometrically wrong.
• Evidence-limited naming. The plan only described 149 of 229 judged parts. Many models have no colour evidence, no stated up or front axis (so left/right and front/back are assumed), and at is relative to each mesh, not the scene.
• Instances SEG cannot separate. Near-identical islands got ordinal or positional names, or were merged.
• Plan-syntax friction. Every island id must be listed (no range or "rest"): the largest plan lists 5595 numbers in one mesh (DragonAttenuation).
• Truncated SEG.describe on large meshes: authors planned by component, or condensed the text with a script, because island lists were cut off.
• basis list. It accepts only position, size, colour and shape; two authors' first checks were refused for listing "material".
• Editor log. Invalid string length appeared in the console while cutting ABeautifulGame; the cut completed and scored (observed in the run, not recorded in the JSONL).
• SEG refusals. 0 across 49 models.

Protocol and deviations

• Blindness. Plan authors saw SEG's report and material names, never part names or the answer key. The barrier was an instruction, not a permission, and authors sometimes recognised the object from shape.
• Slips. One author listed the plans folder (file names only); one wrote and deleted a scratch file in the session folder. Neither read another plan's content.
• Harness bug. An empty README licence section gave an empty attribution, which looked like a refusal on DirectionalLight; fixed before scoring and noted in that plan's log.
• Lantern, --mode single. Lantern has one material, so its single-mode mesh is byte-identical to its materials-mode one. It is reported separately but is not an independent experiment: mean best IoU 0.539, recovery 33.3%.
• SEG version. 1.6.0.

Who judged

All plan authoring, naming labels and verdicts on this page are model output. The 49 plans were written by model agents with no human in the loop, and the verdicts by three model judges: two independent, and a third on the 44 pairs they split. The ground truth is human (the original authors' part names); the verdicts are not.

Judge agreement on the 229 judged pairs: 185 identical verdicts, Cohen's kappa 0.68; same-or-grain versus wrong agreed on 228. The split between "same" and "grain" is a judgement call: judge A ruled 81 pairs same, judge B 40, and the tie-break ruled all 44 disputed pairs grain (39 same in the consensus). Read the same-or-grain figures, not "same".

Structure, 49 models

Filter by tier: allABC Click a heading to sort. Tier A: real objects with author-meaningful names; B: feature tests; C: generic names.

Naming, Tier A, 21 models

Counts of judged parts by consensus verdict. "Same or grain" is the figure to use. The last column is an upper bound on how much the material name gives away.