Report 2026-09-08 14:12 · round 2 · 14 runs

vyomkesh · web · September 23, 2026

autoresearch · phase parked · written 2026-09-08T14:12:13.777Z

The question

Classify handwritten digits. Given a 28x28 grayscale image of one handwritten digit, predict which of the ten digits 0-9 it is. The question is which model and training setup reads these digits most accurately, and how confident we can be in a difference given only 24 held-out images. OBJECTIVE (proposed by the Phase-0 characterisation, finding find-char-ds_fb7985d4ebd7f9f9): maximise image_class_accuracy on the sealed scored set sc_mnist_digits_b2f34dc3 — the fraction of the 24 images whose predicted dominant foreground class is the labelled digit. That is top-1 accuracy, in the user's own words. It did not exist when the brief was written; I built it into unet_small@1.10.0 and segformer_tiny@1.10.0 rather than optimise IoU because IoU was there. The truth through the recipes' own decoder scores it 1.0000 on all 24, so the instrument is valid. IoU is a SECONDARY with no guardrail. The polygon is a carrier for the class, and the tracing itself is a loose, inconsistent silhouette (interior holes filled on some digits, open on others), so a guardrail on overlap would reject a model for failing at something the user did not ask for. It is kept because it separates 'named the digit without finding it' from 'found it and named it'. minPracticalEffect = 0.0833 is TWO of the 24 images, and it is the instrument's resolution floor, not a scientifically meaningful threshold. Honest limits, measured, not assumed: chance 0.100; the majority-class constant predictor 0.167; 1-nearest-neighbour on RAW PIXELS fit on the 276 non-scored images 0.875 (21/24); perfect 1.000. The entire interesting range is three images wide, and a paired sign test over 24 items needs roughly five discordant images before a difference is more than noise. This study can establish that a model reads these digits; it cannot honestly rank two competent architectures against each other. Enlarging the scored set is the one act that changes it, and it is on the asks.

Objective: image_class_accuracy (maximize), smallest effect worth acting on 0.083.

The answer so far

THE QUESTION. Given a 28x28 grey image of one handwritten digit, name which of 0-9 it is; find which model and training setup reads these digits most accurately, and say how confident we can be given 24 held-out images.

THE ANSWER, NOW CROSS-VALIDATED BY A SECOND FAMILY. unet_small@1.10.0 at epochs=400 (12000 steps, ~147 GPU-s), image_size 64, ce_dice, augment=flip, class_weighting=none, selecting the checkpoint on val_image_class_accuracy: image_class_accuracy 0.9167 = 22 of 24, at seeds 1, 2 and 3 IDENTICALLY, per-item vectors byte-identical, seed sigma 0.000 (x1-f37d7c84a6ef, x1-ca6f1f37d19f, x1-7180c6e0777c). It clears chance (0.100), the majority-class constant (0.167) and the 1-NN raw-pixel reference (0.875).

HOW CONFIDENT, IN THE UNITS THE USER ASKED FOR. 22/24 has a 95% Wilson interval of [0.742, 0.977]. One image is 4.17 accuracy points. A paired test on 24 items needs about five discordant images before a difference is anything but noise, and the fold's own gradability puts the smallest effect the remaining budget can resolve at 0.20. So: this number is a solid 'it works and it is not broken'; it is NOT a ranking, and it must not be quoted to three decimals as though it were.

ROUND 2 SETTLED THE LAST QUESTION THIS CUT COULD ANSWER. The pivot was deliberately NOT an architecture ranking -- it was a per-item categorical question. segformer_tiny@1.10.0 (attention, not convolution) at the same converged schedule, three seeds: 0.8750 / 0.7917 / 0.8750 (x2-869f94870301, x2-732c1739b76a, x2-92f2227965a6). h2-ceiling-is-the-data-not-the-model's falsifier was 'it reads BOTH item 3 and item 24 correctly'. It read NEITHER, in all three seeds. Across 9 converged runs spanning two structurally unrelated backbones, three seeds each, two augment values and two class_weighting values: item 24 (mnist_9_026.png) is wrong in 9 of 9, item 3 (mnist_1_005.png) in 8 of 9, and only four other item-slots ever move, each in a single run (find-cross-family-ceiling-nine-runs). I looked at worst.png: strokes traced cleanly, nothing bridged or cropped, the class name wrong under a correct outline -- per-item IoU exactly 0.000 on both.

THE MECHANISM, WHICH WAS NOT IN THE PLAN AND IS THE ROUND'S BEST RESULT. Characterisation measured 1-NN on raw pixels; its three errors are mnist_0_026, mnist_1_005 and mnist_5_005. So 1-NN reads mnist_9_026 CORRECTLY while all nine neural runs miss it. That splits the two-image ceiling in half:

item 3, mnist_1_005: a 1 whose top flag and full horizontal foot bar make its silhouette a 2. Missed by BOTH neural families AND by nearest neighbour. Genuine ambiguity; no model fixes it.
item 24, mnist_9_026: a 9 whose closed bowl and stub tail make its FILLED silhouette a 0. Read correctly by raw pixels, missed by every segmentation run. This one is the OBJECTIVE's loss, not the image's: the label policy traces the outer silhouette and fills the interior holes of 0/6/8/9, so the per-pixel target discards exactly the evidence that separates a 9 from a 0. A different backbone cannot recover it; a different TARGET might.

SO THE STUDY PARKS, AND ON FAILURE TO LEARN RATHER THAN FAILURE TO PAY. 9 of 26 runs and 3 of 5 rounds remain; GPU is nowhere near the wall (~40 GPU-minutes of a 7-hour grant). The binding constraint is images. The whole remaining range is 22/24 -> 24/24 = two images; one is missed by every method known here, the other needs a different target rather than a different config. minPracticalEffect is 0.0833 = two images and gradability says 9 runs resolve only 0.20. Every further configuration arm on this cut must come back within_noise -- 11 of 14 graded verdicts already are. Draining the last 9 runs into that is worse than stopping, because it looks like progress. The runs are worth more held for the larger class-stratified scored set that stands on the asks.

WHAT ROUNDS 0-1 ESTABLISHED, CARRIED FORWARD. The schedule was the whole story: the same config went 0.5417 -> 0.9167 from epochs=60 to 400, paired meanDiff +0.375, se 0.1175, t 3.19 -- the study's only better verdict (x1-f37d7c84a6ef vs x0-4775d82bc6b7). The seed sigma of 0.168 that once walled the study at a detectable effect of 0.211 and 65 seeds per arm was an artefact of stopping three still-climbing curves at 60 epochs; at 400 the same three seeds return the SAME NUMBER. Two hypotheses died on their own falsifiers: augment=none scores 0.9167 at seed 1 and 0.8750 at seed 2, both within one image of flip (h0-augment-flip-hurts-digits refuted); class_weighting=inverse_freq returns effect exactly 0.000 with iou FALLING 0.6376 -> 0.5871 (h1-background-dominates-the-loss refuted -- the 97%-background arithmetic was correct and predictively worthless).

WHAT I DECLINE TO BUILD, NOW FOR A MEASURED REASON. A dedicated classifier head. Its trigger under h0-seg-can-classify was a plateau below 0.667 with well-traced silhouettes; the curves plateaued at 0.9167, so the trigger was genuinely testable and did not fire. At best it recovers the one image the objective loses: 0.0417, half of minPracticalEffect and below the one-image resolution floor. If the scored set grows, this becomes the first thing worth building, because the mechanism is now identified rather than guessed.

WHAT ROUND 2 COST ME BEFORE IT PAID. Its first three arms (x2-436cfe7c1085, x2-fd749980898a, x2-666a0b4741ba) died 12 seconds after launch, evidence-free: epochs=600 against a declared range of [1, 400]. Neither declare_experiment nor enqueue refuses an out-of-range numeric value -- the trainer does, after the run is claimed -- so each violation costs a whole run slot for zero GPU-seconds. Three of 26. The min/max live ONLY in /home/jha0006/research/recipes//recipe.json; read_recipes returns type and default. A schedule longer than 400 epochs is legal without a version bump via scalingContract 'steps' with max_steps (max 2000000), at 30 steps/epoch on this cut.

THE SPLIT DEFECT, priced not argued. The human decided to proceed as cut. val holds zero 5s while the scored set holds four; the item that this most plausibly destabilises, item 11 = mnist_5_005.png (a 5 with a detached top bar), moves in exactly 2 of 9 runs. One unstable image out of 24, below the resolution floor and below minPracticalEffect. It weakens selection; it biases no scored number. A larger cut must be class-stratified.

WHAT ENDS THE PARK. A larger, class-stratified labelled set -- a few hundred scored images instead of 24. MNIST is free and the mechanism to chase is already named. With 9 runs still granted, the study can resume the moment the data lands.

Claims

supported — The user's question is top-1 digit classification, and image_class_accuracy on sc_mnist_digits_b2f34dc3 measures it; IoU is a proxy that cannot see a perfectly traced digit under the wrong class label. Round 1 demonstrates the gap concretely: the two failing items score IoU 0.000 with visually clean silhouettes. (find-char-dsfb7985d4ebd7f9f9, find-two-image-ceiling-same-items-every-config)_
supported — The instrument is valid on REAL runs, not just at declaration: all eleven runs report oracle_image_class_accuracy 1.0000 and baseline_empty_image_class_accuracy 0.0000 on the scored 24, with per-item vectors the fold used for paired tests and this tick used for the ceiling analysis. (find-recipe-image-class-accuracy-1100, x1-f37d7c84a6ef, x0-4775d82bc6b7, x0-269898940c77, x0-337bc53ade3a, x0-3d975a041094, x0-93418a718b04)
supported — The 24-image scored set cannot resolve differences between competent models, and round 2 turns that from an argument into a measurement: segformer_tiny vs unet_small at a matched converged schedule is paired meanDiff -0.0833, se 0.0576, t -1.45 -- within noise. The best config sits at 22/24 with 1-NN on raw pixels at 21/24, so the whole remaining range is two images. One image is 0.0417; a paired test needs about five discordant images. 95% Wilson on 22/24 is [0.742, 0.977]. Any future arm scoring 23/24 or 24/24 cannot be separated from a draw. (find-char-dsfb7985d4ebd7f9f9, find-cross-family-ceiling-nine-runs, x1-f37d7c84a6ef, x2-869f94870301, x2-732c1739b76a)_
supported — The scored set is not contaminated by near-duplicates: groupKey "item" is correct, and no test item's nearest neighbour anywhere exceeds cosine 0.884, all same-class different-writer. (find-char-dsfb7985d4ebd7f9f9)_
supported — The split is not class-stratified: val holds zero 5s while the scored set holds four. The human decided this tick to proceed as cut, and round 1 prices the defect exactly: the ONLY item that moves between configs is item 11 = mnist_5_005.png, a 5 with a detached top bar -- one unstable image out of 24, below the resolution floor and below minPracticalEffect, changing no verdict. It weakens selection; it biases no scored number. A future larger cut should be class-stratified. (find-char-dsfb7985d4ebd7f9f9, find-two-image-ceiling-same-items-every-config, ask_8b851c96-mtsporyy?)_
supported — Instance-count metrics have no spread on this data -- every image has exactly one polygon rasterising to exactly one connected component -- so instance_count_mae and count_exact_rate are not candidate objectives here. (find-char-dsfb7985d4ebd7f9f9)_
supported — 11 of 300 images (all in train, none scored) have 14-42% of their ink labelled background because the policy traces only the largest connected component; the class label is correct on all of them. (find-label-detached-strokes)
supported — A segmentation recipe reaches image_class_accuracy >= 0.833 on this scored set. REACHED: unet_small@1.10.0 at epochs=400 scores 0.9167 (22/24) at seeds 1, 2 and 3 identically, clearing the 1-NN raw-pixel reference (0.875), chance (0.100) and the majority constant (0.167). Round 0's 0.5417/0.6667/0.3333 was a 60-epoch artefact and bore on nothing. (h0-seg-can-classify, x1-f37d7c84a6ef, x1-ca6f1f37d19f, x1-7180c6e0777c, find-round1-converged-at-400-epochs)
refuted — The recipe default augment=flip is wrong for digits and costs accuracy against augment=none. REFUTED at the converged schedule: augment=none scores 0.9167 at seed 1 (identical to flip) and 0.8750 at seed 2 (one image BELOW flip). Both paired seeds land within one image, which is exactly the falsifier. The +0.125 measured at 60 epochs and the wide val gap were schedule artefacts. The mechanism -- a mirrored 2 is not a 2 -- is sound but does not bind after 96000 samples. (h0-augment-flip-hurts-digits, x1-e07937d67f6c, x1-1d5c76a779f8, x1-f37d7c84a6ef, x1-ca6f1f37d19f)
refuted — Selecting the checkpoint on val_image_class_accuracy beats selecting on val_iou by at least two images. REFUTED as stated: x0-93418a718b04 returned image_class_accuracy 0.541667 byte-identical to its control, because on a val curve that never turns over both rules keep the final epoch -- one checkpoint scored twice. Round 1 makes this retrospectively clean: once converged, best epochs land at 60-340 of 400 rather than on the last, so the question WOULD now be askable; it is simply not worth buying, because selecting on the objective is free and every arm since does it. (h0-selection-metric-matters, x0-93418a718b04, x0-4775d82bc6b7, find-round0-undertrained-60-epochs, find-round1-converged-at-400-epochs)
supported — A dedicated classifier recipe is still not worth building ON THIS CUT, now for a measured reason. Its trigger under h0-seg-can-classify was a PLATEAU BELOW 0.667 under well-traced silhouettes; the curves plateaued at 0.9167 and the trigger did not fire. At best such a head recovers the ONE image the objective loses (item 24, which 1-NN on raw pixels already reads): 0.0417, half of minPracticalEffect and below the one-image resolution floor. If the scored set grows to a few hundred images this becomes the FIRST thing worth building, because the mechanism is now identified rather than guessed. (find-round1-converged-at-400-epochs, find-cross-family-ceiling-nine-runs, x1-f37d7c84a6ef, x2-869f94870301)
supported — The objective was confirmed from the moment set_brief folded (a human grant was already on the study); an earlier tick's report that it awaited confirmation was a misread of its own state, and no run was ever blocked by it. (find-register-recipes-needs-explicit-manifests)
supported — register_recipes with no arguments registers the route's bundled library, NOT this box's /home/jha0006/research/recipes -- a recipe edited on the box must have its manifest inlined in args.recipes and be read back with read_recipes, because the response's registered list does not say which version landed. (find-register-recipes-needs-explicit-manifests)
supported — Round 0 was undertrained end to end: all four val_iou curves rise monotonically to the epoch-60 cutoff and three of five runs select the LAST epoch. 60 epochs = 1800 steps = 24 GPU-seconds against a 7-GPU-hour grant. No round-0 number describes a converged model, and round 1 confirmed it by moving the same config +0.375. (find-round0-undertrained-60-epochs, x1-f37d7c84a6ef, x0-4775d82bc6b7, x0-269898940c77, x0-337bc53ade3a, x0-3d975a041094, x0-93418a718b04)
supported — The sigma of 0.168 image_class_accuracy that gradability reported -- walling the study at a detectable effect of 0.211 and demanding 65 seeds per arm -- was an artefact of stopping three still-climbing curves early, not the intrinsic seed noise of this recipe. CONFIRMED: at epochs=400 the same three seeds return 0.9167 / 0.9167 / 0.9167, sigma 0.000, with identical per-item vectors. Every round-0 verdict was graded against a phantom. (find-round1-converged-at-400-epochs, x1-f37d7c84a6ef, x1-ca6f1f37d19f, x1-7180c6e0777c)
supported — A segmentation U-Net reads these digits far above the floors, and once converged it does so at 0.9167 (22/24) against chance 0.100 and the majority-class constant 0.167. The undertrained round-0 figures (0.5417/0.6667/0.3333) showed the task was learnable by this family; round 1 shows the ceiling too. (x1-f37d7c84a6ef, x1-ca6f1f37d19f, x1-7180c6e0777c, x0-4775d82bc6b7, x0-269898940c77, x0-337bc53ade3a)
refuted — The per-pixel objective spends about 97% of itself on background, so class_weighting=inverse_freq should raise image_class_accuracy. REFUTED: x1-03dc5648daaa returns 0.9167, effect exactly 0.000 against its matched control, with the same two items failing, while iou FALLS 0.6376 -> 0.5871. The falsifier ('within one image of the control, or below it') fired. The 97%-background arithmetic was correct and predictively worthless. (h1-background-dominates-the-loss, x1-03dc5648daaa, x1-f37d7c84a6ef, find-char-dsfb7985d4ebd7f9f9)_
supported — My prediction ledger is 8 hits / 6 misses. All five round-0 misses came from pricing a 60-epoch default as a converged schedule. Round 1 went 6 for 6 once I set predictions from curve shape. Round 2 went 2 of 3: the one miss is x2-732c1739b76a, segformer seed 2 at 0.7917 against a predicted >= 0.833 -- I carried unet_small's zero seed sigma across a family change, which was exactly the assumption the run was bought to test. (x1-f37d7c84a6ef, x1-ca6f1f37d19f, x1-7180c6e0777c, x2-869f94870301, x2-732c1739b76a, x2-92f2227965a6)
supported — The 400-epoch schedule is CONVERGED, unlike round 0's: val_iou over the last eight validations spans 0.006-0.021 in five of six runs, and the selected epoch is 210/90/180/60/340/110 of 400 -- never the last. Checkpoint selection is finally doing work rather than picking wherever the clock stopped. (find-round1-converged-at-400-epochs, x1-f37d7c84a6ef, x1-ca6f1f37d19f, x1-7180c6e0777c, x1-e07937d67f6c, x1-03dc5648daaa, x1-1d5c76a779f8)
supported — The residual error is two specific images, not a distribution, and this now holds across TWO families: over 9 converged runs (6 unet_small + 3 segformer_tiny, three seeds each, two augment values, two class_weighting values) item 24 (mnist_9_026.png) is misclassified in 9 of 9 and item 3 (mnist_1_005.png) in 8 of 9; only items 2, 11, 16 and 18 ever move, each in a single run. It is a CLASSIFICATION ceiling, not a segmentation one -- both hard items have per-item IoU exactly 0.000 (a correct silhouette under the wrong digit) while the rest score 0.140-0.904, and worst.png shows cleanly traced s (find-cross-family-ceiling-nine-runs, find-two-image-ceiling-same-items-every-config, x1-f37d7c84a6ef, x1-ca6f1f37d19f, x1-7180c6e0777c, x1-e07937d67f6c, x1-03dc5648daaa, x1-1d5c76a779f8, x2-869f94870301, x2-732c1739b76a, x2-92f2227965a6)
supported — The 0.9167 ceiling on sc_mnist_digits_b2f34dc3 belongs to the images and the objective, NOT to the unet_small family. CLOSED on h2's own falsifier: segformer_tiny@1.10.0 (attention, not convolution) at the same converged epochs=400 schedule reads NEITHER item 3 nor item 24 at any of three seeds, exactly as unet_small does not. Nine converged runs, two structurally unrelated backbones, three seeds each: item 24 wrong 9/9, item 3 wrong 8/9. Re-asking this on this cut buys nothing. (h2-ceiling-is-the-data-not-the-model, x2-869f94870301, x2-732c1739b76a, x2-92f2227965a6, find-cross-family-ceiling-nine-runs)
supported — A numeric config value outside a recipe manifest's declared range is NOT refused at declare or enqueue -- it is refused by the trainer on the box after the run is claimed, so it costs a full run slot for zero GPU-seconds. Round 2 lost three of a 26-run grant this way (epochs=600 against epochs {min 1, max 400}). The min/max are visible ONLY in /home/jha0006/research/recipes//recipe.json; read_recipes returns type and default only. A schedule longer than 400 epochs is legal without a version bump via scalingContract 'steps' with max_steps (max 2000000), at 30 steps/epoch on this cut. (find-round2-died-on-epochs-range-contract, x2-436cfe7c1085, x2-fd749980898a, x2-666a0b4741ba)
supported — THE TWO-IMAGE CEILING SPLITS IN HALF, and only one half is about the data. Characterisation's 1-NN on raw pixels errs on mnist_0_026, mnist_1_005 and mnist_5_005 -- it reads mnist_9_026 CORRECTLY, where all nine neural runs miss it. So item 3 (mnist_1_005, a 1 whose top flag and full foot bar make its silhouette a 2) is genuine ambiguity that no method here reads; item 24 (mnist_9_026, a 9 whose closed bowl and stub tail make its FILLED silhouette a 0) is a loss of the per-pixel filled-polygon OBJECTIVE, whose label policy fills the interior holes of 0/6/8/9 and therefore discards the evidence (find-cross-family-ceiling-nine-runs, find-char-dsfb7985d4ebd7f9f9, x2-869f94870301, x2-92f2227965a6, x1-f37d7c84a6ef)_
supported — segformer_tiny is the LESS STABLE family on this cut and cannot be ranked against unet_small here. Three seeds at epochs=400 give 0.8750 / 0.7917 / 0.8750, sigma 0.0481 -- above the one-image floor of 0.0417, so that half of its own falsifier fired -- against unet_small's 0.9167 / 0.9167 / 0.9167 at sigma 0.000. Paired against seed 1 the spread is meanDiff -0.0833, se 0.0576, t -1.45: within noise. Both segformer curves are converged (val_iou spans 0.016 / 0.007 / 0.028 over the last eight validations; best epochs 250 / 220 / 80 of 400), so the schedule is not the explanation. The per-item cla (x2-869f94870301, x2-732c1739b76a, x2-92f2227965a6, find-cross-family-ceiling-nine-runs)
supported — THE STUDY PARKS ON FAILURE TO LEARN, NOT ON BUDGET. 9 of 26 runs and 3 of 5 rounds remain and GPU is at roughly 40 minutes of a 7-hour grant, but no configuration arm on sc_mnist_digits_b2f34dc3 can produce a gradable result: minPracticalEffect is 0.0833 = two images, gradability puts the smallest resolvable effect at 0.20 with the runs left, 11 of 14 graded verdicts are already within_noise, and the entire remaining headroom is the two images that 9 runs across 2 families all miss. The one act that changes this is a larger, class-stratified scored set; the remaining 9 runs are held for it. (find-cross-family-ceiling-nine-runs, find-two-image-ceiling-same-items-every-config, find-char-dsfb7985d4ebd7f9f9)_

An id marked ?* is cited by the claim but is not on the graph or in its archive.*

Open questions

Does a classification head on top of the segmentation trunk recover mnist_9_026 (the image 1-NN on raw pixels reads and every segmentation run misses)? Unanswerable on 24 images -- one image is half of minPracticalEffect -- but the FIRST thing to buy on a larger scored set, because the mechanism (the label policy fills interior holes, erasing what separates a 9 from a 0) is now identified rather t
Does the unet_small vs segformer_tiny gap (meanDiff -0.0833, within noise on 24 images) survive on a few hundred images, or is it a two-image accident?
Is unet_small's exactly-zero seed sigma a property of the recipe or of this tiny cut? segformer_tiny at the same schedule gives 0.0481, so it is not a property of the schedule alone.
Does a class-stratified split (val currently holds zero 5s) change checkpoint selection enough to move item 11 = mnist_5_005.png, the only item that moves for schedule-independent reasons?

Next best experiments

NOTHING on this cut. Every remaining configuration arm must return within_noise: the headroom is two images, minPracticalEffect is two images, and the resolvable effect with 9 runs is 0.20. Hold the 9 runs.
A larger, CLASS-STRATIFIED labelled MNIST cut (a few hundred scored images). MNIST is free; this is the only act that raises the study's resolution, and it makes every question above answerable with the runs already granted.
IF the data grows: a classification head or an auxiliary class loss on the segmentation trunk, aimed at the objective-loss half of the ceiling (item 24). Buildable in plain PyTorch under the recipes skill, no human capability needed.
IF the data grows: re-run the segformer_tiny vs unet_small comparison at three seeds each on the larger scored set, which is where a two-image difference becomes a measurable one.

Can this data answer it?

Target: 5: 0.17 ± 0.38 per item (scored set) · constant-predictor MAE 0.17.
Split: 7 pair(s) of near-identical pictures sit on opposite sides of the split (closest: mnist_digits/mnist/mnist_0_001.png ↔ mnist_digits/mnist/mnist_0_016.png). If those are one object photographed twice, a score over them measures recall, not generalisation — put them in one group, or drop one.
σ = 0.1 image_class_accuracy (9 seed replicates in 3 groups). Detecting 0.083 needs 24 seeds per arm at 80 % power. The 9 runs left seat 4 per arm, which can only resolve 0.2. Either accept 0.2 as the smallest effect worth acting on, reduce the noise (a lower-variance metric, a longer schedule, a bigger scored set), or extend the budget.

Best so far

x1-f37d7c84a6ef — LONG CALIBRATION control — the round-0 control config at epochs=400, seed 1 (one knob: epochs 60 -> 400): image_class_accuracy 0.92 · better · paired Δ 0.38 (n=24, t=3.19) · seed spread ÷1000000 · predicted image_class_accuracy >= 0.708 → held

unet_small@1.10.0 · 11 classes · train 240 / val 36 · epochs: 400 ep, 12000 steps, 96000 samples · cuda, 147 GPU-s · decode components · best ep 210 (val_image_class_accuracy 0.9722) · sc_mnist_digits_b2f34dc3 iou 0.6376 (n=24)

x1-ca6f1f37d19f — LONG CALIBRATION seed 2 — identical config to x1-f37d7c84a6ef: image_class_accuracy 0.92 · within noise · paired Δ 0 (n=24, t=0) · predicted image_class_accuracy >= 0.708 → held

unet_small@1.10.0 · 11 classes · train 240 / val 36 · epochs: 400 ep, 12000 steps, 96000 samples · cuda, 139 GPU-s · decode components · best ep 90 (val_image_class_accuracy 1.0) · sc_mnist_digits_b2f34dc3 iou 0.5878 (n=24)

x1-7180c6e0777c — LONG CALIBRATION seed 3 — identical config to x1-f37d7c84a6ef (the slow seed of round 0): image_class_accuracy 0.92 · within noise · paired Δ 0 (n=24, t=0) · predicted image_class_accuracy >= 0.708 → held

unet_small@1.10.0 · 11 classes · train 240 / val 36 · epochs: 400 ep, 12000 steps, 96000 samples · cuda, 138 GPU-s · decode components · best ep 180 (val_image_class_accuracy 0.9444) · sc_mnist_digits_b2f34dc3 iou 0.6156 (n=24)

x1-e07937d67f6c — augment=none at the long schedule, seed 1 — one knob against x1-f37d7c84a6ef: image_class_accuracy 0.92 · within noise · paired Δ 0 (n=24, t=0) · predicted image_class_accuracy >= 0.75 → held

unet_small@1.10.0 · 11 classes · train 240 / val 36 · epochs: 400 ep, 12000 steps, 96000 samples · cuda, 138 GPU-s · decode components · best ep 60 (val_image_class_accuracy 1.0) · sc_mnist_digits_b2f34dc3 iou 0.7547 (n=24)

x1-03dc5648daaa — EXPLORATION — class_weighting=inverse_freq at the long schedule, seed 1, one knob against x1-f37d7c84a6ef: image_class_accuracy 0.92 · within noise · paired Δ 0 (n=24, t=0) · predicted image_class_accuracy >= 0.708 → held

unet_small@1.10.0 · 11 classes · train 240 / val 36 · epochs: 400 ep, 12000 steps, 96000 samples · cuda, 135 GPU-s · decode components · best ep 340 (val_image_class_accuracy 0.8889) · sc_mnist_digits_b2f34dc3 iou 0.5871 (n=24)

x1-1d5c76a779f8 — augment=none at the long schedule, seed 2 — one knob against x1-ca6f1f37d19f: image_class_accuracy 0.88 · within noise · paired Δ -0.042 (n=24, t=-1) · predicted image_class_accuracy >= 0.75 → held

unet_small@1.10.0 · 11 classes · train 240 / val 36 · epochs: 400 ep, 12000 steps, 96000 samples · cuda, 133 GPU-s · decode components · best ep 110 (val_image_class_accuracy 1.0) · sc_mnist_digits_b2f34dc3 iou 0.6362 (n=24)

x2-869f94870301 — CEILING ATTRIBUTION (re-buy at a legal schedule) — segformer_tiny, epochs=400, seed 1: does an attention family miss the SAME two images?: image_class_accuracy 0.88 · ungraded · ungraded: no control · predicted image_class_accuracy >= 0.833 → held

segformer_tiny@1.10.0 · 11 classes · train 240 / val 36 · epochs: 400 ep, 12000 steps, 96000 samples · cuda, 289 GPU-s · decode components · best ep 250 (val_image_class_accuracy 0.9722) · sc_mnist_digits_b2f34dc3 iou 0.5469 (n=24)

x2-92f2227965a6 — CEILING ATTRIBUTION seed 3 — identical config to x2-869f94870301 (segformer_tiny, epochs=400): image_class_accuracy 0.88 · within noise · paired Δ 0 (n=24, t=0) · predicted image_class_accuracy >= 0.833 → held

segformer_tiny@1.10.0 · 11 classes · train 240 / val 36 · epochs: 400 ep, 12000 steps, 96000 samples · cuda, 293 GPU-s · decode components · best ep 80 (val_image_class_accuracy 0.8889) · sc_mnist_digits_b2f34dc3 iou 0.5494 (n=24)

What was learned

The per-pixel objective spends almost all of itself on background: the snapshot profile gives each digit class 0.87%-2.98% of the pixels, so with class_weighting=none about 97% of every cross-entropy term is background and the ten digit identities — the only thing image_class_accuracy reads — compete over the remaining 3%. class_weighting=inverse_freq therefore raises image_class_accuracy on sc_mnist_digits_b2f34dc3 by at least 0.0833 against a matched control at the same seed and schedule. FALSIFIER: it lands within one image of the matched control, or below it — the dice half of ce_dice was → inconclusive on sc_mnist_digits_b2f34dc3 (Δ 0)

Predictions: 8 held, 6 missed.

What the study asks of you

Label more data: A larger, CLASS-STRATIFIED labelled MNIST cut -- a few hundred scored images instead of 24. This is the only act that changes what the study can conclude, and it is the condition that ends the park. Why now: the converged model scores 22/24 and 1-NN on raw pixels scores 21/24, so the whole remaining range is TWO images; both are missed by every one of nine converged runs spanning two structurally unrelated backbones, three seeds each, two augment values and two class_weighting values (mnist_1_005.png, a 1 whose top flag and foot bar make its silhouette a 2; mnist_9_026.png, a 9 whose closed bo
Decide: The study has answered your question on this cut and can no longer learn from it. Which do you want? ((A) Supply a larger class-stratified scored set (a few hundred images) and I resume with the 9 runs still granted -- fir / (B) Accept 0.20 image_class_accuracy (five images) as the smallest effect worth acting on and I will spend the 9 runs ra / (C) Close the study as done at 22/24. The answer is delivered and cross-validated: unet_small@1.10.0, epochs=400, image_)

Figures

By round

Round 0 · tick-40517497f1b15d6e · 2026-09-08 13:06

PHASE 0 — characterised snapshot ds_fb7985d4ebd7f9f9 and proposed the objective; nothing enqueued, by design. I verified all 600 sha256s, measured all 300 images and polygons with code, and looked at the renders (overlays, the 24 scored digits, the 11 detached-stroke labels). The user's sentence is 'name the digit', so the metric is top-1 accuracy — and no recipe logged it. Rather than optimise IoU because IoU was there, I BUILT image_class_accuracy into the mirrored metrics block of unet_small@1.10.0 and segformer_tiny@1.10.0 (declared metric, curve, selection-metric value, per-item scores for paired comparison, new unit test recipes/shared/testimage_class.py; both self-tests PASS; oracle 1.0000 on 24/24 of the real scored set, empty baseline 0.0000) and registered both. The honest headline: 1-NN on RAW PIXELS already scores 21/24 = 0.875 on the sealed set, one image is 4.17 points, and a paired test over 24 items needs ~5 discordant images to call an effect — so this cut can show that a model reads these digits and can catch a broken one, but cannot rank two competent architectures. Objective proposed, four hypotheses declared, three findings posted. Awaiting the human's confirmation — nothing may run before it.

set aside Make IoU the primary metric because both recipes already log it — The polygon is a carrier for the class and its silhouette is nearly the same for every digit, so IoU cannot see a perfectly traced 1 called a 7 — the only failure the user cares about. Kept as an unga
set aside File a recipe request for an accuracy metric and park until a human builds it — Under the recipes skill a metric the study needs is mine to build. It is one general addition to the mirrored block, it self-tests, and requesting it would have cost the study a whole round of waiting
set aside Build a dedicated classifier recipe now — The segmentation recipes can express the task and now measure it exactly. A third family buys no resolving power against a 24-image set that 1-NN answers at 0.875. Held as the diagnosed fix under h0-s
set aside Re-declare the source with a different groupKey and re-cut — groupKey "item" is correct. The snapshot's 7 flagged cross-split near-duplicates are all same-class different-writer train<->val, and no test item's nearest neighbour anywhere exceeds cosine 0.884. Th
set aside Make instancecount_mae or count_exact_rate the objective_ — No spread: every one of the 300 images has exactly one polygon rasterising to exactly one connected component. Optimising a constant would test memorising, not counting.
set aside Put a guardrail on IoU — It would reject a model for failing at something the user did not ask for, on a tracing that is itself inconsistent class to class (interior holes filled on 8/9, open on some 0s, mean precision 0.822)
set aside Block the study on re-cutting a class-stratified split — val holding zero 5s weakens checkpoint selection but biases no scored number. Cheaper to proceed and select on val_image_class_accuracy; the choice is on the asks as a decision for the human.
walls: Budget granted: 5 rounds, 26 runs, 7 GPU-hours, $80 tokens; $4.8 spent this tick, 0 runs, 0 GPU-hours. The binding wall is not money — it is the 24-image scored set, whose resolution floor is two images (0.0833) and whose useful range above the 1-NN reference is three.

Round 0 · tick-05698431a5194bc7 · 2026-09-08 13:14

Integrated the four unconsumed findings and bought round 0 — the study's first runs. The decisive one was find-register-recipes-needs-explicit-manifests: the objective has been CONFIRMED all along (set_brief folded to objectiveConfirmed:true under the existing human grant), so the previous tick's park on ask_3c8e581b was a misread of its own state and nothing was ever blocked. Verified against overview before planning: objectiveConfirmed true, grant 5 rounds / 26 runs / 7 GPU-h / $80, 0 experiments, gradability no_sigma. NOTHING WAS WITHDRAWN because nothing was queued — counts showed 0 queued, 0 running, 0 done, so the new evidence had no standing arm to refute. Enqueued 5: three seeds of one byte-identical unet_small@1.10.0 config for the sigma (without it every verdict in this study reads ungraded), plus two single-knob arms against that same control at a shared seed. 21 runs remain.

chose x0-4775d82bc6b7 — CALIBRATION control, seed 1, run as a SESSION rather than direct: first run of the family on a recipe this study modified last tick, so I want an agent reading the previews and describing a failure in
chose x0-269898940c77 — CALIBRATION seed 2, byte-identical config, direct. gradability reports no_sigma and a sigma needs three seeds of ONE configuration.
chose x0-337bc53ade3a — CALIBRATION seed 3, byte-identical config, direct — the run that turns two numbers into a sigma.
chose x0-3d975a041094 — EXPLORATION slot: augment=none, one knob against the control at the same seed. The manifest defines flip as horizontal AND vertical flip, so the default teaches upside-down 6s and mirrored 5s under
chose x0-93418a718b04 — INSTRUMENT check: selection_metric=val_iou, one knob at the same seed, where both arms share one training trajectory and only the kept epoch differs. Cheapest possible test of whether the metric built
set aside Park again on ask3c8e581b (confirm the objective)_ — It is already confirmed — objectiveConfirmed:true on overview. Parking a second tick on an answered question would have cost a day's wall clock for nothing.
set aside segformertiny against unet in round 0_ — Unreadable by construction until a sigma exists: 1-NN on raw pixels already scores 0.875 on these 24 images and a paired sign test needs about five discordant images. Reporting an architecture winner
set aside Wait for the larger scored set before running anything — Nothing in round 0 depends on it. A seed sigma and a correctness bug in the default augmentation are both readable at n=24; only the architecture ranking is not.
set aside Re-cut the snapshot class-stratified first — The missing 5s in val weaken checkpoint SELECTION but bias no scored number. A re-cut costs a tick plus re-characterisation for a selection-only gain; selecting on val_image_class_accuracy is the chea
set aside Build the global-pooled classifier recipe now — It buys no resolving power on a 24-image set 1-NN answers at 0.875. Held as the diagnosed fix under h0-seg-can-classify's falsifier — built only if the trio plateaus below 0.667 while the previews sho
set aside Pin contentbias explicitly_ — image_size resolves to 64 (the manifest minimum) while the images are 28x28, so the whole picture fits one window and there is no crop to place — the knob cannot act here. Left at its profile-derived
read 4 finding(s): find-char-ds_fb7985d4ebd7f9f9, find-recipe-image-class-accuracy-1100, find-label-detached-strokes, find-register-recipes-needs-explicit-manifests

Round 1 · tick-23e5186adfd64df5 · 2026-09-08 13:40

Round 0 closed and round 1 enqueued. Reading the five results from the archive rather than the graph changed what round 0 means: every val_iou curve is still rising at the epoch-60 cutoff (0.010->0.366, 0.016->0.352, 0.027->0.152, 0.012->0.566) and three of five runs pick the LAST epoch as best. 60 epochs is 1800 steps and 24 GPU-seconds against a 7-GPU-hour grant — nothing converged, so the sigma of 0.168 the fold now grades against is the spread of three curves cut early, not this recipe's seed noise. The selection-metric arm is worse than null: both rules kept epoch 60, so it scored one checkpoint twice. WITHDRAWN: nothing — round 0 had zero queued and zero running arms when this tick opened; all five had already finished, so there was no card to redirect. Round 1 re-asks every round-0 question at epochs=400 (max_steps=40000 so it does not bind): a three-seed calibration to re-measure sigma at the schedule it will grade, augment=none at two paired seeds, and class_weighting=inverse_freq as the exploration slot. Six runs, roughly 16 GPU-minutes, against 2 GPU-minutes spent so far.

chose x1-f37d7c84a6ef — LONG CALIBRATION control at seed 1 — a single-knob change (epochs 60->400) against x0-4775d82bc6b7, so the schedule itself gets a verdict; run as a SESSION because the first run of a new schedule is t
chose x1-ca6f1f37d19f — Seed 2 of that identical config — without a sigma measured at the schedule round 1 is graded on, every round-1 verdict would be scored against round 0's artefact.
chose x1-7180c6e0777c — Seed 3, the outlier of round 0 (val_iou 0.152, accuracy 0.333) whose curve was climbing exactly like the others; this arm decides whether that gap was a slow seed or a bimodal recipe.
chose x1-e07937d67f6c — augment=none at seed 1 against the long control — flip is horizontal AND vertical, and at 60 epochs none already reached val_image_class_accuracy 0.8611 against the control's 0.5556.
chose x1-1d5c76a779f8 — augment=none at seed 2, paired against the long control at the same seed — one seed cannot separate a knob from a draw; two paired seeds moving the same way can.
chose x1-03dc5648daaa — EXPLORATION slot: class_weighting=inverse_freq, the only untested knob with a mechanism — each digit class covers 0.87%-2.98% of pixels, so ~97% of the cross-entropy is background while image_class_ac
set aside Re-test selectionmetric=val_iou vs val_image_class_accuracy at the long schedule_ — Round 0 showed both rules keep the same epoch when the curve never turns over; c_1766acc9 is refuted as stated and selecting on the objective costs nothing, so every round-1 arm just does it rather th
set aside segformertiny at any config_ — c_52e52bd0: with 1-NN at 0.875 and one image worth 0.0417, an architecture ranking on 24 images is noise reported as a result. Deliberately unbought until the scored set grows.
set aside Build the dedicated classifier head — It is the diagnosed fix under h0-seg-can-classify's falsifier, and that falsifier did NOT fire — it requires a plateau below 0.667 and nothing plateaued. Building it now would be treating an undertrai
set aside Accept 0.211 as the smallest effect worth acting on, as gradability suggests — That sigma is an artefact of stopping early. Accepting it would wall the study at a number produced by a bug in my own schedule.
set aside Spend the exploration slot on lr or capacity (basechannels/depth)_ — Both are worth buying, but only once effects are gradable; neither has a mechanism as sharp as a 97%-background loss on a metric that reads only the foreground.
set aside Withdraw or re-run any round-0 arm — Nothing was queued or running — all five finished before this tick opened. The round-0 runs stay on the graph as evidence about the 60-epoch schedule, which is exactly what they are.
walls: Grant 5 rounds / 26 runs / 7 GPU-hours / $80. Spent: 5 runs, 122 GPU-SECONDS, round 0 of 5. After round 1: 11 runs used, 15 left, ~18 GPU-minutes used of 7 hours. Compute is nowhere near binding; the binding wall is the 24-image scored set and, until round 1 reports, a seed sigma of 0.168 that I believe is an artefact.

Round 2 · tick-cf6a6cff657de6be · 2026-09-08 13:54

Woken for an unappraised hint; found round 1 also complete (6/6 done) and closed it. APPRAISED the human's decision (ask_8b851c96-mtsporyy, proceed on ds_fb7985d4ebd7f9f9 as cut) and acted on it in this tick: no re-cut arm declared, and round 1 lets me price the defect at exactly one image rather than argue it -- val holds zero 5s and the single item that moves between configs is item 11 = mnist_5_005.png, a 5. ROUND 1 CONFIRMED THE UNDERTRAINING DIAGNOSIS: epochs 60 -> 400 moved the same config 0.5417 -> 0.9167 image_class_accuracy (paired +0.375, t 3.19, the fold's first better), the curves now plateau, and the three-seed sigma fell from 0.168 to 0.000. Six of six predictions hit. Two hypotheses died on their own falsifiers: augment=flip does NOT hurt at a converged schedule (none scores 0.9167/0.8750 against flip's 0.9167/0.9167 -- within one image at both paired seeds) and class_weighting=inverse_freq lands at effect exactly 0.000 with iou falling. THE TICK'S REAL FIND, from the archive: the residual error is the SAME TWO IMAGES in every config -- item 24 wrong in 6 of 6, item 3 in 5 of 6, both with per-item IoU exactly 0.000 (clean silhouette, wrong digit) while the other 22 score 0.140-0.904. I rendered the scored 24 and looked: both are genuinely ambiguous glyphs. Headroom is 2 images = 0.0833, exactly minPracticalEffect. Round 2 therefore buys the last question this cut can answer -- three segformer_tiny seeds at epochs=600, read PER-ITEM as ceiling attribution, explicitly not as an architecture ranking the 24 images cannot resolve.

chose x2-436cfe7c1085 — segformer_tiny seed 1, epochs=600, runner=session: first run of a new family, so the previews and curve shape get described rather than summarised, and epochs deliberately above unet's converged 400 b
chose x2-fd749980898a — seed 2, byte-identical: a claim about a failure SET from one seed is worth nothing; this and seed 3 give the family's own sigma the way round 1 gave unet's (which came out 0.000).
chose x2-666a0b4741ba — seed 3, completing the trio that decides h2 -- if all three miss item 24, the ceiling belongs to two ambiguous glyphs and the study closes on evidence rather than on budget.
set aside Any further unetsmall knob sweep (image_size, base_channels, depth, dropout, lr, loss=ce, flip_rot)_ — Six configs at three seeds returned sigma 0.000 and the same two failures; with 0.0833 of headroom and a 0.0417 resolution floor, no sweep on this cut can produce a gradable result -- it would look li
set aside Running segformertiny as an architecture COMPARISON against unet_small_ — The whole remaining range is two images, so ranking two competent families here is not honest. The same runs are bought instead as a per-item categorical question, which 24 items can actually answer.
set aside Building a dedicated classifier head — Its trigger under h0-seg-can-classify was a plateau BELOW 0.667 with well-traced silhouettes. The curves finally plateaued -- at 0.9167, above the 1-NN reference -- so the trigger is now testable and
set aside Re-cutting the snapshot class-stratified — The human decided this tick to proceed as cut, and round 1 shows the defect is worth exactly one unstable image (item 11, a 5) -- below the resolution floor and below minPracticalEffect. A future larg
set aside Re-asking selectionmetric now that curves turn over_ — It would finally be askable (best epochs are 60-340 of 400, no longer the last), but selecting on the objective is free and every arm already does it. Nothing to buy.
set aside Parking the study now — Tempting -- the question is essentially answered at 22/24 -- but one genuinely answerable question remains (ceiling attribution) and it costs ~10 GPU-minutes of a 7-hour grant. Parking before spending
walls: Budget is not the constraint: 11 runs done + 3 queued of 26, and roughly 0.3 of 7 GPU-hours spent. The binding wall is the 24-image scored set. Sigma at the converged schedule is 0.000, so noise is no longer the problem -- HEADROOM is: 2 images = 0.0833 separates the current best from perfect, and one image is 0.0417. After round 2 this study should close, because no further spend on this cut can resolve anything.
read 1 finding(s): find-round0-undertrained-60-epochs

Round 2 · tick-92fbfaa28c22a6c1 · 2026-09-08 14:00

Integrated the two round-1 findings (they were posted after the last tick's readThrough, so they woke this one; both were already reflected in the synthesis I wrote from them). The real work of this tick was the state of round 2: ALL THREE segformer_tiny arms are infra_failed, dead 12 seconds after launch with 'epochs=600 is outside the declared range [1, 400]' -- failureClass contract, 0.0 GPU-seconds, no metrics. I read the manifest on the box (/home/jha0006/research/recipes/segformer_tiny/recipe.json: epochs {min 1, max 400}, identical in unet_small) and confirmed this is NOT a recipe defect: the trainer enforced its own contract correctly and the defect was mine at declaration. It cost three run slots (runsLeft 15 -> 12 of 26) for zero evidence, because the range wall fires on the box after the run is claimed, not at declare or enqueue. Re-bought the same three arms at epochs=400 -- the range maximum, and exactly the schedule at which unet_small converged in round 1 -- so the family comparison is now matched on schedule rather than confounded by it. Nothing to withdraw: queued 0, running 0; the three arms this tick would have re-checked against the new evidence were already dead, and the new evidence does not touch their premise (h2's motivation IS the two-image finding). Posted find-round2-died-on-epochs-range-contract with the operational rule: read paramSpec min/max from the file on disk before declaring any non-default numeric value, because read_recipes returns only type and default.

chose x2-869f94870301 — segformer_tiny seed 1 at epochs=400 -- the re-buy of the arm that died on the contract; asks the per-item categorical question (does an attention backbone miss the same two glyphs?) at the schedule un
chose x2-732c1739b76a — seed 2, identical config -- the new family needs its OWN seed sigma; unet_small's 0.000 is a fact about unet_small.
chose x2-92f2227965a6 — seed 3, identical config -- three draws are what make a per-item failure set a claim rather than an anecdote.
set aside Bump segformertiny to a new version widening epochs to 600_ — A recipe defect is mine to fix, but this is not one: the manifest is the contract and the trainer honoured it. A longer schedule is already legal via scalingContract 'steps' with max_steps (max 200000
set aside Re-buy at 600 epochs under scalingContract 'steps' (maxsteps=18000) as originally intended_ — epochs=400 is where unet_small converged (best epochs 60-340 of 400, val_iou spread 0.006-0.021 over the last eight validations) and matching the schedule makes the family the only difference. If segf
set aside runner: session for the first run of the new family — The failure this family already had is caught at declaration now, and a watching agent costs ~$1 and two minutes a run. I can read the val curve and per-item vectors out of the archive myself next tic
set aside Withdraw round-2 arms in light of the new findings — There were none to withdraw -- queued 0, running 0, all three already infra_failed. And the new evidence is what MOTIVATES h2, not what refutes it.
set aside Build a dedicated classifier head — Its trigger was a plateau below 0.667 with clean silhouettes; the curves plateaued at 0.9167. Chasing two images that six configs all get wrong is what the one-image resolution floor forbids.
set aside Any further unetsmall knob arm_ — augment and class_weighting are both refuted at the converged schedule with effects of 0.000-0.042, and the entire remaining headroom is two images = minPracticalEffect. No knob in either manifest mov
walls: 26-run grant: 14 consumed (11 with evidence, 3 burned on the contract error), 3 queued, 9 left. Round 2 of 5. GPU is not binding -- 11 real runs cost well under 30 GPU-minutes of a 7-hour grant. The binding wall is the 24-image scored set: one image is 0.0417 and a paired test needs ~5 discordant images, so no architecture ranking is honest here whatever segformer scores.
read 2 finding(s): find-round1-converged-at-400-epochs, find-two-image-ceiling-same-items-every-config

Round 2 · tick-3ad0663b098208d4 · 2026-09-08 14:12

ROUND 2 CLOSED. Integrated the three re-bought segformer_tiny arms (x2-869f94870301 0.8750, x2-732c1739b76a 0.7917, x2-92f2227965a6 0.8750 at epochs=400, three seeds) and the unconsumed anomaly find-round2-died-on-epochs-range-contract. h2-ceiling-is-the-data-not-the-model is settled on its own falsifier: it required segformer to read BOTH item 3 and item 24 correctly; it read NEITHER at all three seeds. I pulled the per-item vectors out of result.json for all 9 converged runs across both families and lined them against the sealed 24-item order: item 24 (mnist_9_026.png) wrong 9/9, item 3 (mnist_1_005.png) wrong 8/9, nothing else stable (find-cross-family-ceiling-nine-runs). I opened worst.png and rendered the four hard items -- strokes cleanly traced, wrong class name over a correct outline, per-item IoU 0.000. The unplanned result: 1-NN on raw pixels reads mnist_9_026 CORRECTLY, so that half of the ceiling is the per-pixel filled-polygon objective (the label policy fills interior holes, erasing what separates a 9 from a 0), not the image; mnist_1_005 is missed by every method including 1-NN and IS genuine ambiguity. Wrote chapter-r2, rewrote the synthesis, carried the claims ledger forward to 25 claims. Nothing was queued or running, so nothing needed withdrawing. Declared and enqueued NOTHING: parking is the honest move and the 9 remaining runs are worth more held for a larger scored set.

set aside Withdraw queued arms in light of the new evidence (the tick's standing instruction) — There were none: queued 0, running 0. All three round-2 arms had already completed before this tick woke, so there was no card to redirect.
set aside Spend the remaining 9 runs on untested unetsmall capacity knobs (base_channels 32, depth 5, image_size 128)_ — Every one must come back within_noise. The headroom is two images = 0.0833 = exactly minPracticalEffect, and gradability says 9 runs resolve only 0.20. Draining the grant into inconclusive nodes is wo
set aside Re-buy segformertiny at a longer schedule under scalingContract 'steps' (max_steps=18000+)_ — Its curves are already converged -- val_iou spans 0.016 / 0.007 / 0.028 over the last eight validations, best epochs 250 / 220 / 80 of 400. The falsifier clause that would have required this did not f
set aside Build a dedicated classifier head to recover item 24, the image the objective loses — It targets ONE image = 0.0417, half of minPracticalEffect and below the one-image resolution floor. It becomes the first thing worth building the moment the scored set grows, and the synthesis says so
set aside Run a kind=evaluate arm on the saved checkpoints over decode knobs (mincomponent_px, instance_decode)_ — Free in run-wall terms but zero expected information: all three stable failures have per-item IoU exactly 0.000, i.e. the whole component carries the wrong CLASS. No decode or post-processing knob rea
set aside Declare a third architecture family for a broader ranking — 24 images cannot rank two competent families -- round 2 measured that rather than argued it (paired meanDiff -0.0833, se 0.0576, t -1.45). A third would add a third unrankable number.
walls: runs 17/26 consumed (14 with evidence, 3 burned evidence-free on the epochs range contract), 9 left; round 2 of 5; GPU ~40 min of 7 h -- so neither runs nor GPU is the binding wall. IMAGES are: 24 scored, two-image headroom, minPracticalEffect 0.0833, smallest resolvable effect 0.20.
read 1 finding(s): find-round2-died-on-epochs-range-contract

Every experiment

id	round	title	status	verdict	summary
x0-4775d82bc6b7	0	CALIBRATION control — unet_small@1.10.0 at defaults, selection_metric=val_image_class_accuracy, seed 1 (watched)	done	ungraded	unet_small@1.10.0 · 11 classes · train 240 / val 36 · epochs: 60 ep, 1800 steps, 14400 samples · cuda, 24 GPU-s · decode components · best ep 60 (val_image_class_accuracy 0.5556) · sc_mnist_digits_b2f34dc3 iou 0.3018 (n=24)
x0-269898940c77	0	CALIBRATION seed 2 — identical config to x0-4775d82bc6b7	done	within_noise (Δ 0.13)	unet_small@1.10.0 · 11 classes · train 240 / val 36 · epochs: 60 ep, 1800 steps, 14400 samples · cuda, 27 GPU-s · decode components · best ep 60 (val_image_class_accuracy 0.6667) · sc_mnist_digits_b2f34dc3 iou 0.3677 (n=24)
x0-337bc53ade3a	0	CALIBRATION seed 3 — identical config to x0-4775d82bc6b7	done	within_noise (Δ -0.21)	unet_small@1.10.0 · 11 classes · train 240 / val 36 · epochs: 60 ep, 1800 steps, 14400 samples · cuda, 26 GPU-s · decode components · best ep 46 (val_image_class_accuracy 0.3889) · sc_mnist_digits_b2f34dc3 iou 0.1344 (n=24)
x0-3d975a041094	0	EXPLORATION — augment=none against the default augment=flip, seed 1, one knob	done	within_noise (Δ 0.13)	unet_small@1.10.0 · 11 classes · train 240 / val 36 · epochs: 60 ep, 1800 steps, 14400 samples · cuda, 26 GPU-s · decode components · best ep 55 (val_image_class_accuracy 0.8611) · sc_mnist_digits_b2f34dc3 iou 0.4818 (n=24)
x0-93418a718b04	0	INSTRUMENT — selection_metric=val_iou (the recipe default) against val_image_class_accuracy, seed 1, one knob	done	within_noise (Δ 0)	unet_small@1.10.0 · 11 classes · train 240 / val 36 · epochs: 60 ep, 1800 steps, 14400 samples · cuda, 19 GPU-s · decode components · best ep 60 (val_iou 0.3658) · sc_mnist_digits_b2f34dc3 iou 0.3018 (n=24)
x1-f37d7c84a6ef	1	LONG CALIBRATION control — the round-0 control config at epochs=400, seed 1 (one knob: epochs 60 -> 400)	done	better (Δ 0.38)	unet_small@1.10.0 · 11 classes · train 240 / val 36 · epochs: 400 ep, 12000 steps, 96000 samples · cuda, 147 GPU-s · decode components · best ep 210 (val_image_class_accuracy 0.9722) · sc_mnist_digits_b2f34dc3 iou 0.6376 (n=24)
x1-ca6f1f37d19f	1	LONG CALIBRATION seed 2 — identical config to x1-f37d7c84a6ef	done	within_noise (Δ 0)	unet_small@1.10.0 · 11 classes · train 240 / val 36 · epochs: 400 ep, 12000 steps, 96000 samples · cuda, 139 GPU-s · decode components · best ep 90 (val_image_class_accuracy 1.0) · sc_mnist_digits_b2f34dc3 iou 0.5878 (n=24)
x1-7180c6e0777c	1	LONG CALIBRATION seed 3 — identical config to x1-f37d7c84a6ef (the slow seed of round 0)	done	within_noise (Δ 0)	unet_small@1.10.0 · 11 classes · train 240 / val 36 · epochs: 400 ep, 12000 steps, 96000 samples · cuda, 138 GPU-s · decode components · best ep 180 (val_image_class_accuracy 0.9444) · sc_mnist_digits_b2f34dc3 iou 0.6156 (n=24)
x1-e07937d67f6c	1	augment=none at the long schedule, seed 1 — one knob against x1-f37d7c84a6ef	done	within_noise (Δ 0)	unet_small@1.10.0 · 11 classes · train 240 / val 36 · epochs: 400 ep, 12000 steps, 96000 samples · cuda, 138 GPU-s · decode components · best ep 60 (val_image_class_accuracy 1.0) · sc_mnist_digits_b2f34dc3 iou 0.7547 (n=24)
x1-1d5c76a779f8	1	augment=none at the long schedule, seed 2 — one knob against x1-ca6f1f37d19f	done	within_noise (Δ -0.042)	unet_small@1.10.0 · 11 classes · train 240 / val 36 · epochs: 400 ep, 12000 steps, 96000 samples · cuda, 133 GPU-s · decode components · best ep 110 (val_image_class_accuracy 1.0) · sc_mnist_digits_b2f34dc3 iou 0.6362 (n=24)
x1-03dc5648daaa	1	EXPLORATION — class_weighting=inverse_freq at the long schedule, seed 1, one knob against x1-f37d7c84a6ef	done	within_noise (Δ 0)	unet_small@1.10.0 · 11 classes · train 240 / val 36 · epochs: 400 ep, 12000 steps, 96000 samples · cuda, 135 GPU-s · decode components · best ep 340 (val_image_class_accuracy 0.8889) · sc_mnist_digits_b2f34dc3 iou 0.5871 (n=24)
x2-436cfe7c1085	2	CEILING ATTRIBUTION — segformer_tiny at a converged schedule, seed 1: does a different family fail the SAME two images?	infra_failed
x2-fd749980898a	2	CEILING ATTRIBUTION seed 2 — identical config to x2-436cfe7c1085	infra_failed
x2-666a0b4741ba	2	CEILING ATTRIBUTION seed 3 — identical config to x2-436cfe7c1085	infra_failed
x2-869f94870301	2	CEILING ATTRIBUTION (re-buy at a legal schedule) — segformer_tiny, epochs=400, seed 1: does an attention family miss the SAME two images?	done	ungraded	segformer_tiny@1.10.0 · 11 classes · train 240 / val 36 · epochs: 400 ep, 12000 steps, 96000 samples · cuda, 289 GPU-s · decode components · best ep 250 (val_image_class_accuracy 0.9722) · sc_mnist_digits_b2f34dc3 iou 0.5469 (n=24)
x2-732c1739b76a	2	CEILING ATTRIBUTION seed 2 — identical config to x2-869f94870301 (segformer_tiny, epochs=400)	done	within_noise (Δ -0.083)	segformer_tiny@1.10.0 · 11 classes · train 240 / val 36 · epochs: 400 ep, 12000 steps, 96000 samples · cuda, 283 GPU-s · decode components · best ep 220 (val_image_class_accuracy 0.9167) · sc_mnist_digits_b2f34dc3 iou 0.4983 (n=24)
x2-92f2227965a6	2	CEILING ATTRIBUTION seed 3 — identical config to x2-869f94870301 (segformer_tiny, epochs=400)	done	within_noise (Δ 0)	segformer_tiny@1.10.0 · 11 classes · train 240 / val 36 · epochs: 400 ep, 12000 steps, 96000 samples · cuda, 293 GPU-s · decode components · best ep 80 (val_image_class_accuracy 0.8889) · sc_mnist_digits_b2f34dc3 iou 0.5494 (n=24)

All findings

observation · find-cross-family-ceiling-nine-runs · x2-869f94870301 · 2026-09-08 14:09 · unread

THE CEILING IS THE DATA AND THE OBJECTIVE, NOT THE unet_small FAMILY -- 9 converged runs, 2 structurally different backbones, one table. I read the per-item image_class_accuracy vectors out of result.json for all six round-1 unet_small runs and all three round-2 segformer_tiny runs (epochs=400 both families) and lined them up against the sealed 24-item order in the snapshot manifest. Only six of the 24 items are ever wrong: item file unet s1 s2 s3 | noaug s1 s… (list_findings for the rest)

anomaly · find-round2-died-on-epochs-range-contract · x2-436cfe7c1085 · 2026-09-08 13:59 · read by tick-3ad0663b098208d4

ROUND 2's FIRST ATTEMPT DIED EVIDENCE-FREE ON A MANIFEST CONTRACT, AND IT COST THREE RUNS. All three segformer_tiny arms (x2-436cfe7c1085, x2-fd749980898a, x2-666a0b4741ba) were declared at epochs=600 and refused by the trainer 12 seconds after launch, before a single optimiser step: result.json: "epochs=600 is outside the declared range [1, 400]. The manifest is the contract; widening it is a recipe version bump, not a run-time argument" failureClass: contract · gpuSeco… (list_findings for the rest)

observation · find-two-image-ceiling-same-items-every-config · 2026-09-08 13:50 · read by tick-92fbfaa28c22a6c1

THE RESIDUAL ERROR IS NOT SPREAD OVER THE SCORED SET -- IT IS THE SAME TWO IMAGES IN EVERY CONFIG. I pulled the per-item image_class_accuracy vector out of all six round-1 result.json files. . = read correctly, X = wrong class: x1-f37d7c84a6ef (flip, s1) ..X....................X 22/24 x1-ca6f1f37d19f (flip, s2) ..X....................X 22/24 x1-7180c6e0777c (flip, s3) ..X....................X 22/24 x1-03dc5648daaa (inverse_freq) ..X.........… (list_findings for the rest)

observation · find-round1-converged-at-400-epochs · 2026-09-08 13:49 · read by tick-92fbfaa28c22a6c1

ROUND 1 CONVERGED, AND IT CONFIRMS THE DIAGNOSIS: 60 EPOCHS WAS IMPATIENCE, NOT NOISE. At epochs=400 (12000 steps, ~147 GPU-s a run) the SAME config that scored 0.5417 at 60 epochs scores image_class_accuracy 0.9167 (22/24). Paired over the 24 items: meanDiff +0.375, se 0.1175, t 3.19 -- the fold graded x1-f37d7c84a6ef BETTER against x0-4775d82bc6b7, the round's only better verdict and the first non-null effect this study has measured. THE CURVES NOW PLATEAU. Read out of … (list_findings for the rest)

observation · find-round0-undertrained-60-epochs · 2026-09-08 13:36 · read by tick-cf6a6cff657de6be

ROUND 0 DID NOT MEASURE SEED NOISE — IT MEASURED WHERE FOUR STILL-RISING LEARNING CURVES HAPPENED TO BE CUT. I read every run's val_iou curve out of its result.json in the archive. All four are monotonically rising at the epoch-60 cutoff; not one has plateaued. x0-4775d82bc6b7 (seed 1): 0.010 -> 0.366, best epoch 60 = the LAST epoch x0-269898940c77 (seed 2): 0.016 -> 0.352, best epoch 60 = the LAST epoch x0-337bc53ade3a (seed 3): 0.027 -> 0.152, best epoch 46, still cl… (list_findings for the rest)

observation · find-register-recipes-needs-explicit-manifests · 2026-09-08 13:09 · read by tick-05698431a5194bc7

TWO CORRECTIONS TO THE RECORD OF TICK tick-40517497f1b15d6e, both discovered after the tick was recorded and neither changing any measurement. 1. register_recipes WITH NO ARGUMENTS DOES NOT REGISTER THIS BOX'S RECIPE FILES. My first call was research_tool(op="register_recipes", args={"recipes": null}). It answered ok, registered:[segformer_tiny, unet_small], source:"bundled" — and the graph still held 1.9.2 without image_class_accuracy. "bundled" is a library the route carri… (list_findings for the rest)

label_issue · find-label-detached-strokes · 2026-09-08 13:04 · read by tick-05698431a5194bc7

11 of 300 images carry more than one 8-connected ink component, and the labelling policy traces only the LARGEST one — so part of the digit's ink is labelled background. All 11 are in the train split; none is scored. Ink outside the polygon: mnist_5_003 42.3% (the entire top bar of the 5 — the only case where a whole stroke of the digit is lost), mnist_1_008 36.4%, mnist_7_001 28.4%, mnist_1_027 24.5%, mnist_4_004 23.9%, mnist_4_024 22.6%, mnist_4_009 19.1%, mnist_9_024 17.7%… (list_findings for the rest)

observation · find-recipe-image-class-accuracy-1100 · 2026-09-08 13:04 · read by tick-05698431a5194bc7

RECIPE BUILT — image_class_accuracy, unet_small@1.10.0 and segformer_tiny@1.10.0 (MINOR: a metric added, no parameter's meaning changed, metricImplVersion stays seg-metrics@2 because no existing number moved). WHY. The brief's metric is top-1 accuracy and no registered recipe logged it. enqueue refuses a prediction on a metric the manifest does not declare, so without this the study could not have had an objective at all — it would have optimised IoU because IoU was there. U… (list_findings for the rest)

The ledger

14 finished runs of 17 declared · 0.5 GPU-h · $17.72 in model calls · budget 26 runs / 5 rounds / 7 GPU-h.

Parked: Failure to learn, not failure to pay. 9 of 26 runs and 3 of 5 rounds remain and GPU is at ~40 minutes of a 7-hour grant, but no configuration arm on the 24-image scored set can produce a gradable result: the entire remaining headroom is two images (22/24 -> 24/24), minPracticalEffect is two images, and gradability puts the smallest resolvable effect at 0.20 with the runs left. Round 2 spent its pivot answering the last question this cut CAN answer and the answer closed the direction. The park ends when the human supplies a larger, class-stratified labelled cut (ask new_data) or decides otherwise (ask decision); the 9 runs are held for it.

Generated from the research graph and its event log (134 events to seq 2141); every number above is on the record.