2026-11-13
Accuracy worklist: what we have tried and what we have not (13 Nov 2026)
Short answers to Alex's three questions
1. Was last night's vote apples to apples? Both models got the same treatment, but it was the wrong treatment. The page showed each model's raw lines, before the clean-up a customer scan goes through. Production removes a second line on the same stripe (that switch is on in the live server), and the vote page did not. So the doubled stripes you saw would not have reached a customer. They were not one-sided either: the live model doubled up in 3 pairs (0f4a, 8ab6, c9c1), the retrain in 1 (4e06), and fa36 is unclear. Three photos with no draft stripes were also in the pool. The vote is being rerun with both models through the full production pipeline and the no-stripe photos removed.
2. Lane testing. Never trained. It is task S1 in the worklist, proposed on 5 Sep, 17 Sep and 2 Nov. The worklist put it after the new scorecard and the baseline retrain, and both of those now exist. The first lane-model run started today.
3. What in the two worklists is untried. Of 58 numbered tasks: 12 done, 6 partly done, 40 not started. Most of the 12 are the checks and the scorecard. Of the retraining ideas, only three have been run (baseline, tighter point grading, the flip-order fix) and none beat the live model. The full list is below.
One lead from the done pile: check C12 found the live model already draws 32 of its 43 missed stripes, but scores them under our confidence cut-off. Lowering the cut-off gained 13 photos on raw output. That has not been tested through the full pipeline yet, so nothing has changed for customers.
Counts
- DONE: 12 of 58 items
- PARTLY: 6 of 58 items
- NOT STARTED: 40 of 58 items
Not started yet
"Waits on" is the "Can start" column of the worklist's checklist.
| id | task | what the doc says it waits on |
|---|---|---|
| S1 | Lane-detection model as a challenger finder (stripes drawn as whole curves, not 8 loose dots) | After R1 (the new scorecard, now built) and T0 (the baseline, partly done). Also needs a licence check on the code and no weights trained on research-only data |
| C4 | Jitter test: scan each customer photo 10 times with tiny changes and see how much the numbers wobble | Now |
| C5 | Per-point confidence: does the model's own "how sure am I" track real error | Now |
| C8 | Half-pixel fixes re-scored through the full pipeline | Now |
| R2 | Practice set and sealed set of customer photos | After Alex's keep/drop pass on the customer test photos. A try on 13 Nov found none of his choices saved, so nothing could be frozen |
| R3 | Gold answers: two independent hand traces per photo (measures human noise) | After Alex approves the label guide, and he must settle "very short stripes at the head: always label or never?" |
| T2 | Flip rate 0.5 | After C1 and C2b (both now done) |
| T4 | Cut-stripe training that matches the label rule | After C3 (now done) |
| T5 | Train on labels moved to the bottom edge of the tape | After T0 |
| T6 | Clean finish: short last training stage on trusted labels only | After T0 |
| T7 | Photos with no stripes added as background examples | After C6 (now done) |
| T8 | Remove duplicate photos and group near-copies before splitting | After C6 (now done) |
| T9 | Pick the best checkpoint using our own scorecard instead of the standard score | After R1 (now built) |
| T10 | More pixels on the top stripe (bigger input, finer detail head) | After T0 |
| T11 | Stack the winning changes | Last, only changes that passed the win rule |
| T12 | Curve-distance loss (grade each point by distance to the drawn curve) | After T1 |
| P1 | Average the curves of the three finders we already have | Build now, judge on R1 |
| P2 | Stop lines at the last confident point | After C3 (now done) |
| P3 | Turn the photo upright from the phone's tilt reading | After confirming tilt is in the shipped app build |
| P4 | Average over several slightly shifted copies of the photo | After C4 |
| P5 | Clean-up rules tuned separately for each finder | After R2 |
| P6 | Measure on the largest stored image, not the small copy | With M1 |
| P7 | Twist that does not depend on a bad boom | After C13 (partly done) |
| P8 | Use the whole sail to flag a doubtful top stripe | After C5 or M2 |
| M1 | Straightened-strip measuring stage, no training | Build now, judge on R1 and R3 |
| M2 | Same stage as a small trained model | After M1 passes |
| S2 | Pseudo-labels from unlabelled customer uploads | After T6 |
| S3 | Agreement across boom-cam frames | After C4 |
| S4 | Photo-quality gate ("cannot measure this photo") | After T7 |
| S5 | Lens distortion check | After R1 |
| S6 | Automatic bad-label ranking | Now |
| S7 | Use SAM 3's sail mask to find luff and leech ends | After R1 |
| S8 | Camera-angle correction for twist | After R1 |
| S9 | Synthetic sails | With M2 |
| S10 | Human re-trace noise floor | With R3 |
| S11 | Learned keep-or-drop judge over the model's candidate lines | After R2 |
| S12 | 3D estimate from the photo | After S8 |
| S13 | Carry labels along boom-cam sequences | After T8 |
| S14 | Choose what to label next by finder disagreement | After C4 |
| S15 | Automatic exposure for the tape | Later |
Partly done
| id | task | what was done | what is left | source file |
|---|---|---|---|---|
| C13 | Boom and twist error | Boom error measured on the 44 photos with a boom label, using the 28 Sep boom model: found 41 of 44, typical angle error 1.2 degrees but a few bad booms push the average to 5.0 and twist error average to 13.2 degrees. Stored scans show twist was often based on a guessed frame-edge boom before 21 Oct and almost never since | The boom model was not re-scored on today's setup; counts come from stored scans, not every customer scan | report47.md |
| R4 | Three seeds and paired tests | Each retrain was run with several seeds. Found that the training software gave identical results for different seeds, so past "seed sweeps" measured nothing. Patched so seeds really differ | Only 2 truly different seeds for the baseline and the order-safe arm. No paired significance tests run (only a raw spread of about 9 photos between two identical retrains) | report50.md |
| R5 | Win rule | Item 6 (Alex's blind vote on 40 photos) was run: live model won 20, best retrain 14, 2 ties, 4 both wrong | Items 1 to 5 (paired error comparison, jitter spread, practice/sealed sets) were not applied; the retrain was picked on camber plus end error | t50-vote.md, report50.md |
| T0 | Baseline retrain of today's recipe, 3 seeds | Retrained exactly as the live model was; best and last checkpoints both scored. Spread between the two different seeds is about 9 photos | Only 2 genuinely different seeds (a third run was an exact replica); not yet compared on practice/sealed halves | t50/ledger.md |
| T1 | Tighter sigmas (how strictly training grades point placement) | Tried 0.05 starting from the live model, 3 different seeds. Photos-right stayed about the same but the strict score fell (89 vs 108 for the live model) and camber error rose | 0.03 not tried; not tried from scratch rather than from the live model | t50/ledger.md, report50.md |
| T3 | Order-safe training (fix a bug that scrambled point order when flipping photos upside down) | Option A built and proven on the pod: scrambled share dropped from about 26 to 29 percent to 0.00 percent. Retrained; ends of lines moved slightly in the right direction. Alex's vote: live model 20, this retrain 14, so it does not ship | 2 genuinely different seeds, not 3. Option B (loss accepts either direction) not tried. Practice/sealed judging not possible yet | report50.md, t50-vote.md |
Done
| id | task | result in one line | source file |
|---|---|---|---|
| C1 | The live model's real recipe | Trained 19 Oct, 100 epochs max (95 ran), flips left-right 0.5 and up-down 0.3, rotation 20 degrees. Shipped file is the best epoch (75). Sigmas are the library default, never set by us | report46.md |
| C2a | Point order in stored labels | About 35% of stored stripes run right to left (34.7% of training, 35.8% of test). Training export re-sorts them left to right, so it does not reach the model directly. 0.3% are scrambled | report46.md |
| C2b | Point order after photo changes in training | Up-down flip scrambles point order on 29.07% of stripes (left-right flip 0.00%, rotation 0.39%). Cause is the up-down flip reusing the left-right flip's point swap. Confirmed again on the pod in the retrain run | report46.md, report50.md |
| C3 | Does the model say which points are off the photo | Yes. Points removed by the kink-trim rule had median visibility score 0.013 against 0.978 for kept points (small sample: 9 stripes). Edge points score 0.42 vs 0.99 interior | report46.md |
| C6 | Training set facts | No empty-label images. Duplicates remain (540 extra exact copies; 879 near-copy pairs cross the train/validation/test split) though the audit's "421" was not reproduced. Bottom-edge labels only 0.53% of training stripes | report46.md |
| C7 | Which draft customers see | Plain draft; the steadier draftStable is computed but no app or desktop code shows it |
report46.md |
| C9 | How a hand label becomes 8 training points | Current runs use a straight even-spacing resample, no smoothing. But the code default is still the smoothing mode, and the live model inherits weights first trained on smoothed targets | report47.md |
| C10 | Labels hit by the 29 Sep labeller bug | Fix found (28 Sep). Between 156 and 545 labels match the bug's signature (upper bound, cannot prove which); none are tagged and all are in training data | report47.md |
| C11 | What the reviewer sees and saves | Reviewer was shown stale pre-live-model lines on about 60% of the queue. No reviewer saves exist yet. A save nudged by a few pixels could have trained as human. A fix (seed follows live model; unedited reviewer saves tagged and excluded) is now in master; deployed to production 13 Nov 04:27 | report47.md, report49.md |
| C12 | Missed stripes below the confidence cut-off | Of 43 missed stripes, 32 were drawn by the model but scored below the 0.45 cut-off, 6 drawn roughly, 4 not drawn at all. A 0.15 cut-off gains 13 photos net on raw model output (0.05 loses 9). Production filters not replayed | report54.md, c12.md |
| R1 | Measurement scorecard | Built and used on every retrain: line distance, end errors, camber, draft, entry, exit, twist, split by position. Live model scores 264 of 327 photos right, typical camber error 0.30 | report50.md, t50/ledger.md |
| R6 | Results cannot be lost | Every retrain and the C12 run saved settings, results and weights to cloud storage before the GPU shut down | report50.md, report54.md |