Did our latest retraining attempts beat the model that's live?

1. The question we asked

The model that finds the stripes on your sail photos is called v39. We tried several ways of retraining it that we thought might make it a little better: grading itself more strictly while it trains, and fixing a bug that scrambled the order of a stripe's points whenever training flipped a photo upside down. Each recipe was trained more than once, so we could tell a real improvement from plain luck.

2. The answer: no new model beat the one that's live

Every retrain landed within luck-of-the-draw variation. Training the exact same recipe twice with different random starts already swings the score by about 9 photos, which is bigger than any gain from the changes we tried. So none of them is a proven upgrade.

3. The one bright spot

The flip-bug fix is the only change that moved the thing it was aimed at: it slightly improves where the lines start and stop on the sail (by a few hundredths of a percent of the stripe's length). That's small, and it could still be luck, so we're not claiming it. Alex's 40-photo vote at https://rate-t50.sailscanapp.com decides whether it ships: if his picks favor the new model, it moves forward to a stricter final check before going live.

4. What it cost and what we try next

About $100 of GPU rental. Roughly $20 of that went on two bugs in our own training setup, which we found and fixed along the way. One of them means older "different random start" tests on this software never actually varied, so some past comparisons were weaker than we thought.

Next: wait for Alex's vote, then try the strict-grading idea again from a fresh start instead of from the live model, plus a different way of scoring how close a line is to the true stripe.

	Live model (v39)	Best new model (flip-bug fix)
Photos with all lines right, out of 327	264	270
Typical camber error	0.30	0.30

The 6-photo gap is smaller than the 9-photo spread between two identical retrains, so treat it as a tie.

Technical detail

Everything below is the original working report, for the record.

task50: T0/T1/T3 retrain matrix + R1 scorecard + blind vote (12-13 Nov 2026)

RESULTS (the short version)

Did tight sigmas (T1) or order-safe flips (T3) beat T0 noise? NO — both land inside T0's own seed spread.

Full table: https://reports.sailscanapp.com/t50-ledger/ (source: server/inference/r1/LEDGER-t50.md, copy at t50/ledger.md). All rows = the same raw-model R1 harness (conf 0.45) on the 332 key / 327 live subset; v39-ref = shipped v39 weights through it (normal 264/327, strict 108, camber med 0.30, luff/leech end med 0.79/0.80 %chord).

T0 (exact v39 recipe, distinct seeds s0+s2): camber 0.30, ends 0.77/0.83, normal 262-271. The 9-photo spread between two seeds IS the noise floor — bigger than either treatment effect.
T3 (order-safe flipud): camber 0.35, ends 0.75/0.77 (the only arm that moves its target metric — ends — in the right direction, by ~0.03-0.06 %chord), normal 264-270, strict ~108. Within noise.
T1 (kpt_oks_sigmas 0.05, warm-start from v39): normal 271 every seed (best-ckpt collapses to ~epoch 1-2 off the warm start; last.pt rows differ), camber 0.40 (worse), strict collapses to 89 (v39: 108). Tight sigmas as-warm-started did NOT sharpen shape on this ruler.
T1T3: camber 0.33, ends worst of all arms (luff 0.86), wide normal range 254-270.
Winner by camber+end (task rule): t50-T3-s0 (1.06) over T0-s2 (1.09) and v39-ref (1.09) — effectively a tie. T3-s0 best.pt goes to the blind vote vs v39.

Two methodology findings that outlast this task: 1. ultralytics 8.4.92 trains seed-identically in our setup (args.seed never reaches the dataloader generator, data/build.py:359; sampler shuffle is epoch-seeded only). T0-s1 == T0-s0 to 5 decimals over all 100 epochs proves full-recipe determinism. Every historical "seed sweep" on this version measured nothing. Fixed in t50_patches via a sys.modules-wide rebind of build_dataloader + generator reseed (first patch attempt silently didn't fire — trainers bind the symbol at import; guards added so that can't recur). 2. C2b flipud bug confirmed on-pod and fixed cleanly: stock augs reverse point order on 26-29% of train stripes; the order-safe wrap takes it to 0.00% (selftest, 500 samples, both measured every pod).

Status: blind vote LIVE for Alex — 40 customer pairs (batch t50-95a0eb), T3-s0 best.pt vs shipped v39, https://rate-t50.sailscanapp.com (key sent in chat; unblind = t50/blind/unblind.mjs + mapping.json, never deployed). Spot-checked pair dfbe by eye: upright frame, lines on the sail, genuine 3-vs-4-stripe difference between arms.

Plan (from task50.md + worklist sections 1.2/1.5/3/4/9 + report46 C1/C2b)

v39 recipe (report46 C1): yolo26l-pose, ultralytics 8.4.92, init-l = COCO yolo26l + v33-soup overlay, SGD lr0 0.01, 100 ep patience 20, imgsz 960 batch 16, degrees 20 fliplr 0.5 flipud 0.3, raw 8-kp sampling, seed 0, best.pt shipped (epoch 75). Sigmas = default 0.125. flipud reverses point order on ~29% of stripes (C2b) — T3 fixes.
Arms, 3 seeds (0/1/2) each: T0 unchanged; T3 order-safe (option A: reverse-to-ascending-x after geometric augs; degrees 20 already ≤ 45 cap); T1 kpt_oks_sigmas 0.05 warm-started from v39; T1+T3 both.
R1 scorecard (worklist §3): per matched stripe vs 332 answer key — point distance (px at stored-upload size + % of chord, mean/worst), luff/leech end error (% chord), length vs chord, camber/draft(stable)/entry/exit/twist error; splits top/mid/bottom, main/jib, portrait/landscape, cut-stripe; medians + p90; plus normal(3%) and strict counts. v39 scored first as reference. best.pt AND last.pt per run.
R6: every run uploads config+seed+weights+per-photo results to R2 scratch/t50/ before pod stop. SAILSCAN RunPod account, H100 preferred. $ per run logged.
Then: ledger row table → publish-report.sh (task48); blind A/B page, 40 customer photos (custset candidates) where best-by-R1(camber+end) differs most from v39; ONE Boats message at the end.

Log

18:2x started. report46 present; worklist read from /Volumes/MacMiniExtra/scratch/alex-accuracy-doc/fable52-worklist2-2026-11-12.md (docs/ path in task does not exist). publish-report.sh (task48) not present yet — will poll.
Dispatched: Explore (scorer/blind-page/sail-type inventory), opus-low (t50 training kit: pod_t50.sh, t50_patches.py order-safe + sigma injection with hard verification, launch50.sh per arm).
Explore done. Key facts: normal score = prod_baseline_score.mjs (resample 20, cost meanDist/diag, optimal match, HEADLINE_CAP 0.03, linesPerfect = counts equal + no miss/extra); strict = DevCaches clean-pool/ruler (5% cap match, 60-pt resample, interior-80% vertical offset, pass if all matched dyMaxAbsDiag ≤ 0.30% diag; calibrated on Alex's spot30). 327 = 332 − server/inference/holdout-scoring-exclusions.json (5 ids, Alex-approved 18-19 Oct). Anchor: docs/onnx-serving.md:74 raw v39 conf≥.45 = 274/327. Blind-page template: server/inference/blind-ensemble/ on branch exp/blind-sam3-keepdraft (pairs.json + Pages Function → R2 sailscan-blind-ratings, X-Blind-Key sha, mapping.json never deployed). sailType field = mainsail|headsail. Twist without boom: computeTwist().perStripeVsBottom. Cut-stripe: stripeEdgePartialFlags margin 2%.
Decision: R1 harness scores RAW model output (no prod chain on pod) through ONE fixed harness for all rows incl. v39-ref; the 274/327 doc anchor validates it. Normal+strict counts in the ledger are same-harness numbers, labelled as such (prod-chain 279-style numbers are a different path).
Dispatched sonnet: server/inference/r1/ scorer (r1_infer.py, r1_score.mjs using production stripeMetrics.js, score_run.sh, synthetic fixture). Contract sent to kit agent: pod exports DS, extracts rules.tgz, calls score_run.sh per checkpoint; T0 pod scores v39-ref first.
Kit done (opus agent). KEY FINDING: ultralytics 8.4.92 never reads kpt_oks_sigmas from data.yaml (string absent from the wheel) — T1 sigma is applied by patching v8PoseLoss.init (loss) AND PoseValidator.init_metrics (val mAP → best.pt pick), with fatal verification lines the pod greps for. Order-safe = wrap of Format.call (last transform): whole-array reversal to ascending-x using visible points; selftest gates ON ≤1% / OFF ≥5% reversed (fatal for T3/T1T3). Side effect noted: T1 best.pt is selected under tight-sigma val mAP, so val mAP is not comparable across arms — R1 is the only cross-arm ruler. --cache ram added (speed only; v39 ran uncached).
R1 scorer done (sonnet agent): server/inference/r1/ (r1_infer.py conf default 0.45 per docs/onnx-serving.md:74 anchor 274/327; r1_score.mjs ports prod_baseline_score matching (normal 3%) + clean-pool ruler strict (0.3% diag, interior 80%); production stripeMetrics.js for camber/draftStable/entry/exit + computeTwist perStripeVsBottom; signed end errors, + = stops short). Fixture passed (perfect/offset/missing behave as specified).
18:3x LAUNCHED all 4 arm pods (background): T0, T3, T1, T1T3; CAP_USD 40 each, SEEDS "0 1 2", H100-first. T0 pod also scores v39-ref first.
Launch hiccups, both fixed: (1) launcher's final ssh hung (remote detach held the session) — T0 launched fine but blocked the loop ~27 min; patched to subshell-detach, killed the hung ssh. (2) r1_infer.py recorded synthetic ids ("image0.jpg") for list predicts → v39-ref first scored "pred missing 332"; fixed to zip input paths, re-scored. (3) kit pycache (my py_compile) broke T1T3's scp under set -e → pod was created but never started; completed .env + detach manually; verified all kit files present. Scorer fix pushed to T3/T1 pods (launched with stale copy).
Pods (all RUNNING): T0=uolhhpelsf0csl H100 AP-IN-1 $4.39/hr; T3=b0ybxikklkp7tl H200 EUR-IS-4 $5.29/hr; T1=s214nz4p8xcqjh H200 $5.29/hr; T1T3=roscn2yqb2hlh2 H100 CA-MTL-1 $4.39/hr. Combined $19.36/hr, MAX_MIN watchdogs ≈$40/pod cap.
v39-ref REFERENCE ROW (R1 harness, raw v39 conf 0.45): normal 264/327 live (266/332), strict 108/327, camber err median 0.30 / p90 1.3, frame mismatches 3, pred missing 0. Doc anchor was 274/327 (onnx-serving.md) — 10-photo gap vs the old scorer run, cause not chased; ALL rows use this same harness so arm comparisons are internally consistent.
T0 selftest (patch OFF→ON informational): ON reversed 0.00% — and OFF confirmed ≥5% (PASS line), i.e. the C2b bug reproduces on-pod and the order patch kills it.
Blind-vote kit built (t50/blind/): launch_blind.sh → cheap pod renders 40 most-different pairs (both models on EXIF-transposed clean.jpg, diff = count mismatch sentinel or max matched-stripe mean distance %diag, seed 20261112) → deploy_blind.sh (Pages sailscan-rate-t50, domain rate-t50.sailscanapp.com, sha256-keyed ratings → R2 sailscan-blind-ratings blind-ratings//, mapping.json + .rating-key never deployed) → unblind.mjs. At use time I must produce candidates.json (flat [{id}] from custset-candidates.json minus holdout ids).
19:4x INCIDENT: ultralytics 8.4.92 nests runs at runs/pose/runs//train/ — kit's fixed weights path missed it (v39's original script used find; the kit regressed that). T1 + T1T3 trained s0 fine (T1T3: sigma proof lines present, early-stopped ~22 epochs off the v39 warm start), then failed the weights check and self-terminated — s0 weights lost (~$3.5+$2.9 wasted). T0/T3 still mid-s0: hot-fixed via symlink watcher (linkfix.sh, runs//train → runs/pose/runs//train) so no running-script edit. Kit patched (find-based discovery, early train.log upload, RUNDIR for results.csv/args.yaml). T1/T1T3 RELAUNCHED with fixed kit.
Selftest numbers on-pod: T0 OFF 25.99% → ON 0.00%; T3 OFF 29.26% → ON 0.00% (matches C2b's ~29%). Order patch verified working.
T0 pace: ~1.05 min/epoch (234 it @ ~3.9 it/s) → ~1.8-2 h per 100-epoch run; warm-start arms early-stop ~22 epochs → ~30 min/run.
20:2x: relaunched T1=i96v0ato706jz2 (H100 EU-FR-1), T1T3=d5v5h9xlhem1xh (H100 US-MO-1), both $4.39/hr. Patched kit WORKS end-to-end: T1-s0 and T1T3-s0 trained (wall ≈1,380 s each, early stop off the v39 warm start), GATES OK, best+last uploaded, both checkpoints R1-scored, s1 training. T0 at 95/100 on s0; T3 mid-s0.
First single-seed headlines (same harness; v39-ref normal 264/327 live, strict 108): T1-s0 best 271/327 normal, 89 strict; T1-s0 last 256/89. T1T3-s0 best 254/100; last 258/74. Pattern so far: tight sigmas up on normal, down on strict — judge on camber+end medians over 3 seeds, not this.
21:1x MAJOR FINDING (worklist-grade, corrects R4's assumption): ultralytics 8.4.92 trains seed-identically in our setup. build_dataloader seeds its generator 6148914691236517205 + RANK (args.seed NOT mixed in; wheel source, data/build.py:359) and the sampler epoch-shuffle is manual_seed(self.epoch) only. init_seeds(seed+1+RANK) only touches main-process RNG → with a FIXED warm start (v39.pt or init-l.pt), dropout 0 and deterministic=True, different --seed = bit-identical runs. PROOF: t50-T1 s0/s1/s2 results.csv identical except wall-time column; all three seeds scored identically (271/89 best). T1T3 same (254/100). So v39-era "seed sweeps" on this ultralytics version measured nothing.
Mitigation: t50_patches now reseeds the DataLoader generator (6148914691236517205 + 1000003×seed) when --seed>0; seed 0 stays stock = exact-replica row. Deployed to T0/T3 pods (their s2 picks it up; their s1, already in flight, will be an s0 replica = determinism check). T1/T1T3 relaunched with SEEDS="3 4" CAP_USD 15 for two genuinely varied runs each.
T0-s0 done end-to-end via symlink fix (wall 6693 s ≈ $8.2); T3-s0 done (6122 s ≈ $9.0). T1/T1T3 original pods DONE + self-terminated; per-run cost ≈$1.7 (wall ≈1,380 s each on H100).
config.json files in R2 are valid JSON after a torchvision-warning preamble — parse from first "{" (cfgsum.py).
22:2x SECOND seed-patch failure, now really fixed: seeds 3/4 ALSO trained identically (results.csv differs only in wall-time col; scores byte-equal). Cause: the hook patched ultralytics.data.build.build_dataloader, but trainers bind the name via from ... import build_dataloader at import time, so the wrapper never ran (its log line absent in s3 train.log while all other [t50] lines present). Fix: sys.modules-wide rebind of every module-level reference + counter + TWO fatal guards (<2 rebinds at install → abort; wrapper never fired after train → abort). Pushed to T0/T3 pods in time (both still mid-s1, verified md5), so their s2 will be genuinely varied. T1/T1T3 relaunched with SEEDS="5 6" (3/4 keys burned as replicas). publish-report.sh now exists (task48 done).
Seed ledger bookkeeping: per arm the DISTINCT runs will be: T0/T3 → s0 (+s1 replica proof) + s2 varied; T1/T1T3 → s0 (s1-s4 replicas) + s5 + s6 varied.
00:5x: all 12 runs + v39-ref scored; all pods self-terminated (RunPod list clean). Ledger built → winner T3-s0. T1 last.pt rows differ across varied seeds (259/88 vs 258/84) proving streams varied; identical T1 best.pt scorecards = warm-start best-epoch collapse, not a harness bug. Blind render pod launched (T3-s0 vs v39, 180-candidate pool, top-40 most different).
01:1x FINAL: blind page deployed (Pages sailscan-rate-t50 → https://rate-t50.sailscanapp.com, pairs.json + index 200, batch t50-95a0eb, 40 pairs; render pod ofjk0gn07cxq50 self-terminated, ~$0.30). Ledger published https://reports.sailscanapp.com/t50-ledger/, this report https://reports.sailscanapp.com/report50/ (both 200). Boats message DELIVERED (verified in the group via BlueBubbles read). Unblind: t50/blind/unblind.mjs with t50/blind/mapping.json + .rating-key (local only).
TASK COMPLETE. Follow-ups for a future task: (a) Alex's blind verdict → unblind; (b) if T3 wins the vote, T3 is the promote candidate but needs the R5 win-rule run (practice/sealed + paired tests — R2/R3 still wait on Alex's custset pass); (c) T1 strict-collapse suggests trying sigmas 0.05 from scratch (not warm-start) or T12 curve-distance loss; (d) ultralytics seed bug memo saved to memory.
01:4x PUSH SAGA (all resolved, master green): exp(t50) commit f5c4a4d5 pushed after clearing two pre-existing blocks on the shared checkout — (1) server suite crashed on missing nodemailer (another session committed welcomeMail.js without installing; npm install in server/ fixed it), (2) brandRollout.test.ts still asserted the pre-F-07 site strings while the committed homepage (4790df0f) shipped new ones, so EVERY push failed; aligned the guard with the shipped copy (fec6f502, 3/3 pass). Also repaired a stash-pop conflict in .gitignore (merged both additive sides) and restored the other session's WIP to unstaged — tree left as found, stash dropped.

Spend

Training wall-time captured in configs: $75.94 over 20 runs (includes $13.75 of seed-bug replicas). Not captured: pod setup/scoring overhead (~25-40 min × 8 pods ≈ $15-20) + the two first-launch T1/T1T3 pods lost to the weights-path bug (≈$6) + blind render pod (≤$3). Honest total ≈ $100.
Wasted on the two seed-patch failures: T1/T1T3 s1+s2 replicas ≈ $6.7, s3+s4 replicas ≈ $7 incl. setup overhead.
T1 s0+s1+s2: $5.05 (but s1/s2 are replicas of s0). T1T3 s0+s1+s2: $5.03 (same). T0 s0: ~$8.2. T3 s0: ~$9.0. Setup overhead per pod ~25-40 min. Wasted on the weights-path incident: ~$6.4 (first T1/T1T3 pods).

(per-run GPU $ appended here as pods report)