← All reports

2026-11-12

task47: worklist v2 checks C9–C13 (12 Nov 2026)

Method: read-only. Code read on master (ae0f5778). Live data pulled read-only from R2 over the S3 path (labels, sam3-seeds/, photo meta, predictions/) and Railway variable names/values for a few flags. No prod change, no model load, no GPU. Raw pulls are in /Volumes/MacMiniExtra/scratch/elvstrom-poc/t47/ (labels-live.jsonl 2,114 records, seeds-current.json, seeds-boom.json, metas.json, predictions.json).

Status line

ID Status
C9 done
C10 done (count is a signature upper bound, see caveat)
C11 done, risk is real but contained today (nothing trained yet)
C12 not answerable from stored data: no v39 raw-candidate dump exists anywhere; needs one pod run (plan below)
C13 done for the 28 Sep boom model; prod-side boom-source counts done from stored seeds

C11 (urgent): what the reviewer sees and saves

Question Answer Evidence
Which recipe seeds Combined Review today? The cached seed in R2 sam3-seeds/<photoId>__r0__49adfc4afb.json, recipe finder_v27_flow_v26_refine_1 (the finder=v27&flow=v26&refine=1 query). Not a live v39 run. A live chain runs only when no cached seed exists (cache miss) or on "Redraw". Editor open: labeler/web/src/App.tsx:683 (fetchSeed), labeler/web/src/lib/labels.ts:641-667 → GET /api/photos/:id/predict. Server: server/index.js:2569-2578 (getSeed hit returns the stored seed), server/sam3SeedCache.js:85 (key = recipe hash only), :197-212 (valid if recipe hash matches), server/sam3Live.js:31-32,50-62 (recipe = refine 1, flow v26, finder from SAM3_FINDER; Railway has SAM3_FINDER=v27; checkpointRevision is null).
Why is it stale? The cache key has no model identity (no checkpoint, no stripe-model version). Changing the stripe model (v39 went live 21 Oct) does not invalidate any seed. A saved label always wins over the seed (App.tsx:55-65, planLoadRestore), so only unlabeled photos show the stale seed. same files
How stale are the seeds? 2,986 seeds under the current key. Stage label in the seed's own recipe: v27 on 2,509, v39 on 21, unstamped on 455. Computed 21 Sep to 12 Nov; 2,234 of them on 2-3 Oct (bulk backfill). t47/seeds-current.json
The reviewer's queue (user-uploaded, unlabeled) 372 photos (doc says 349; my filter is uploader email not internal and no live label, so it includes some the queue hides). 335 have a cached seed: 56 stamped v27, 1 stamped v39, 278 unstamped. 200 of 335 were computed before 21 Oct (v39 switch). 134 unstamped ones were computed after 21 Oct and probably came from v39; the stamp is missing so I cannot prove it. 37 have no seed and will run live (v39). t47/queue_unlabeled_user.json
So is the reviewer tracing stale lines? Yes for about 60% of the queue (200/335 cached seeds predate v39), and for the 56 stamped v27 the lines come from the old finder. above
Can an unedited seed saved by a reviewer become a training label? Exactly unedited (every point within 0.002 normalised, about 3 px): no. It is tagged drawnBy:"model"; approvedBy is only written for role admin, so a reviewer-role save stays unapproved and train.py drops it. Nudged by a few pixels: yes. One point moving more than 0.002 flips the tag to drawnBy:"admin" (a reviewer is "trusted", so isTrustedReviewer makes the tag admin), it is not a customer correction (saver is not the photo's owner, so no pending stamp), and it trains as a human label. server/index.js:1011-1020, server/reviewTriage.js:205-229 (MODEL_POINT_EPS, stripesMatchSeed, computeDrawnBy), server/auth.js:99-103, server/index.js:1300-1307, server/inference/train.py:289-304 (gate), :768 (call).
How likely is "nearly unedited"? Since the tag started (16 Oct) there are 70 labels with a seed. 50 of 70 sit within 0.3% of the seed (mean curve distance < 0.003, no stripe added or removed). Only 15 of those 50 were tagged model (10 Alex-approved, 5 unapproved and excluded). 21 were tagged admin, 8 user, 3 approved and would train. Alex is the only person in that sample; a contractor in quick-confirm mode would look the same. computed from labels-live.jsonl (seedCorrection.meanCurveDist)
Count of reviewer saves so far 0. Every label saved since the role shipped (3 Nov 17:29 ET) is either Alex (9 saves, 4-6 Nov, on customer photos; 3 tagged model and approved by him) or a customer saving their own photo (10). No label exists whose saver is neither Alex nor the photo's uploader. I cannot read users.role (prod SQLite), so this is inferred from saver vs uploader. labels-live.jsonl + metas.json
How many of those are byte-identical to their seed? n/a (0 reviewer saves). For Alex's 9: 3 tagged model (point-identical), 6 changed. same
Is anything already trained on this? No reviewer save exists, and the latest dataset on disk (ds-v31, v43, built 20 Oct) predates the role. But the dataset builder strips drawnBy, approvedBy and customerReviewStatus (ds-v30/ds-v31 labels.jsonl carry only id, imageRef, imageSize, sailType, source, stripes; 0 of 5,962 records have those fields). train.py gates read those fields, so on a dataset built that way every gate is a no-op. The builder (build_v30_labelaudit_dataset.mjs) is not in the repo. A dataset rebuilt from R2 after a reviewer starts saving would train their saves unfiltered. ds-v31 already holds 4 labels now pending and 6 now rejected (some were relabeled after the build). DevCaches/rawkp-l-v43/stage/ds-v31/labels.jsonl, DevCaches/mirrorfix/stage/ds-v30/labels.jsonl

Remediation proposal (not applied; nothing in prod was changed). (1) Re-seed the 372-photo queue from the live v39 chain before the reviewer works on it: bump the seed key to include the stripe-model version (MODEL_VERSION or the sidecar's reported version, plus SAM3_CHECKPOINT_REVISION) so a model change invalidates the cache, then recompute the ~335 stale seeds on the serverless path; at minimum drop the 200 seeds computed before 21 Oct. (2) Tag, do not block: make every label saved by role:"reviewer" carry drawnBy:"reviewer" and trainingEligible:false until Alex approves it (same idea as the customer pending stamp), and treat a save within 0.01 of the seed (not 0.002) as model, so a nudged copy of a stale seed cannot train. (3) Make the dataset builder copy drawnBy, approvedBy, customerReviewStatus and meta.pipelineRecipe into labels.jsonl so the train.py gates actually run, and add a check that fails the build if those fields are missing. Reviewers can keep working meanwhile: nothing trains until a dataset is rebuilt.

C10: labels saved by the 29 Sep labeller bug

Question Answer Evidence
What was the bug / fix? The editor did not reload an already-labeled photo's saved stripes; it showed a fresh model prediction and any save (boom, outline, head) overwrote the human stripes with the prediction. Fixed in b1d1f45d (28 Sep 20:01 ET = 29 Sep 00:01 UTC): fetch the persisted label on open, label beats seed. Companion fb1dd5bf (19:51 ET) stopped the same overwrite for the frozen 332. Outline mode landed 18:57 the same evening. git show b1d1f45d fb1dd5bf
Affected window From when the main editor first pre-seeded from the model while allowing reopen (App.tsx fetchSeed present by 29 Jul, boom-blitz routes from 6 Jul) to 28 Sep 20:01 ET (plus however long until the labeler deploy; the Pages deploy time is not in git). Signature hits run 7 Jul to 24 Sep. labels-live.jsonl
How many? A saved line cannot be proven bug-made, only seed-identical. Signature = created before the fix, has a seed, stripes within 0.003 of it, none added or removed. 545 labels match; 156 of those also have a boom or outline (the bug trigger); 136 / 44 are near-exact (< 0.0002). Some are genuine "accept the model's line" saves, so these are upper bounds. labels-live.jsonl
Tagged? No. drawnBy only exists from 16 Oct (3daf5b3d). Legacy records "are untouched" by design (train.py:293-295). train.py:289-304
In training? Yes. All 545 are in ds-v31 (the v43 training set); 156 boom/outline-signature: 151 as app, 5 as app-live-holdout (test only). 44 near-exact ones are all app and train. The 5 holdout ones are excluded from training by the frozen list. ds-v31/labels.jsonl ∩ labels-live.jsonl
Suggested follow-up Add a drawnBy:"model-suspect" backfill for the 156 (or 545) and drop them from the next build; Alex reviews the 44 exact ones first.

C9: how a hand label becomes 8 training points

Question Answer Evidence
Rule today train.py --keypoint-sampling raw (what every recent run passes, including v37 to v43 and the v39 launcher pod_v39_rawkp_l_full.sh:107): linear arc-length resample of the drawn polyline to exactly 8 points, endpoints kept, no spline, no smoothing. Before that, the points are flipped to ascending x if drawn right to left. Bbox from min/max of the points. server/inference/train.py:463-466 (ascending-x flip), :481-483 (sampler pick), :618-646 (_resample)
Is smoothing still applied anywhere? Not in a raw run. The train.py default is still fixed (default="fixed", train.py:702): cubic smoothing spline s = n*(0.004)^2 then cosine-spaced points (:519-534). Any launcher that omits the flag gets smoothing. The labeler, server save path and computeSeedCorrection do no resampling of human points. train.py:519-534,702
Which sources? Same sampler for all sources (archive, human-retrace, app, app-live-holdout). Points per stripe in ds-v31: archive median 9 (22% exactly 8), human-retrace 9, app 8 (51% exactly 8). So about half of app stripes and 78% of archive stripes are actually resampled. ds-v31/labels.jsonl
The 20 Aug 3.05-draft-points claim Consistent with the code comment: the 17 Jul offline audit found the smoothing step alone added 2.4 pt draft / 3.3 deg entry / 3.1 deg exit at 8 keypoints (train.py:477-480). raw was added in 90ced938 (26 Jul).
Catch for v39 v39 is a fine-tune of v33-soup, which was trained on the fixed (smoothed) targets; only the last fine-tune used raw targets (pod_v37_rawkp.sh:4-7 says v29 and v32-bsr used fixed, and that the bottom-stripe target was flattened: perpendicular error 0.228 vs 0.042 %diag). So v39's weights may still carry smoothing bias from the inherited weights. DevCaches/rawkp/stage/pod_v37_rawkp.sh:1-8

C12: missed stripes below the confidence cut-off

Question Answer
Count Cannot be answered from stored data. I found no stored raw-candidate dump for v39 on the 332: R2 predictions/ holds 1,957 memos, all from v13 / v26 (940 v26, 532 v13, 462 with no model field) and hold final stripes only; conditioned/deep20-candidate-dump.jsonl is the old deep20 finder; server/inference/runs/ has no v39 run.
Prior measurement (not mine, v29 era, from the 15 Oct memory note) Of 26 missed bottom stripes on the holdout, 19 had no raw candidate even at conf 0.01 (model-side miss); the other 7 were found but dropped by filters. Stale for v39.
What closes it One pod run: v39 (ensemble primary) on the 332 with SIDECAR_CONF=0.001, dump every raw candidate (points, conf), match to the key by curve distance, list for each of the current 15 misses the best candidate's confidence and which rule dropped it. CANDIDATE_MIN_CONF and the stripe trace (cv.js) already support the "deep pool, select as prod" setup. About 10 minutes on the smallest GPU. Say go and I will launch it on the SAILSCAN RunPod account.

C13: boom and twist

Question Answer Evidence
Boom-angle error on the 44 photos with a boom label Stored run boom-holdout-20260928 (the boom model of 28 Sep, not re-scored on v39 / the 21 Oct boom split). 44 labelled booms, 41 detected (recall 93.2%), 3 missed (2 headsails, 1 mainsail). Angle error: median 1.2 deg, mean 5.0 deg, p90 11.6 deg, max 36.5; 15 over 2 deg, 10 over 5, 7 over 9, 5 over 15, 1 over 30. Resulting boom-relative twist error: median 1.8 deg but mean 13.2 deg, p90 38.7 deg; 13 of 41 over 5 deg, 10 over 10. False-boom rate 12.7% (8 of 63 labelled no-boom, all mainsails). So "typical error about 9 deg" (16 Aug) is not what a typical photo sees; a handful of bad booms drive the mean. server/inference/runs/boom-holdout-20260928/report.json, false-boom-by-sailtype.md
How often does prod show twist from an outline-fallback boom? Before 21 Oct: very often. Since 21 Oct: essentially never. Stored seeds before BOOM_STRICT: 2,372 of 2,769 boom lines (86%) were outline-fallback with confidence 0 (a frame-edge guess, never a detected boom). BOOM_STRICT=1 shipped 21 Oct (2f55f7a3) and is live on Railway (also BOOM_MIN_LEN=0.15); seeds since then: 1 outline-fallback in 204. t47/seeds-boom.json, server/pipeline/stripeFilters.js:966-976, server/pipeline/modes/cv.js:816-843, Railway vars
What happens now with no boom? Since 21 Oct, mainsail seeds: 73 real boom, 86 no boom (54%), 1 outline-fallback. A null boom makes twist fall back to the bottom stripe (stripeFilters.js:964-965 comment). Of the 73 real booms, median confidence 0.615; 18 under 0.5, 52 under 0.7. Headsail seeds never use a boom (40 no-boom since 21 Oct). Caveat: these are cached seeds (uploads and labeler opens), not every customer analyze. t47/seeds-boom.json + metas.json sailType

Things I could not verify