← All reports

2026-11-09

SailScan accuracy deep dive: where we are, do we need more data, and what to do next

For: Nate and Skippy From: Alex Date: 9 Nov 2026 Built from: - the Boats thread, 27 Oct to 9 Nov - the 27 Oct audit and handoff - the 2 Nov deep dive - a fresh search for anything released since then

1. Short version

  1. Score: 317 of 332 test photos with every stripe right (95.5%), after the duplicate-stripe fix on 2 Nov. Up from 313 on 27 Oct and 287 on 13 Oct.
  2. The 332 test set is getting too easy, and too unlike real use. Customer photos are harder: - In my blind check of 30 recent customer sails, I picked today's version 17 times and the new SAM 3 version 4 times, with 3 ties and 6 where both were wrong. - The misses on customer sails are duplicate stripes, stripes stopping short of the luff and leech, and missing stripes. - We need a customer test set, and to score on it.
  3. Do we need more data? Not more of the same. Every "add more data" retrain has lost: - six in the September bake-off - 293 vs 309 on 20 Oct, when near-copies made it worse

What we need is: - one labelling rule across all labels (bottom edge of the tape) - clean labels - a customer test set - targeted hard cases from failed and abandoned customer scans, which we now keep 4. New tech since 2 Nov is thin. Almost nothing new has come out for thin-line or keypoint precision. The useful item is an RF-DETR update (1.12, 5 Nov), which gives an uncertainty for every point and fixes a 1-pixel flip bug. The bigger gains are still the ideas in the 2 Nov deep dive, most of which haven't been started. 5. Biggest risk right now: a Fiverr reviewer is about to correct 349 customer photos. If they don't follow the exact bottom-edge rule, we add exactly the kind of label noise that caps the strict score. They need a one-page labelling guide and a test before they start (section 5).

2. What happened 27 Oct to 9 Nov

Date What Result
27 Oct RTMO (Apache) licence test Live 313. RTMO-M in place of v39: 289. RTMO-M plus RF-DETR: 291. On its own, before clean-up, RTMO finds lines better than v39 (291 vs 276). It loses in the clean-up steps, which were tuned to v39. Its starting weights were pretrained on research-only photo sets, so it needs a legal read.
27 Oct ONNX switch Live. Models 25 to 40% faster, same lines.
28 Oct SAM 3 on vs off Same stripe count either way (314/332). With SAM 3, lines sit about 9% closer to the human traces, but entry and exit angles get worse.
28 Oct SAM 3 "line mix" (SAM 3's middle, the model's ends) Lines 27.6 vs 28.0 from the human traces (Skippy's distance measure; please state the unit). Entry and exit back to normal. Fewer rule breaks (2.7% vs 3.3%). Draft error 3.42 vs 3.37. On 33 customer scans it was mixed. Not switched on yet.
28 to 29 Oct Warm SAM 3 when the app opens Live. About 40% of scans had been waiting 16 to 39 s on a cold SAM 3.
29 Oct Keep every scan unless the user opts out In 1.0.13 (live). Before this, only about 120 of about 1,300 scan photos were kept with a user. Abandoned scans are our hardest cases, and now we keep them.
29 Oct My blind check, 30 customer sails, new SAM 3 version vs today Today 17, new 4, ties 3, both wrong 6. Not shipped.
29 Oct Stripe spacing rule from me Stripes sit at about 25/50/75% of height, and race mains often have a 7/8 stripe at about 87.5%. Now used to pick between duplicate lines.
2 Nov Duplicate-stripe fix 313 to 317 of 332, none worse. I picked it 6 to 1. Live.
3 Nov Lens data and phone tilt captured with every photo Built, in the next TestFlight.
3 Nov Chris's shape database About 250 jib and 230 main measurements with target ranges by wind speed, loaded into a sandboxed Elvstrøm demo.
4 Nov Blender pilot: 50 synthetic sails Not real enough yet: too-clean cloth, flat light, empty scenes (no boat, crew, sheets, water).
5 Nov Reviewer login for a Fiverr helper Can only fix lines in Combined Review on user photos (349 waiting). Every save is logged under their name.

Not reported yet from the 27 Oct handoff: - the label rule check (A1: middle vs bottom edge split) - the clean-up of 421 duplicates and the unproven archive labels (A2) - RTMO on clean data (A3) - the flip test (A4) - the confidence cut-off test (A6) - the straightened-strip refiner (2 Nov deep dive, A1) - the per-step timing table (S1)

3. Do we need more data?

No, not more of the same. The evidence:

Fixing our answer key moved our score the same way in October.

What we do need:

  1. A customer test set. The 332 are mostly archive photos and now score 95.5%. Customer photos are harder (section 1). - Build a second frozen set of about 150 to 200 recent customer scans, never trained on. - Label every one carefully to the bottom-edge rule, and checked by me. - Score every change on both sets from now on, normal and strict.
  2. One labelling rule. Bottom edge of the tape, all the way to the luff and leech. Check it and convert (27 Oct handoff, A1). Do the test set first, because that's where it moves the number.
  3. Targeted hard cases, not volume. - The scans users abandoned or corrected, and the photos that fail the strict score: tiny top stripes, faint bottom stripes, seams and graphics, lines stopping short. - Label those carefully and add them. - No near-copies of the same shot.

How to prove it. Run these on the frozen sets, 2 to 3 training runs each. They tell us whether we're limited by data, label noise, or the model.

Test How What it tells us
A. Learning curve Train on 12.5%, 25%, 50% and 100% of the clean data Flat from 50% to 100% means more data won't help
B. Bias vs scatter For every predicted line, the signed distance from the bottom edge of the tape Consistently above or below: a convention or coordinate bug, fixable. Centred but spread out: noise or precision
C. Human ceiling Two people, plus one person twice, blind re-label 50 test photos; score human vs human If the model is already near the human-vs-human score, improve the labelling tool and rule, not the model
D. Train vs test Strict score on training photos vs test photos Low on both: model, resolution or label noise. High on train, low on test: more varied data would help
E. Clean-only training Train only on approved, bottom-edge, de-duplicated labels Matches or beats the full set: we're noise-limited
F. Failure counts Count training examples of each failure type Thin categories are where new labels pay off

4. New tech since 2 Nov (checked 9 Nov)

Very little new for thin-line or keypoint precision; most releases were vision-language models, auto-labelling and video models. What matters:

  1. RF-DETR 1.12.0 (5 Nov, Apache). Checked against the official changelog on 9 Nov: - Per-point uncertainty:
    • predict() on a keypoint model returns key_points.data["covariance"], an uncertainty for each point. 1.12.0 fixed it so it's always present (empty frames used to leave it out).
    • Use it to weight the combined finder (2 Nov deep dive, A3), and to replace the fixed 0.66 cut-off for adding RF-DETR lines.
    • 1-pixel flip fix: keypoints flipped by the torchvision augmentation path "now land where the flipped image content is". Visible flipped keypoints move 1 px.
    • Our RF-DETR training used flips, so those labels were 1 px off.
    • Check our own YOLO training, ONNX pre and post processing, and the planned flip test for the same kind of bug.
    • mAP fix: keypoint mAP and mAR with mixed keypoint counts were fixed, and "checkpoint selection can change on affected datasets". The checkpoints picked in our earlier RF-DETR runs may not have been the best ones.
    • Static INT8 export for ONNX and OpenVINO is now built in. This is useful for speed test S4 in the 27 Oct handoff.
    • Earlier fix (1.11.1, 30 Sep): moved "the keypoint fp16 clip fix onto the correct Lightning hook". That suggests fp16 keypoint training before 1.11.1 didn't get the fix.
    • Check which RF-DETR version and precision our runs used. If any trained in fp16 on a version before 1.11.1, or with the flip bug, rerun RF-DETR on 1.12.0 before treating "RF-DETR can't beat v39" as settled.
    • Links: https://rfdetr.roboflow.com/develop/changelog/ and https://rfdetr.roboflow.com/latest/learn/run/keypoints/
  2. Coordinate bias audit: - Resize, flip and letterbox steps that mix up pixel-centre and pixel-corner positions add a steady 0.5 to 2 px error ("Unbiased Data Processing", CVPR 2020). - Cheap to check across our training code, ONNX pre and post processing, and the SAM 3 hand-off. - Combined with test B above, this tells us whether "a few pixels off" is a bug or a limit. - Link: https://arxiv.org/abs/1911.07524
  3. TPose (NeurIPS 2026): - Reframes thin-object detection as top-down pose estimation: crop each object, then find its points. - Supports the straightened-strip refiner idea, but there's no code, so it's not something to adopt. - Link: https://neurips.cc/virtual/2026/179344
  4. YOLO27: announced 13 Sep, not released, licence unknown. Watch it, don't wait for it.
  5. Not useful to us: - Roboflow's October updates: auto-label with large vision models, which are already ruled out as tracers. - NVIDIA Cosmos 3 for synthetic data: the edges aren't exact enough for our labels. - SAM 3.1: built for video.

Everything in the 2 Nov deep dive still stands. The straightened-strip refiner, the combined finder, the joint stripe fit and the Apache model at the end are still the main bets, and none has been started.

5. The Fiverr reviewer (do before they start)

Adding 349 corrected customer photos is exactly the "right data" we want, but only if the lines are consistent.

  1. A one-page guide with pictures: - Bottom edge of the tape, not the middle. - Run the line all the way to the luff and to the leech. - One line per stripe. - Include tiny top stripes. - Don't label seams, battens or graphics. - Skip photos you can't read.
  2. A test first: they label 10 photos I've already labelled. Score them with the strict scorer, and only let them continue if they match me closely.
  3. Spot checks: I review 1 in 10 of their saves. Their saves are tagged with their name, so we can pull them if needed.
  4. Their saves count as reviewer, not my approval. They go into training only after my check, same rule as always.

6. Plan, in order

# What Why Effort
1 Build the customer test set (section 3) We're optimising on a set that's nearly solved Medium
2 Label rule check and conversion (27 Oct A1), test sets first Most likely cause of the strict-score gap Low to medium
3 Bias vs scatter (B) and the coordinate bias audit (section 4) Tells us if "a few pixels off" is a fixable bug Low
4 Fiverr guide and test (section 5) Protects the new labels Low
5 Clean the training data (27 Oct A2) and save every result Every retrain result is muddied until then Low to medium
6 Human ceiling (C) on 50 photos Tells us how high strict can realistically go Low
7 RTMO on the clean, single-rule data, with clean-up steps re-tuned for it. Get the legal read on its pretrained weights. It already finds lines better on its own (291 vs 276); also settles the licence Medium
8 Check the RF-DETR version and precision used (fp16 clip fix, flip 1 px fix, mAP checkpoint fix); retrain on 1.12.0 if affected The "RF-DETR can't win" result may not be a fair test Low
9 Straightened-strip refiner (2 Nov A1) Main bet for few-pixel accuracy Medium
10 Combined finder with per-point uncertainty (2 Nov A3, RF-DETR 1.12) Placement and extra lines Low to medium
11 Learning curve (A) and clean-only training (E) Settles the "more data" question with numbers Low (GPU time)
12 Plausibility check from Chris's shape database: reject or flag shapes outside the measured range for that sail type and wind Catches impossible lines Low

Rules, as always: - one frozen set per test, scored normal and strict - per slice - save every result - a person picks - nothing auto-promotes - nothing goes to users without my OK

7. One security item

A Google service-account private key (sailscan-mailer) was pasted in full in Boats on 9 Nov. Skippy said to rotate it after METS. Rotate it now. In the Google console, delete the key, create a new one, and send it to Nate directly, never in Boats.

Sources