2026-11-13
Retrain batch 2: five new models with every known fix, against the live model
Verdict: the live model (v39) stays. On their own, two of the new models find a few more photos fully right than v39 does. But run through the full app pipeline, the one customers actually get, every new model gets fewer photos fully right than v39. The gap is larger than the luck-of-the-draw spread between retrains.
What we stacked
Every new model got all of these at once, which is the "best chance" you asked for:
- v39's own recipe: same model size, same image size, same length of training, same starting weights.
- The flip-order fix: stripe points stay in order when training flips a photo upside down. Without the fix, about 27% of stripes got scrambled; with it, 0%.
- Cleaner data:
- Exact duplicate photos removed: 1,009 extra copies, out of 4,986 training photos.
- Near-copies of the same shot kept together: 322 groups, so the model can't be checked on a photo it has nearly seen.
- The 14 wrong-view photos: excluded. None of them were in the training data anyway.
- Our own scorecard picks the checkpoint: the best moment of each training run is chosen by our line scorecard on practice photos, not by the training software's own number.
- Two genuinely different training runs per model, so we can tell a real gain from luck. The leading model got a third.
The five versions:
| Added on top of the clean base | |
|---|---|
| B1 | nothing (clean base only) |
| B2 | sail photos with no draft stripes, so it learns not to invent lines. Only 6 photos passed a full-size check, so this is a very small dose |
| B3 | bigger images (1280 instead of 960 pixels) |
| B4 | a short final pass on lines you drew or approved only |
| B5 | flipping photos upside down more often in training (half the time instead of 30%) |
Results
Model on its own (raw lines, 327 test photos, "all lines right"):
| Run 1 | Run 2 | Run 3 | |
|---|---|---|---|
| Live model v39 | 264 | ||
| B1 clean base | 261 | 260 | |
| B2 + no-stripe photos | 268 | 268 | |
| B3 + bigger images | 269 | 270 | 266 |
| B4 + final pass on your lines | 259 | 262 | |
| B5 + more upside-down flips | 268 | 263 |
Through the full app pipeline (the same steps a customer's photo goes through), out of 327:
| All lines right | |
|---|---|
| Live model v39 | 299 |
| B3, run 1 | 291 at 960 px, 285 at 1280 px |
| B3, run 2 (1280 px) | 289 |
| B2, run 2 | 291 |
| B2, run 1 | 286 |
The verdict in three lines
- On their own, B3 and B2 get 2 to 6 more photos fully right than v39, but that is inside the normal run-to-run spread, so it isn't a proven gain.
- Through the full pipeline, v39 beats every new model by 8 to 14 photos. That gap is real: on the photos where the two disagree, v39 is right most of the time.
- No new model goes live, and there is no vote this time, because none of them is close enough to v39 to need one.
Next step
Our best guess, not yet tested: the pipeline's later steps (the confidence cut-off and the second model it combines lines with) were set up around v39's output, so a new model loses some of its gain there. Next, re-tune those settings for the best new model (B3) on practice photos, never the test set, and run it through the pipeline again. If it still loses, the retraining ideas tried so far are done, and the effort moves to the pipeline itself.