A reviewer on my generator build said 'your model is 90% confidence on 10% of the data' and I've been arguing about it since | AI Innovations - Opinions Matter
A reviewer on my generator build said 'your model is 90% confidence on 10% of the data' and I've been arguing about it since
Back in March I put up a small open source model that labels parts in warehouse photos. A guy named Dev on the repo wrote that line in a comment and honestly it stung, but he was right, my training set was 400 images from one warehouse in Dallas. I cut the confidence threshold way down and added 1,100 more shots from two other sites, and the false labels dropped a lot. Now I'm stuck wondering if that was the right fix or if I just hid the problem. Should you tune the model to match the data or go get more data to match the model? Curious where people land on this.
Cutting the threshold was the smarter move and grabbing more data just papered over it. A high threshold means the model already knows when it's unsure, so forcing it to speak up on the rest is just inviting bad labels into your pipeline. More data from new sites helps, sure, but it doesn't fix a model that was trained to sound sure when it wasn't.