Ran into the same thing on a batch of brackets last year and never went back to conventional on aluminum. The finish pass just cuts cleaner because the cutter is biting into solid material instead of rubbing first, and on 6061 that difference shows up fast on the edges. Only time I switch back is on roughing with a worn setup where the backlash gets sloppy.

I heard a guy on a knife making podcast say he counts "one mississippi" out loud while the part goes in, and if he gets past three before the red fades he knows he dragged his feet. Sounds dumb but it gives you a beat to work with instead of guessing. The color thing gets me too, especially in a dim shop where everything looks dull orange anyway. One trick is glancing at the smoke or steam off the oil, when it really bites you get a fast puff instead of a lazy wisp. Might be worth running a few coupons at different counts and seeing where the scale stops flaking.