TensorFlow Serving ran circles around my Flask setup for a 40ms model
Last month I was serving a small image classifier through a Flask endpoint and it fell over at about 30 requests a second on a 4GB box. I swapped in TensorFlow Serving on the same hardware and it held steady past 300 a second, mostly because it was batching requests and keeping the model warm instead of reloading it on every call. Is there a reason people still spin up raw Flask for small models, or are we just doing it out of habit?