Latency vs Throughput: What Matters for Inference?
Introduction When deploying a machine learning model, two performance metrics deserve particular attention: latency and throughput. Although they are closely related, they measure different aspects of an inference system. Latency…