Concurrency: How many requests are in flight (queued + running) at the moment.
Latency: How long a task takes from start to finish.
Throughput: Number of tasks completed per second.
SLA (Service Level Agreement): The performance target the system is expected to meet.
Example
100 requests complete in 10 seconds: Throughput = 10 requests per second.
Each request takes an average of 0.5 seconds: Latency = 0.5 seconds.
On average, 2 requests are running and 3 are queued: Concurrency = 5 requests.
Let's say users need responses within 0.5 seconds. We want to select a GPU or hardware system, such as a T4, A100, or RTX 4090, that can handle concurrent requests while giving us the best balance between latency and throughput. The goal is to meet the latency requirement, maximize throughput, and keep the cost per request low.
So, the constrained optimization problem here is:
Maximize Throughput -> Minimize Cost
Latency (p99) ≤ User SLA Target (< 500 ms)
In the next note, we will discuss how increasing the batch size can increase the latency per user and also increase throughput. There is a tradeoff.