Software and engineering
Little's law
In any stable system the average number of items inside equals the arrival rate times the average time each spends inside: L = λW.
John D. C. Little, "A Proof for the Queuing Formula L = λW", Operations Research 9(3), 1961.
L, the average number of items in a system, equals λ, the average rate at which they arrive, times W, the average time each one spends inside. Little proved in 1961 that this holds for any stable queue whatever the arrival pattern, the service times or the order of service; it is long-run accounting, not a model. For an engineer the handy form is concurrency = throughput × latency. A service taking 200 requests a second at 50 milliseconds each has 10 requests in flight on average. Read the other way, 32 workers that each take 40 milliseconds per request can sustain at most 800 requests a second; push more in and the queue, and W, grow without bound.
Little's fifty-year retrospective in 2011 collects uses in operations and in computer architecture, where the same identity relates outstanding memory requests to bandwidth and latency. The hospital case shows the mechanism: an emergency department's length of stay is W, patient arrivals are λ, patients in the department are L, so staffing follows from arrivals and the measured patients per hour each doctor handles. In engineering it sizes thread pools, connection pools and API concurrency, and it works as an alarm: if traffic is flat and the in-flight count climbs, latency is rising somewhere, because the law allows no other reading.
It is a statement about long-run averages in a stable system, and it is silent about everything else. It says nothing about the distribution, so a fine average hides a bad 99th percentile, and nothing about what happens near saturation, where waiting time grows steeply as utilisation approaches 100 percent; the hospital rule of thumb adds 10 to 20 percent staff above the law's minimum for that reason. Measured over a finite window it is biased by the items present at the start and end, which Song-Hee Kim and Ward Whitt showed how to correct in 2013 using 18 weekdays of call-centre data. And all three quantities must use the same definition of 'in the system', or the identity fails for a reason that has nothing to do with queues.
Example
In his 2011 retrospective in Operations Research, Little describes how Mark Harris of TeamHealth, a company that ran more than 500 hospital emergency departments, used the law to staff them. Length of stay is W, the arrival rate of patients is λ and the number of patients in process is L, so the minimum staff follows from arrivals divided by the measured patients per hour each doctor handles: if a doctor treats 2.5 patients an hour and 10 patients arrive each hour, 4 doctors is the floor. Harris added 10 to 20 percent above that minimum as a rule of thumb, because queues with variable arrivals blow up as utilisation nears 100 percent, with a typical target of keeping stays under three hours during the busiest eight hours of the day.
Where it applies
Size pools and large language model (LLM) API concurrency from the law: needed concurrency = requests per second × seconds per request. If a model call averages 4 seconds and you expect 50 a second, you need 200 in flight, and a provider limit of 100 concurrent calls caps you at 25 a second whatever else you tune. If in-flight requests climb at constant traffic, latency is rising; the law is the alarm.
Think about it
For the queue you care about most, which two of L, λ and W do you measure, and does the third match what you believe?
Applied it to something real? Log it.
Test yourselfA service handles 400 requests a second at 25 milliseconds average latency. How many requests are in flight on average, and what happens to throughput if latency doubles under a fixed concurrency cap?
10 in flight (400 × 0.025). Under a fixed cap L, doubling W halves the throughput the cap allows, because λ = L / W.
Related models
- Bottlenecks and the theory of constraintsA system's throughput is set by one constraint; improving anything else is wasted effort until that constraint is fixed.
- Stocks and flowsA stock is what has accumulated; flows are the rates that change it. The stock moves only as fast as inflow minus outflow.
- Amdahl's lawSpeeding up one part of a job is capped by the share of time that part takes; whatever you leave untouched sets the ceiling.
- Redundancy and slackSpare parts and spare capacity look like waste until the day they are the only thing between a fault and a failure.
- Averages hide the spreadA mean describes a group, not any member of it; without the spread you cannot tell whether anyone is near it.