Union.ai self-service deployment is now available on AWS Marketplace.

Learn more
GPU-utilization study · FSG-26-04

Reuse or Reload

A GPU-utilization study of batch LLM inference: keep the worker warm, or spin a fresh container per call.

Qwen2.5-7B on one NVIDIA L4 | 500 GSM8K questions, ten 50-question chunks | measured with nvidia-smi every 2 s
AbstractServing large-model batch inference under an orchestrator forces a choice: keep a GPU worker warm across many calls, or spin a fresh container per call. We run one identical vLLM workload — 500 GSM8K questions through Qwen2.5-7B on an L4 — two ways. With container reuse on a managed Union cluster the job finishes in 404 s; a no-reuse variant on open-source Flyte takes 1,665 s — 4.1× slower — and, measured with nvidia-smi, keeps the GPU actually computing only 16% of the run while a model sits resident 86% of it, where the reuse GPU, warm, holds a steady 100%.

Read the full paper (PDF) ↗

4.1×
slower without container reuse (1,665s vs 404s)
100%
sustained GPU utilization with reuse, once warm
10
model loads without reuse — vs. just 2 with it
3.3×
higher peak decode throughput with reuse
Key results

What the numbers show

Every chart below is drawn straight from the measured data in the paper — hover any bar or point for the exact value.

500 GSM8K questions — total wall-clockFig. 1 · wall-clock
0 s 463 s 925 s 1,388 s 1,850 s Union · reuse 404 s Flyte · no-reuse 1,665 s

Loading the model twice instead of ten times, and packing concurrent callers into 256-prompt batches rather than isolated 50-prompt calls, completes the identical job 4.1× faster.

Compute utilization + GPU memory over the runFig. 2a · Union · reuse

compute util GPU memory

One model load, then a steady 100% through batched inference — mean utilization over the whole run is 35%, rising toward 100% as more work amortizes that single load.

Compute utilization + GPU memory over the runFig. 2b · Flyte · no-reuse

compute util GPU memory

Each chunk loads its own copy of the weights, fires one ~33-second 100% burst, then releases the GPU — ten loads for ten chunks. Integrated, Flyte computes for only 16% of the run while holding a model resident 86% of it: almost always occupied, almost never working.

Same workload, two execution modelsTable 1
Metric Union · reuse Flyte · no-reuse
Wall-clock, 500 questions 404 s 1,665 s (4.1×)
Model loads for the job 2 10
Cold start, amortized / question ~0.5 s ~3.7 s
Effective batch size up to 256 50
Peak decode throughput 1,424 tok/s 433 tok/s
Time computing (util > 50%) 35% 16.1%
Time holding a model, idle 61% 86.5%
Flyte · no-reuse  1,665 s

See it for yourself

Spin up a real Flyte cluster in minutes and run the same workloads on your own hardware. No infra to provision, no YAML to write first.

Sign up

30 day free trial

Try the devbox