LLM Serving: Continuous Batching, Distributed Execution, and Service-Level ObjectivesLLM 正式環境部署：批次處理、分散式推論與服務等級目標

This is the final article in a three-part series. The first article deconstructed the mathematical anatomy of a decoder-only transformer. The second article zoomed in on hardware execution, exploring how KV caching, FlashAttention, and model compression tame the strict memory-bandwidth bottlenecks of the generation loop. This article focuses on running LLMs in the production system.

But a highly optimized model is only half the battle. In a vacuum, generating tokens is an algorithmic challenge; in production, it is a distributed systems challenge.

This final article steps outside the model's architecture. We focus on the infrastructure required to turn a raw computational graph into a scalable, highly concurrent service. We will explore how to dynamically schedule continuous request queues, distribute massive models across multi-GPU clusters, and navigate the unavoidable tension between throughput, latency, and hardware costs to guarantee strict Service-Level Objectives (SLOs).

The content is structured as:

1. Request orchestration: Batching strategies
2. Scaling execution: Model parallelism
3. The core tension: Throughput, latency, and cost
4. SLO thinking and serving frameworks
5. Infrastructure and system design reference
6. Conclusion

1. Request orchestration: Batching strategies

Unlike traditional microservices where inputs are uniform (predictable data types or fixed-schema JSON payloads), LLM inference workloads are highly heterogeneous:

• Asynchronous arrival: Users and automated systems send prompts at entirely unpredictable intervals.
• Variable lengths: One request might be a five-token classification prompt, while the next requires processing a 10,000-token document.

If a GPU processes these requests sequentially, it sits mostly idle, starved for compute. To maximize hardware utilization, serving engines group multiple requests into batches. However, because requests do not finish at the same time, how the engine forms these batches dictates the system's performance.

The Evolution of Batching

In production, a batch usually contains independent requests from multiple users. Batching never mixes the actual text of different users; it simply groups their independent tensor operations for parallel execution on the accelerator, with each request maintaining its own sequence state, decoding parameters, and KV cache.

Strategy	Mechanism	Production Limitation
Static batching	The server waits for a predefined number of requests, pads them to the length of the longest prompt, and executes them together.	Severe latency degradation; short sequences remain trapped waiting for long sequences to finish.
Dynamic batching	The server opens a brief time window (e.g., 50ms) to collect incoming requests into a batch before execution.	Improved throughput, but batch membership still only changes at request boundaries.
Continuous batching	The scheduler operates strictly at the token level. It ejects finished sequences and pulls new requests into the active batch at every single iteration step.	Requires advanced cache-aware memory management (like PagedAttention) to function.

Continuous batching (iteration-level scheduling) is the modern industry standard. Because finished sequences leave immediately and waiting requests take their place in the very next decode step, GPU downtime drops to nearly zero. This is the key throughput innovation in vLLM and TGI. GPU utilization goes from ~40-60% (static) to ~90%+ (continuous).

Time (Iteration Steps) ──────────────────────────────────────►

Step 1: [ Req 1 | Req 2 | Req 3 | Req 4 ]
Step 2: [ Req 1 | Req 2 | Req 3 | Req 5 ]  ← Req 4 finishes, Req 5 enters instantly
Step 3: [ Req 1 | Req 6 | Req 3 | Req 5 ]  ← Req 2 finishes, Req 6 enters instantly
Step 4: [ Req 1 | Req 6 | Req 7 | Req 5 ]  ← Req 3 finishes, Req 7 enters instantly

Result: Zero wait time at request boundaries. The GPU is always fully utilized.

Advanced Scheduling Policies

Beyond the core batching loop, schedulers must balance fairness, priority, and physical memory limits using advanced policies.

Handling Memory Exhaustion: Preemption

If active requests generate longer responses than anticipated and the GPU's KV cache pool reaches 100% capacity mid-generation, the scheduler cannot simply crash. It must trigger preemption. The engine pauses the lowest-priority request, evicts its KV cache blocks from the GPU's HBM to the slower host CPU memory (swapping), and allows the remaining requests to finish. Once VRAM frees up, the evicted request is swapped back in (or its prefix is recomputed) to resume decoding.

Mitigating Resource Contention: Chunked Prefill

When a massive new prompt arrives, processing its entire prefill phase in one go can stall the iteration loop, causing latency spikes for all other users currently in the decode phase. Chunked prefill breaks that giant prompt into smaller segments, spreading the heavy matrix multiplication across multiple scheduler steps so it does not starve the concurrent decode loop.

Bypassing Compute: Prefix-Aware Routing

In complex production environments—especially multi-agent backends or applications with massive, standardized system prompts—many requests share identical context. Prefix-aware scheduling (Prompt Caching) maps requests with shared prefixes to the exact same physical KV cache blocks in memory. Instead of executing the compute-heavy prefill phase for every request, the engine computes it once, effectively turning a slow prefill into an instantaneous decode step for all subsequent users.

Ultimately, maximizing raw throughput is not always the correct business objective. A highly aggressive batching strategy will maximize device utilization, but it heavily increases queue times and the duration of each decode iteration.

2. Scaling execution: Model parallelism

What happens when an uncompressed 70-billion parameter model (roughly 140 GB in FP16 precision) physically cannot fit into a single 80 GB GPU? Or when the KV cache for a massive continuous batch exceeds a single accelerator's memory? The model must be partitioned across a distributed cluster.

This is called Model Parallelism. Distributing an LLM solves memory and compute constraints, but introduces a massive new bottleneck: network communication overhead.

Technique	Partition Strategy	Main communication	Primary Use Case	Hardware Limitation
Data parallelism	The entire model is replicated across different GPUs; user requests are divided among them.	Request routing; little per-token model communication	Scaling raw throughput for models that fit on a single GPU.	Does not solve VRAM capacity limits if the model is too large.
Tensor parallelism (Vertical Split)	Shards individual weight matrices (and attention heads) across GPUs within the same layer.	Collective communication at every layer boundary.	Fitting wide models and reducing single-request latency.	Requires ultra-high-bandwidth interconnects (e.g., NVLink) to prevent latency spikes.
Pipeline parallelism (Horizontal Split)	Shards the model sequentially; GPU A takes layers 1-16, GPU B takes 17-32.	Point-to-point passing of intermediate activations between stages.	Fitting extremely deep models across multiple physical server nodes.	Introduces "pipeline bubbles" (idle time where GPUs wait for the previous stage to finish).
Expert parallelism	Distinct experts in a Mixture-of-Experts (MoE) model reside on different GPUs.	Token-routing all-to-all to the right expert GPU	Serving massive MoE architectures.	Creates highly complex, bursty network routing overhead.
Context parallelism	A massive prompt is chunked, and different GPUs process different sequence segments.s	Attention-specific collective communication.	Processing extreme context windows (e.g., 1M+ tokens).	Substantial network traffic during the prefill phase.

Large production deployments compose these dimensions together (e.g., using 3D parallelism: Tensor parallelism + Pipeline parallelism + Data parallelism). The governing rule of distributed inference is topology-aware placement: communication-heavy partitions (like Tensor Parallelism) must remain on the fastest available local interconnects (like intra-node NVLink), while less chatty partitions (like Pipeline Parallelism) can span across standard network boundaries (like inter-node InfiniBand). (The foundational approach to this topology is detailed in Megatron-LM).

When adding additional GPUs fails to improve token latency or throughput, engineers must measure collective time, pipeline bubbles, and interconnect saturation before adding more hardware. Parallelism solves compute constraints by accepting communication overhead; physical topology determines whether that trade is worthwhile.

3. The core tension: Throughput, latency, and cost

In LLM serving, an unqualified "tokens per second" is a meaningless metric.

The fundamental physics of an inference server are dictated by a single lever: Batch Size. Pumping more requests into a continuous batch utilizes the GPU's compute cores more efficiently and raises the system's aggregate throughput. However, because the hardware must process more sequences concurrently, the time it takes to calculate each individual token step increases. This drives up Inter-Token Latency (ITL), making the model feel sluggish to the end user.

Therefore, measuring performance requires explicitly defining the phase, the load, and the perspective of the measurement.

Metric	Definition	Perspective
Time to first token (TTFT)	Request arrival to the first streamed token.	User experience (Waiting for the system to "think")
Inter-token latency (ITL)	Gap between consecutive streamed tokens.	User experience (Reading speed)
Time per output token (TPOT)	Average generation time after the first token.	System average
End-to-end latency	Request arrival to the final generated token.	Total transaction time
Input throughput	Prompt tokens processed per second.	System capacity (Prefill bound)
Output throughput	Generated tokens completed per second.	System capacity (Decode bound)
Request throughput	Requests completed per second.	Business capacity

Increasing concurrency improves aggregate throughput until the memory, compute, scheduler, or interconnect saturates. Simultaneously, latency rises as requests queue and batches grow. The result is a Pareto frontier—there is no single optimal configuration, only a deliberate trade-off between maximizing hardware utilization and preserving user experience.

The Anatomy of a Valid Benchmark

Because of this tension, a meaningful production benchmark cannot just be a single throughput number. It must explicitly state:

• Prompt and output-length distributions
• Arrival rate and concurrency levels
• Streaming and decoding settings (e.g., greedy vs. beam search)
• Model architecture, precision, and KV cache format
• Hardware topology and interconnect bandwidth
• Serving engine version and scheduler configuration
• Latency percentiles (e.g., p90, p99), never just averages

When a live system struggles, the runtime symptoms dictate the necessary interventions:

Observed Symptom	Likely Bottleneck	Candidate Interventions
Slow token streaming (High ITL)	Decode bandwidth limit, oversized batches, or slow distributed communication.	Cap the continuous batch size, apply KV-cache quantization, evaluate speculative decoding, or inspect interconnect traffic.
High p99 TTFT with an acceptable mean	Severe queuing or head-of-line blocking during prefill.	Implement strict admission control, fair scheduling, and chunked prefill.
Low accelerator utilisation	Insufficient work or scheduler overhead gaps.	Ensure continuous batching is active, increase the token budget, or utilize CUDA graph capture.

Calculating True Cost

For a fixed deployment, the baseline infrastructure cost is straightforward:

$$\text{Cost per 1M output tokens} = \frac{\text{GPU count} \times \text{Price per GPU-hour}}{\text{Output tokens per hour}} \times 10^6$$

However, the denominator (output tokens per hour) must be measured at a declared load where the service still meets its latency SLO. Driving a server beyond its sustainable arrival rate might inflate the aggregate throughput number, but it causes queuing latency to grow without bound.

Throughput that violates your user-facing latency requirements is useless. Furthermore, a complete application-level cost must also account for retries, structured-output parsing failures, and human review overhead caused by aggressive model compression or optimization regressions.

4. SLO thinking and serving frameworks

A Service-Level Objective (SLO) translates runtime behavior into a strict application contract. In production, engineers cannot simply mandate, "We want the model to be fast." That is an unmeasurable aspiration. Instead, an SLO defines exactly what "fast" means, under what precise conditions, and the statistical reliability with which the system promises to deliver it.

For example:

For interactive prompts of at most 2,000 tokens:
  p95 TTFT <= 800 ms
  p95 ITL  <= 60 ms
  error rate < 0.5%
measured over rolling 30-minute windows

The exact thresholds depend entirely on the product. The critical properties of a valid SLO are a tightly defined workload, a specific percentile, a hard threshold, and a measurement time window.

The Tail Latency Trap

"Average latency" is a massive trap in LLM serving. Because workloads are so variable (as we discussed in batching), nine users might experience lightning-fast responses while the tenth user gets stuck behind a massive 100,000-token prefill job, waiting 45 seconds for a single word.

If you average those ten users, the system looks completely healthy. But in reality, 10% of your users are experiencing an utterly broken product. This is why SREs track tail latency (p90, p95, p99). Tracking the p99 forces the engineering team to look at the absolute worst-case scenarios and optimize the system for the users waiting the longest.

Defensive Engineering

To successfully meet these SLO promises, you must aggressively defend your servers from being overwhelmed. GPU compute and KV cache memory are highly constrained resources. An enterprise-grade deployment requires strict operational safety mechanisms to prevent malformed or abusive requests from degrading the experience for everyone else:

• Bounded Contexts: Hard-cap the maximum allowed prompt and output lengths to prevent cache exhaustion.
• Admission Control & Load Shedding: Apply strict queue timeouts, enforce per-tenant rate limits, and proactively return HTTP 429 Too Many Requests before the service exhausts its VRAM.
• Prompt Validation: Reject malformed inputs or unsupported modalities at the API gateway before they are ever allocated accelerator work.
• Traffic Isolation: Physically or logically separate long-running background batch jobs from latency-sensitive interactive user traffic.

The Quality Constraint: If you aggressively optimize your engine (using extreme quantization or tiny beam widths) just to hit a 10ms ITL SLO, but the model begins outputting hallucinations or broken JSON, the infrastructure has failed. Latency and throughput metrics must always be balanced against the model's actual intelligence and structural validity.

Serving Frameworks

A basic Python script can execute an LLM one token at a time, but it cannot provide the cache management, continuous scheduling, observability, or defensive overload controls required by a shared production service. Modern serving frameworks supply these capabilities natively alongside hardware-specific execution optimizations.

Framework	Strong Fit	Notable Capabilities
vLLM	General high-throughput accelerator serving.	PagedAttention, continuous batching, prefix caching, quantization (AWQ/GPTQ), and distributed inference.
SGLang	Complex, multi-agent backend chatbot services and structured workflows.	RadixAttention for extreme prefix reuse, structured JSON output enforcement, and disaggregated serving.
TensorRT-LLM	NVIDIA-specific, hyper-optimized deployment.	used kernels, in-flight batching, paged cache, and maximum hardware utilization.
llama.cpp	Local, edge, CPU, Apple Silicon, and mixed-device inference.	GGUF quantisation, broad hardware backends, and lightweight server, continuous batching.

(Note: While Hugging Face TGI remains present in many established deployments, its current documentation describes the project as being in maintenance mode, recommending engines such as vLLM, SGLang, and llama.cpp for new architectural deployments).

Ultimately, framework selection should be based on the target model, hardware topology, structured decoding requirements, operational support, and your measured workload—never on a single, isolated public throughput benchmark.

5. Infrastructure and system design reference

This final reference section condenses the resources, workload phases, and operational signals needed to connect theoretical model behavior with live production performance.

5.1 Production Troubleshooting

Optimisation should begin with the observed constraint.

Symptom	Likely Bottleneck	Candidate Interventions
High TTFT for long prompts	Prefill compute or scheduling queue	FlashAttention, chunked prefill, prefix caching, prompt reduction, or disaggregated prefill.
Slow token streaming (High ITL)	Decode bandwidth or network communication	Smaller continuous batches, weight quantization, migrating to a GQA model, speculative decoding, or topology-aware parallelism.
Cache OOM or low concurrency	KV cache capacity	Paged cache management, models with fewer KV heads, shorter context limits, KV quantization, or adding device memory.
High p99 with an acceptable mean	Queuing or head-of-line blocking	Admission control, strict traffic classes, fair scheduling, and chunked prefill.
Low accelerator utilization	Insufficient work or scheduling gaps	Ensure continuous batching is active, increase the token budget, use CUDA graph capture, or tune the scheduler.
Quality regression	Compression or decoding change	Revert to higher precision, improve calibration data, lower sampling variance, and run task-level evaluations.

5.2 Infrastructure Memory Capacity

Component	Hardware Approximation
Model weights	Parameter count $\times$ stored bytes per parameter
KV cache	$2LBSH_{KV}d_{head}p$
Training gradients	Parameter count $\times$ gradient bytes
Optimiser state	Optimizer-dependent; commonly multiple values per parameter
Activations	Highly dependent on architecture, batch size, sequence length, checkpointing, and specific kernels

5.3 Workload Phases

Phase	Primary Work Unit	Persistent State	Frequent Bottleneck
Pretraining	Complete training batch	Parameters and optimiser	Compute, communication, activation memory
Prefill	Prompt or prompt chunk	Writes to KV cache	Compute and attention IO
Decode	One position per active sequence	Reads and extends KV cache	Weight/KV memory bandwidth and scheduling

5.4 Production Monitoring

Monitoring must connect demand, scheduler behavior, cache pressure, hardware saturation, user-visible latency, and model quality. No single metric explains the system by itself.

Monitoring Layer	Core Signals	What It Helps Answer
Demand and workload	Arrival rate, concurrency, queue depth, prompt/output distributions.	Is the overall traffic volume or request shape changing?
User-visible latency	TTFT, ITL, TPOT, and end-to-end latency at p50, p95, and p99.	Which inference phase is violating the user experience?
Throughput and scheduling	Input/output tokens per second, requests per second, active sequences, batched tokens per iteration.	Is the scheduler keeping the accelerator productively occupied?
KV-cache health	Utilization, allocation failures, preemptions, evictions, swapped blocks, and prefix-cache hit rate.	Is cache pressure limiting concurrency or causing latency spikes?
Accelerator and network	Compute utilization, VRAM usage, memory bandwidth, power, temperature, and interconnect traffic.	Is the hardware bottleneck rooted in compute, memory, or network?
Reliability	Cancellations, timeouts, overload responses (429s), OOM events, restarts, and error rates.	Is the service shedding or failing work safely?
Model and application quality	Task success, safety violations, structured-output validity, retries, and human-review rate.	Did an infrastructure optimization destroy useful model behavior?

Metrics should be strictly segmented by model version, serving configuration, workload class, and traffic priority. Engineers must use histograms and percentiles rather than averages alone, alert on SLO burn rates alongside leading indicators (like queue saturation), and relentlessly correlate performance changes with quality evaluations.

6. Conclusion

The complete dependency chain is now visible. Causal attention creates reusable keys and values; their mathematical shape dictates physical cache capacity; the growth of that cache drives paged memory allocation and continuous request scheduling; and rigorous production monitoring reveals exactly which resource or policy must be adapted to meet strict SLOs.

Ultimately, exposing a raw model checkpoint to the internet does not constitute a production service. To guarantee consistent, reliable behavior at scale, a release must be treated as a fully reproducible artifact. This requires locking down every moving part of the execution pipeline: not just the model weights, but the tokenizer version, chat templates, quantization schemes, the specific serving engine, hardware kernels, and generation hyperparameters.

Before routing live traffic to an endpoint, the infrastructure must be rigorously validated against the unpredictable realities of production environments. A robust serving layer must account for:

• Operational resilience: Managing cold-start warm-ups, absorbing massive traffic spikes without crashing, and executing graceful out-of-memory (OOM) recovery when the KV cache fills up.
• Edge cases and client volatility: Handling malformed requests, gracefully truncating prompts that exceed the maximum context window, and safely halting compute when a client rapidly disconnects or cancels a request mid-generation.
• Security and execution boundaries: Strictly enforcing structured-output constraints (e.g., guaranteed JSON schema adherence) and maintaining absolute multi-tenant memory isolation to ensure different users' data remains completely separate.nally, your offline evaluation suite must also run against the live, served API.

Finally, a successful deployment requires running your offline evaluation suite directly against the live, served API. Modifying the quantization strategy, updating a GPU kernel, or swapping the serving framework can subtly alter the model's output distribution—even if your core application code remains completely untouched.

The transformer's architecture and the infrastructure serving it are not separate concerns. A model is only as deterministic as the infrastructure serving it.

時間（迭代步驟）──────────────────────────────────────►

步驟 1：[ 請求 1 | 請求 2 | 請求 3 | 請求 4 ]
步驟 2：[ 請求 1 | 請求 2 | 請求 3 | 請求 5 ]  ← 請求 4 完成，請求 5 立即遞補
步驟 3：[ 請求 1 | 請求 6 | 請求 3 | 請求 5 ]  ← 請求 2 完成，請求 6 立即遞補
步驟 4：[ 請求 1 | 請求 6 | 請求 7 | 請求 5 ]  ← 請求 3 完成，請求 7 立即遞補

結果：完成的序列可立即被替換，減少批次槽位閒置。

對於長度最多 2,000 個 token 的互動式提示詞：
  p95 TTFT <= 800 毫秒
  p95 ITL  <= 60 毫秒
  錯誤率 < 0.5%
以滾動式 30 分鐘時間窗計算