Inside SGLang

SGLang is an open-source inference engine: the server software that takes a model's weights, puts them on GPUs, and answers thousands of requests at once. I wrote this guide to walk through how it works, using the real code at github.com/sgl-project/sglang, with small simulations you can poke.

Most chapters take 5-10 minutes (chapter 5, the scheduler, takes longer). Chapters 1-3 cover ideas every engine shares, vLLM included. Chapters 4-8 open up SGLang itself. Chapters 9-13 are for when you want to run it or read the source.

Chapter 01 · Foundations

Why inference engines exist

If you download an open model from Hugging Face, you can generate text in about five lines with the transformers library: load the weights, call model.generate(), print the result. That works, and it gets you roughly 1-5% of what the same GPU can do when it serves many users.

Three things go wrong when that script becomes a service:

1. One request at a time wastes the GPU. Generating a token for one user moves all the model's weights through the chip. Generating a token for 64 users in the same pass moves the same weights once. Batching is close to free throughput, and the engine's job is to keep the batch full.
2. Users arrive and leave at random. One person wants 3 tokens ("Yes."), another wants 3,000. A batch that waits for its slowest member wastes most of its seats. Engines add and remove requests between every step.
3. Memory runs out before compute does. Every token in every conversation leaves behind a block of saved numbers on the GPU (the KV cache). How many users you can serve is mostly a question of how well you manage that memory.

An answer is written one token per decode step. In each step the GPU reads every weight of the model once and makes the next token for every request in the batch. Below is that trip for Llama 3.1 8B (its weights fill 16 GB) on one H100 GPU.

Each request also keeps its own KV cache (the saved numbers from problem 3), and every step reads those too. So the length of each conversation matters as well as how many there are. All the caches have to fit in the memory the weights leave free, about 55 GB on this GPU.

In this model, batching takes the GPU cost of a million output tokens from about $4 to under 5 cents, which is why an engine works so hard to keep the batch full. SGLang is built for the right half of that chart. In practice KV memory fills first, as the chart shows, which is why chapter 3 is all about that memory.

SGLang, vLLM, TensorRT-LLM and others are different answers to the same three problems. SGLang came out of LMSYS (the group behind Chatbot Arena) in 2024. Its best-known idea is RadixAttention: noticing that many requests start with the same text, and computing that text once. Today it serves production traffic at many AI labs and clouds, and it is a common engine for generating rollouts for reinforcement learning (RL).

What "SGLang" stands for

Structured Generation Language. The original 2024 project had two halves: a Python mini-language for writing multi-step LLM programs (gen, select, fork), and a fast runtime underneath it. The runtime won. The language lives on in python/sglang/lang/, but the package README marks it as deprecated and no longer maintained. When people say SGLang today, they mean the runtime, called SRT (SGLang Runtime) in the code: python/sglang/srt/.

Chapter 02 · Foundations

Prefill and decode: the two halves of every request

A transformer predicts one next token from everything before it. To answer a 1,000-token prompt with a 200-token reply, the engine does this:

1. Prefill: push all 1,000 prompt tokens through the model at once. The GPU is good at this, since it is one huge matrix multiplication. The output is the first reply token.
2. Decode: feed that one new token back in, get the next one, and repeat 199 more times. Each pass handles a single token per request, so the GPU spends most of its time waiting for weights to arrive from memory rather than doing math.

That difference explains two things you have seen on API pricing pages. Input tokens are cheaper than output tokens, because prefill processes them in parallel and decode processes output one at a time. And a streamed answer starts after a pause and then flows at a steady rhythm, because the pause, time to first token (TTFT), is mostly prefill and the rhythm after it is decode.

In every layer, the newest token compares itself with every token so far, itself included, and pulls in information from the ones that match best. That step is called attention, and it is the only place where tokens see each other.

Attention needs two vectors from every earlier token, a key and a value. Step through the same dinner bill below, and switch the cache off to see what saving them buys.

This is the whole reason a KV cache works. A token's key and value depend only on that token and the ones before it, so adding new tokens never changes them. The prompt's keys and values are computed once, during prefill, and every later decode step only adds one more pair per layer.

Here is the same split as you feel it in a chat window, on a GPU you share with other users.

Time to first token also includes queueing, so it grows with how busy the server is as well as with your prompt. SGLang's benchmark reports both numbers, computing time per output token as (total time - time to first token) ÷ (output tokens - 1), and a server started with --enable-metrics exports them as Prometheus histograms.

The engine's name for these

In SGLang's code, a prefill batch is called EXTEND, because it extends a sequence by many tokens at once (and sometimes the start of the sequence is already cached, so it only extends the rest). A decode batch is DECODE. These are values of ForwardMode, and you will see them everywhere in the scheduler.

Chapter 03 · Foundations

The KV cache is the budget

Chapter 2 showed that attention needs a key and a value vector for every earlier token in every layer, and that the engine computes them once and saves them. That saved set is the KV cache.

The cost per token is fixed by the model's shape:

bytes per token = 2 (K and V) × layers × KV heads × head size × bytes per number

For Llama 3.1 8B that is 2 × 32 × 8 × 128 × 2 bytes = 131 KB per token. A single 100,000-token conversation needs about 13 GB of GPU memory just for its cache, on top of 16 GB of weights. That is why long context windows are expensive, and why providers limit them.

SGLang decides the split at startup. It loads the weights, measures what is left, and turns a fraction of the GPU into one big pool of KV slots. The flag --mem-fraction-static sets how much of the GPU goes to weights plus that pool. The rest is scratch space for intermediate results. If you leave it unset, SGLang picks a value from your GPU's size (about 0.83 on an 80 GB H100).

Things to try: switch Llama 3.1 70B onto one H100 and watch it fail to fit, then give it 4 GPUs, which splits every layer across them (tensor parallelism, chapter 9). Change the KV cache to FP8 and the token count doubles. Pick DeepSeek-V3, which uses a compressed cache design called MLA (multi-head latent attention): its per-token cost is about half of Llama 8B's despite being 80 times bigger.

Two tables, not one big array

Here is a detail that matters later. SGLang does not give each conversation its own contiguous chunk of memory. It keeps two lookup tables:

• ReqToTokenPool: one row per live request. Row r, column i says which KV slot holds token i of request r.
• TokenToKVPool (the KVCache classes): the actual memory. For each layer, a big tensor of shape [slots, kv_heads, head_dim] for keys and another for values.

A free-slot allocator hands out slot numbers like a coat check. Because a request's tokens can live in any slots, two requests can point at the same slots for a shared beginning. That one fact is what makes RadixAttention possible. vLLM's PagedAttention is the same idea with fixed-size blocks of 16 tokens. On most GPUs SGLang's default "page size" is 1 token, which lets it share prefixes down to the exact token. Some attention kernels need pages of 16-128 tokens, and on Blackwell GPUs such as the B200 the default kernel is one of them, so the default page size there is 64.

Below, one stream of requests runs through two ways of handing out a pool of 48 KV slots. In the first (the way serving systems worked before PagedAttention), a request books one unbroken block for its prompt plus its whole max_tokens limit the moment it arrives. The second is SGLang's way. A request takes one slot per token from the front of the free list, the queue of unused slot numbers.

Because SGLang takes each slot right before it writes a token there, a request never holds an empty slot. With --page-size 16 it hands out 16-slot pages instead, so a request can leave up to 15 slots of its last page empty, which is still far less than booking max_tokens ahead. Real SGLang also estimates, before admitting a request, whether its answer and the answers already running will fit, without booking slots for any of them (chapter 5).

Chapter 04 · Inside SGLang

Life of a request

When you run sglang serve, you get more than one process. The split exists because Python can run only one thread of Python code at a time in a process. If tokenizing a big prompt or formatting JSON ran in the same process as the scheduler, the GPU would sit idle while that work happened. So the work that must never stall (keeping the GPU fed) gets a process of its own.

Here is one chat request passing through those three processes, with the token ids a Qwen model really sees. Step through it hop by hop and then token by token, or play the whole request to watch the answer come back one token per forward pass. The message panel shows what crosses each hop, using SGLang's own class and field names.

POST /v1/chat/completions
{
  "model": "Qwen/Qwen2.5-7B-Instruct",
  "messages": [
    {"role": "system",
     "content": "You are terse."},
    {"role": "user", "content": "Hi"}
  ],
  "temperature": 0.7,
  "max_tokens": 64,
  "stream": true
}

In the default single-machine setup, each ZeroMQ hop is a PUSH socket feeding a PULL socket over an ipc:// address, which is a Unix socket file rather than a network port. The messages are the Python objects above, serialized on one side and rebuilt on the other.

With more than one GPU

With --tp-size 4, which splits every layer of the model across four GPUs (tensor parallelism, chapter 9), SGLang starts four scheduler processes, one per GPU. Only the first of them, called rank 0, reads from the socket. It broadcasts each batch of new requests to the other three, and all four then make identical scheduling decisions, so their batches line up exactly when the model's layers talk to each other across GPUs. Only rank 0 sends results back.

Chapter 05 · Inside SGLang

The scheduler: continuous batching

This loop is the core of the engine. Trimmed down, here is what event_loop_normal in scheduler.py does, forever:

# python/sglang/srt/managers/scheduler.py (trimmed)
while True:
    self.ingest_requests()                    # pull new requests off the socket
    plan = self.get_next_batch_to_run(...)    # prefill new arrivals, or decode the running batch
    batch = plan.batch_to_run
    if batch:
        result = self.run_batch(batch)        # one forward pass on the GPU
        self.process_batch_result(batch, result)  # append tokens, check stop, stream out
    else:
        self.on_idle()
    self.last_batch = batch

The interesting part is get_next_batch_to_run. Each turn it:

1. Merges the requests that were just prefilled into the running batch, the set of requests currently decoding.
2. Drops finished requests from the running batch.
3. Tries to build a prefill batch from the waiting queue. If it can, that runs this turn. Prefill gets priority, so new users see their first token quickly.
4. Otherwise, it runs one decode step for everyone in the running batch.

That is continuous batching (sometimes called in-flight batching). Compare it with the obvious approach below.

Look at the faded cells in the continuous grid. When a prefill pass runs, every request that is already decoding pauses for that pass. With a short prompt that costs little. With a 30,000-token prompt it stops everyone's stream for a noticeable moment.

Chunked prefill

SGLang cuts a long prompt into chunks of --chunked-prefill-size tokens and prefills one chunk per pass. That caps how much scratch memory any single pass needs. The default chunk size grows with GPU memory (8,192 tokens on an H100), because bigger GPUs are usually faster and need bigger passes to stay busy. The chunk-size menu below lists the default for each class of GPU. A half-done prompt carries over between passes as self.chunked_req and is admitted first on the next pass.

Chunking alone does not restart the other streams, because prefill keeps priority and the chunks run back to back. Two flags change that. --enable-mixed-chunk puts the decoding requests' next tokens into each chunk's pass (a MIXED batch), and --prefill-decode-interval N runs N decode passes after each chunk. The full fix is to prefill on other GPUs (chapter 9).

Below, four users are partway through their answers when a fifth request arrives with a 24,000-token prompt (say, a long contract to summarize). Pick how the engine handles that prompt, then watch the four streams and the new request's wait for its first token.

Admission: betting on how long answers will be

The scheduler cannot know how many tokens a request will generate. If it admits too many, they run out of KV memory mid-answer. If it admits too few, the GPU idles. The PrefillAdder class handles this: for each running request it reserves room for its remaining max_tokens times a new-token ratio. The ratio starts at 0.7 (scaled by --schedule-conservativeness) and slowly decays toward about 0.1 while things go well, since most answers stop long before their limit.

When the bet loses and the pool fills up, the scheduler retracts: it pauses the requests with the fewest generated tokens, frees their memory, and puts them back in the queue to resume later. You will see KV cache pool is full. Retract requests. in the log. A few retractions are normal. Frequent ones mean you should raise --schedule-conservativeness or lower --max-running-requests.

Who goes first

--schedule-policy picks the order of the waiting queue. The default is fcfs (first come, first served). The cache-aware option lpm (longest prefix match) sorts waiting requests by how much of their prompt is already cached, which raises cache hits when many requests share prefixes. Sorting costs CPU time, so LPM quietly falls back to FCFS when more than 128 requests are waiting. Other options include dfs-weight, lof (longest output first), hrrn and random.

The control room below runs all of these rules together on a deliberately small KV pool, so the pressure shows. Each tick is one trip around the loop: new arrivals join the queue, then the scheduler runs a prefill if it can admit someone, or one decode step for everyone running. Each card shows a request's max_new, the max_tokens it was sent with (SGLang stores it as max_new_tokens).

Yellow in the pool is cache, the KV that finished requests and shared system prompts left behind. A later prompt that starts the same way reuses it, and the scheduler evicts it when it needs room (chapter 6).

Things to try: play Rush hour to about pass 300, then raise conservativeness to 1.6. The server replays the same arrivals and retracts fewer requests. Drop it to 0.3 and it retracts more. In Shared system prompt, play to about pass 400, then switch the policy from lpm to fcfs. lpm serves requests whose prefix is already cached first, so it can leave others waiting. Under lpm the cache hit rate is a point or two higher and the finished counts stay close, but the longest wait for a first token is much longer.

The overlap scheduler

Scheduling takes CPU time: matching prefixes, allocating memory, building tensors. In a naive loop the GPU waits while the CPU does that between passes. SGLang's default loop, event_loop_overlap, launches pass N on the GPU and then, while it runs, processes pass N-1's results and prepares pass N+1. The catch is that pass N+1 needs the tokens pass N has not produced yet. The second half of the simulation below shows how a GPU buffer called FutureMap gets around that. Turn overlap off with --disable-overlap-schedule when debugging.

Planning one pass ahead has a side effect. A request that emits its stop token in pass N is already in pass N+1, because the CPU planned that pass before it saw the stop. The GPU computes one extra token for it, and SGLang skips that token while processing results, so it is never sent. A request that reaches its max_tokens limit is predictable, so the overlap loop leaves it out of the next pass instead.

Chapter 06 · Inside SGLang

RadixAttention: never compute the same prompt twice

Real traffic repeats itself constantly. Every request to a chatbot starts with the same system prompt. Every turn of a conversation resends all previous turns. An agent re-reads its tool list and history on every step. Few-shot prompts share their examples across thousands of calls. Without a cache, the engine prefills all of that again each time.

The "cached input tokens" discount on API pricing pages is this same idea, run by the provider. With SGLang, you run it yourself, and it is on by default.

The data structure is a radix tree (a compressed trie). Each edge holds a run of tokens. Each node remembers which KV slots hold those tokens. Requests that share a beginning share a path from the root. The tree stores slot numbers, not the keys and values themselves, so sharing costs nothing: two requests simply point at the same slots.

Try the multi-turn conversation preset: each turn reuses almost everything. Then try unrelated prompts: nothing helps. Shrink the memory slider and watch the least recently used branches get evicted.

When it helps, and when it does not

Two practical tips follow. Put the fixed parts of your prompts first and the variable parts last, because only the beginning can match. And avoid putting a timestamp or request id at the top of the system prompt, which makes every prompt unique from the first token. Launch with --enable-cache-report and every OpenAI-style response reports usage.prompt_tokens_details.cached_tokens, so you can measure your hit rate.

What the real code does, step by step

1. Match. When a request is considered for admission, match_prefix walks the tree from the root and returns the slots of the longest matching beginning. If the match ends halfway along an edge, the node is split in two so the shared half becomes its own node. Splitting copies no KV data; it only slices the lists of slot numbers.
2. Lock. The matched path gets its lock_ref raised, so those slots cannot be evicted while this request depends on them.
3. Allocate the rest. Only the unmatched tail gets new KV slots and gets prefilled. The log's #cached-token counts the reused part.
4. Insert early. Right after prefill, the prompt is inserted into the tree, so other requests can share it while this one is still writing its answer.
5. Insert again on finish. When the request finishes, the full sequence, answer included, goes into the tree. That is why the next turn of a chat hits: its prompt is the previous prompt plus the previous answer.
6. Unlock and evict lazily. The lock is released. Nothing is deleted yet. Only when the allocator runs short does the cache evict, picking unlocked leaves by --radix-eviction-policy (default lru; also lfu, slru, priority, tlru). Removing a leaf may turn its parent into a leaf, so eviction eats branches from the tips inward and the shared trunk stays longest.

A note on the code you will find: mem_cache/radix_cache.py is the clearest version to read, and the playground above mirrors it. Production now runs UnifiedRadixCache, with the tree's core written in Rust by default, which applies the same ideas to newer model types (sliding-window and Mamba-style layers).

Beyond one GPU's memory: HiCache

GPU memory is small, so popular prefixes get evicted. --enable-hierarchical-cache adds two more tiers: evicted KV is kept in CPU memory (by default twice the size of the GPU pool), and optionally in a storage backend shared across servers (Mooncake, 3FS, NIXL, local files). A request that matches an entry in CPU memory copies it back layer by layer, overlapped with compute, which is much faster than recomputing a long prompt.

Across many servers: cache-aware routing

With several SGLang replicas, a prefix cached on server A is useless if the next turn lands on server B. SGLang's main router, the SGLang Model Gateway (written in Rust, in sgl-model-gateway/), keeps an approximate radix tree of recent request text for each server and sends each request to the one with the longest match. When load gets lopsided it switches to the shortest queue, so no server falls far behind. cache_aware is its default policy. The repo also holds a slimmer router in sgl-router/, whose default policy is power_of_two.

Below, one router spreads eight chats across three SGLang servers. Every turn resends the whole chat so far, so a turn is cheap only on a server that still holds the turns before it. All three policies route the same requests, so you can switch between them at any point and compare.

Things to try: switch to round robin. Each new turn usually lands on a server without the turns before it, so it recomputes the whole chat and pushes other chats out of that server's cache. The hit rate stays at about a third or less. Back on cache-aware, press Step once to see the exception. Server C has three requests running and the others have none, so the router skips its tree for this one request and picks the shortest queue. A and B tie at zero, and the gateway breaks a tie at random without looking at its tree, so chat 1 lands on A, which holds none of it, while B holds its earlier turns. Near the end of a cache-aware run, dashed rows show chats the router still expects on a server that has already evicted them.

Chapter 07 · Inside SGLang

On the GPU: kernels, CUDA graphs and sampling

From batch to tensors

The scheduler's ScheduleBatch is Python bookkeeping. Right before the GPU runs, it becomes a ForwardBatch: a handful of tensors that describe the whole batch at once, such as the token ids to run, their positions, each request's row in the slot table, and the KV slots this pass writes to. Step 5 of the request diagram in chapter 4 shows one.

A model file, briefly

Each supported architecture has a file in python/sglang/srt/models/ (there are hundreds). They read like the Hugging Face versions, except that linear layers are tensor-parallel and attention goes through RadixAttention. Here is the attention block from llama.py:

# python/sglang/srt/models/llama.py · LlamaAttention.forward (trimmed)
qkv, _ = self.qkv_proj(hidden_states)               # one matmul makes Q, K and V
q, k, v = qkv.split([self.q_size, self.kv_size, self.kv_size], dim=-1)
q, k = self.rotary_emb(positions, q, k)             # encode positions (RoPE)
attn_output = self.attn(q, k, v, forward_batch)     # RadixAttention → attention backend
output, _ = self.o_proj(attn_output)

Despite its name, RadixAttention has no tree logic in it. It is a thin layer that hands q, k and v to whichever attention backend is active. The backend writes the new k and v into the KV pool at the slots chosen for this pass (out_cache_loc), then runs a kernel that gathers every earlier token's keys and values through the slot table. Prefix sharing is invisible at this level: the kernel reads slots and does not care who else points at them.

Attention backends

Attention over a paged cache is the one operation PyTorch cannot do well by itself, so SGLang plugs in kernels from several projects and picks one for your hardware. Override with --attention-backend, or separately with --prefill-attention-backend and --decode-attention-backend.

DeepSeek-style MLA models have their own kernels (flashmla, trtllm_mla, cutedsl_mla), and so do sparse-attention models.

CUDA graphs

A decode step for a small batch is hundreds of tiny GPU kernels, each finishing in microseconds. Launching each one from Python costs more time than running it. A CUDA graph records the whole sequence of launches once and replays it with a single call.

The catch: a graph is recorded for one exact batch size. So at startup SGLang captures decode graphs for a list of sizes (1, 2, 4, 8, 12, 16, 24, 32, and so on up to 256 on an H100) and pads each real batch up to the nearest captured size. That is why startup takes a minute, and why the log line says cuda graph: True on most decode steps. Recent versions also capture prefill in pieces, with attention run normally between them.

With 4 or more GPUs in tensor parallel, the default on an 80 GB card rises to 512. Setting --cuda-graph-max-bs-decode higher captures more graphs, which costs startup time and some GPU memory for the graphs' buffers.

From logits to a token

The model's last layer produces a score (logit) for every token in the vocabulary, about 150,000 of them for Qwen. The sampler then applies, in order:

1. Repetition, frequency and presence penalties
2. The grammar mask, if the request asked for structured output (next chapter)
3. logit_bias
4. Temperature, then top-k, top-p and min-p filtering, then a random draw

Temperature 0 becomes plain argmax (greedy). When every request in a batch is greedy, the whole sampling step is a single argmax. These are the same knobs you set in an API call; here you can see where each one acts.

Chapter 08 · Inside SGLang

Structured output and speculative decoding

Structured output: grammar masks

If you have used response_format with a JSON schema on the OpenAI API, you have used this. The engine compiles your schema into a grammar. Before every sampling step, the grammar computes which tokens are legal next, and every other token's logit is set to -∞. The model can only pick among what the schema allows, so the output parses as long as max_tokens does not cut it off (check for finish_reason: "length").

SGLang's default grammar engine is xgrammar (--grammar-backend also accepts outlines and llguidance). You can constrain with a JSON schema, a regular expression or an EBNF grammar, one per request. Through the OpenAI API, use response_format for JSON and extra_body={"regex": ...} or {"ebnf": ...} for the others. For reasoning models launched with --reasoning-parser, the constraint switches on only after the model finishes thinking, so the thinking text stays free-form.

Speculative decoding: guess, then check

Decode is slow because each token needs a full pass through the big model, and that pass is mostly spent waiting on memory. Checking 5 tokens costs the big model about the same as generating 1, because it reads its weights once either way. So:

1. A small, cheap draft model guesses the next few tokens.
2. The big target model checks all the guesses in a single pass.
3. Guesses are kept up to the first one the target disagrees with. At that point the target's own token is used instead, so every pass gains at least one token.

With greedy sampling, the output is exactly what the big model would have produced alone. With temperature above 0, a rejection-sampling rule keeps the distribution the same. You get the big model's answer, faster.

The speedup depends on how often the draft is right, which depends on how predictable the text is. Code and structured text accept long runs; creative writing accepts fewer. Each round also costs the draft model's time, which is why good drafts are tiny.

SGLang supports several drafting methods through --speculative-algorithm:

• EAGLE3 (the docs' recommendation) and EAGLE: a one-layer draft head, cheap and well informed, that drafts a small tree of guesses. The widget below walks through one pass. You need a trained draft head for your model, published for many popular ones. SpecForge, a sibling project, trains them.
• MTP: models like DeepSeek-V3 ship with built-in multi-token-prediction layers. There is no plain MTP value; you pass EAGLE (or its alias NEXTN) and SGLang uses those layers as the draft.
• STANDALONE: a small ordinary model from the same family as the big model serves as the draft, for example Qwen2.5-1.5B for Qwen2.5-7B.
• NGRAM: no extra model at all; it guesses by finding the current text earlier in the context. It works well for editing and retrieval tasks that copy text.

Once it runs, the decode log line shows accept len, the average number of tokens gained per big-model pass.

The simulator above drafts a single line of guesses, so one wrong guess ends the run. EAGLE hedges its bets. Its draft head works from the big model's hidden states, the vectors the big model computes for each token on its way to choosing the next one. At each step the draft head keeps its top few guesses, and the guesses grow into a small tree. Here is one pass, picking up the sentence after "Each person owes".

Two details make the single pass work. Guesses at the same depth share a position number, because each would sit at the same place in the text, and the mask keeps them from reading each other. The verify pass also saves the big model's hidden states, so the draft head can start the next tree from the token the big model just added. Bigger trees are not free. Each draft token adds work and memory to the verify pass, and each draft step is one more draft-head pass, so with a full batch a bigger tree can be slower. The docs suggest leaving all three flags unset so SGLang picks values for your model. SGLang's automatic choice for Llama models is 5 steps, topk 4 and 8 draft tokens. That makes 68 candidate guesses, pruned to the best 7.

Chapter 09 · In practice

Many GPUs, many machines

You will see these acronyms in every launch command for a big model. Each one is a different way to cut the same work. Pick a tab.

python -m sglang_router.launch_router \
  --pd-disaggregation \
  --prefill http://127.0.0.1:30000 \
  --decode http://127.0.0.1:30001

These combine. A typical large deployment of a DeepSeek-class model uses PD disaggregation across machines, expert parallelism and DP attention within each server, and the gateway in front. For a model that fits on one machine, plain TP plus a few DP copies behind the gateway covers most needs.

Chapter 10 · In practice

SGLang, vLLM and the rest

Everything in chapters 1-3 applies to both. vLLM made paged KV memory famous in 2023 with PagedAttention; SGLang made prefix sharing famous in 2024 with RadixAttention. Each has since adopted the other's idea in its own form. The README of SGLang credits vLLM among the projects it learned from, and SGLang's own benchmark tool can drive a vLLM server for side-by-side tests.

The prefix cache row hides a small practical difference. SGLang's radix tree compares a new prompt with cached ones token by token. vLLM's automatic prefix caching cuts every prompt into fixed blocks (16 tokens by default for most models on NVIDIA GPUs; its MLA kernels for DeepSeek-style models need 32 to 128), hashes each full block together with the hash of the block before it, and reuses a block only when that whole chain of hashes matches. A block that is only partly shared is computed again.

The rounding never costs more than one block or page minus a token per request, however long the shared prefix is. With a system prompt of a few thousand tokens that is under 1% of the prompt for 16-token blocks and up to about 3% for 64-token pages, so for most chat and agent traffic the two designs reuse nearly the same amount. The gap shows up when shared prefixes are short. SGLang rounds down too when an attention backend needs pages of more than one token. On a B200 its default kernels use 64-token pages, so there a 16-token block cache rounds less than SGLang does for Llama- and Qwen-style models. For DeepSeek-V3, vLLM's own MLA kernels use blocks of 32 tokens or more too.

Benchmarks between the two change with nearly every release, and both teams publish numbers that favor their own setups. If performance decides it for you, run python -m sglang.benchmark.serving (or vLLM's equivalent) against both, on your model, your GPU and a dataset shaped like your traffic.

Other options you will hear about

When would you reach for SGLang?

• Self-hosting an open model for cost, privacy or control, at a volume where a GPU stays busy.
• Big batch jobs with a shared prompt: tagging 500,000 reviews with the same instructions and examples. The radix cache computes the shared part in the first batch and reuses it for the rest, and the offline Engine skips HTTP entirely.
• Agents with long, repeated context: tool definitions, files, history.
• Training with reinforcement learning, where a trainer needs millions of sampled answers quickly. verl, slime, AReaL and others use SGLang for this.

When traffic is low, a hosted API is usually cheaper than a GPU idling at a few dollars an hour.

Chapter 11 · In practice

Running it yourself

1 · Get a GPU

You need a Linux machine with an NVIDIA GPU (AMD and others work too; see the hardware docs). Renting one by the hour from a cloud GPU provider is the easy path. A 24 GB card is plenty for Qwen/Qwen3-0.6B or an 8B model. SGLang also runs on Apple Silicon Macs through Metal and MLX, which is handy for poking around, though real serving means datacenter GPUs.

2 · Start the server

# with Docker (includes everything)
docker run --rm --gpus all --ipc=host -p 30000:30000 \
  lmsysorg/sglang:latest \
  sglang serve Qwen/Qwen3-0.6B --host 0.0.0.0 --port 30000

# or with uv in a Python 3.12 environment
uv pip install --prerelease=allow sglang
sglang serve Qwen/Qwen3-0.6B --host 0.0.0.0 --port 30000

Wait for The server is fired up and ready to roll!. The older spelling, python -m sglang.launch_server --model-path …, still works and appears in many docs pages.

3 · Call it like OpenAI

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

resp = client.chat.completions.create(
    model="Qwen/Qwen3-0.6B",
    messages=[{"role": "user", "content": "Split $90 three ways."}],
    temperature=0.7,
    stream=True,
)
for chunk in resp:
    print(chunk.choices[0].delta.content or "", end="")

Structured output uses the standard OpenAI parameter:

resp = client.chat.completions.create(
    model="Qwen/Qwen3-0.6B",
    messages=[{"role": "user", "content": "Extract the dish and price: 'Pizza, $12'"}],
    response_format={
        "type": "json_schema",
        "json_schema": {"name": "line_item", "schema": {
            "type": "object",
            "properties": {"item": {"type": "string"}, "price": {"type": "number"}},
            "required": ["item", "price"],
        }},
    },
)

4 · Or skip HTTP for batch jobs

The offline engine runs the same three-process system inside your Python script, which is ideal for one-off jobs over a big file:

import sglang as sgl

llm = sgl.Engine(model_path="Qwen/Qwen3-0.6B")
prompts = [f"Label the review as positive or negative: {r}\nLabel:" for r in reviews]
outputs = llm.generate(prompts, {"temperature": 0, "max_new_tokens": 4})
labels = [o["text"].strip() for o in outputs]
llm.shutdown()

Every prompt in that list shares its instruction prefix, so the radix cache computes it in the first batch and reuses it for every batch after that.

5 · The flags worth knowing

The full list is long; run sglang serve --help or see the server arguments page in the docs. The SGLang Cookbook gives ready-made launch commands per model and GPU, which beats tuning by hand.

Most of those flags have good defaults, so a real command needs only a few. Pick a model, your GPUs and what you need below. The builder sizes tensor parallelism with the memory math from chapter 3, adds only the flags your needs call for, and writes the client code that goes with them.

sglang serve \  meta-llama/Llama-3.1-70B-Instruct \  --tp-size 4 \  --dp-size 2 \  --tool-call-parser llama3 \  --enable-cache-report \  --host 0.0.0.0 --port 30000

from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [{"type": "function", "function": {    "name": "split_bill", "description": "Split a bill evenly",    "parameters": {"type": "object", "properties": {        "total": {"type": "number"}, "people": {"type": "integer"}},        "required": ["total", "people"]}}}] resp = client.chat.completions.create(    model="meta-llama/Llama-3.1-70B-Instruct",    messages=[{"role": "user", "content": "Split $90 three ways."}],    tools=tools,)msg = resp.choices[0].messageprint(msg.tool_calls)  # parsed by --tool-call-parserprint(msg.content)print(resp.usage.prompt_tokens_details.cached_tokens)  # needs --enable-cache-report

JSON output is the one goal with no flag. Structured output is on by default with the xgrammar backend, and each request asks for it with response_format. Speculative decoding (chapter 8) is left out. EAGLE-style drafting needs a draft model trained for your model, DeepSeek-V3 carries its own draft layer (multi-token prediction, MTP), and NGRAM needs no draft model at all, so take the exact flags from the model's Cookbook page.

6 · Measure it

python -m sglang.benchmark.serving --backend sglang \
  --host 127.0.0.1 --port 30000 --num-prompts 500

It reports the numbers serving people argue about: TTFT (time to first token, mostly prefill and queueing), ITL or TPOT (time between output tokens, mostly decode), and total throughput. Try the same run with --disable-radix-cache on the server, or with a different --max-running-requests, and you will see the trade-offs from this guide in your own numbers.

Chapter 12 · In practice

Reading the code

Almost everything that matters is under python/sglang/srt/. The rest is kernels (python/sglang/kernels/, 3rdparty/), the Rust gateway (sgl-model-gateway/), image and video generation (python/sglang/multimodal_gen/), docs, benchmarks and tests. The paths below are relative to python/sglang/srt/ and link to the exact commit this guide was written against.

Here is srt/ drawn to scale. Each box's area is its lines of Python. The blue numbers show where the 14 files of the reading order below live; open a folder to find them, or walk the reading order and watch a request's path jump between folders.

Nearly half of the code outside the reading order sits in models/, layers/ and hardware_backend/, which hold per-model code, attention and MoE variants, quantization formats and support for other chips. A first read can skip all of that.

1. 1entrypoints/http_server.py

   FastAPI app and every HTTP route. Find /v1/chat/completions and follow the call.

2. 2entrypoints/openai/serving_chat.py

   Turns an OpenAI chat request into a native request: chat template, tools, response format.

3. 3managers/tokenizer_manager.py

   generate_request tokenizes and sends; handle_loop receives results and wakes waiting requests.

4. 4managers/io_struct.py

   The message types passed between processes. Skim it to learn the vocabulary.

5. 5managers/scheduler.py

   The event loops and get_next_batch_to_run. 6,000+ lines; read the two loops and that one function first.

6. 6managers/schedule_policy.py

   Queue ordering and PrefillAdder, the admission budget.

7. 7managers/schedule_batch.py

   Req and ScheduleBatch: per-request state, prepare_for_extend, prepare_for_decode, retraction.

8. 8mem_cache/radix_cache.py

   The radix tree in its clearest form, about 830 lines, plus RadixKey, the prefix key every cache uses. A default server builds the same idea from mem_cache/unified_radix_cache.py on a Rust core (chapter 6). Then mem_cache/README.md for the map of the rest.

9. 9mem_cache/memory_pool.py

   ReqToTokenPool and the KV pool classes.

10. 10managers/tp_worker.py

    The bridge: ScheduleBatch → ForwardBatch → forward → sample.

11. 11model_executor/model_runner.py

    Loads the model, sizes memory, picks the graph runner or eager path.

12. 12models/llama.py

    A whole model in one file. Compare it with the Hugging Face version.

13. 13layers/attention/flashinfer_backend.py

    Where K/V gets written to the pool and attention runs.

14. 14managers/detokenizer_manager.py

    Incremental decoding of token ids back into text.

Easier ways in

• Mini-SGLang: a compact rewrite by the same project, meant for learning. If the real scheduler is too much to start with, read this first.
• DeepLearning.AI short course on efficient inference with SGLang.
• docs.sglang.io: the advanced features section has a page per topic in this guide.
• The LMSYS blog: release write-ups that explain why each big feature exists.

Chapter 13 · In practice

Glossary