vLLM & SGLang
Engineering:
End-to-End.
A twelve-lecture live course on serving large language models with vLLM and SGLang, taught by Dr. Sreedath Panat (MIT PhD). You read both engines, benchmark them, tune, quantize and scale each, and finish with a production endpoint on the engine your own numbers picked.
Every session is recorded. Recordings included.
What an engine does every step
Requests arrive, the scheduler admits them, one decode step advances every sequence in the batch, and each sequence's KV cache is handed out and returned as it runs. vLLM and SGLang both run this loop. How each one schedules the step and stores the cache is what the course is about.
Two answers to the same question, both from Berkeley
vLLM started in 2023 at UC Berkeley, when Woosuk Kwon, Zhuohan Li and their collaborators asked why LLM serving wasted so much GPU memory. Their answer, PagedAttention, manages the key-value cache the way an operating system manages memory: in fixed blocks, allocated on demand, shared where two requests overlap. It became a community project with thousands of contributors, joined the PyTorch Foundation in 2025, and rewrote its core as the V1 engine.
SGLang came out of the same Berkeley groups a few months later, from Lianmin Zheng, Ying Sheng and the LMSYS team. It asked a different question: what if serving understood the program calling it? Its frontend language expresses multi-call, branching workloads, and RadixAttention keeps the KV cache in a prefix tree so every shared prefix across requests and turns is stored once. Both engines ship weekly today, and this course reads, benchmarks and deploys both as they are now.
Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, Li, Zhuang, Sheng, Zheng, Yu, Gonzalez, Zhang, Stoica
ReadSGLang: Efficient Execution of Structured Language Model Programs
Zheng, Yin, Xie, Sun, Huang, Chen, Bai, Cao, Zhang, Gonzalez, Barrett, Sheng and others
ReadWhy learn both, properly
Three reasons, in short.
These are the two engines you will meet
vLLM and SGLang are the two most used open-source inference engines, with over 92,000 and 36,000 GitHub stars and millions of downloads a month between them. If you serve open-weight models, one of them is your engine and the other is the one you are compared against.
The knobs decide the GPU bill
The same model on the same GPU can differ several times over in throughput depending on scheduler budgets, cache settings, quantization and parallelism, and the right settings differ between the two engines. This course teaches you to measure each one instead of copying a config.
The choice is a measurement, not an opinion
Which engine wins depends on the workload: shared prefixes, multi-turn agents, long prompts, mixture-of-experts models. You learn both well enough to run the comparison yourself and defend the answer.
The two most used open-source inference engines
Both repositories from the public record, and how much each project ships each year.
Merged pull requests per year
How fast both codebases move, and why reading them is a skill
Source: GitHub search, vllm-project/vllm and sgl-project/sglang, counted 23 September 2026.
What the two engines change about serving
A plain generate loop pads a batch, preallocates the cache, and waits. Both engines schedule at the token level and manage the cache cleverly, but not in the same way. The table shows where they agree and where they differ.
The request path you trace in lectures 1 and 2
Pick an engine. Each block is highlighted in turn with a note on what it does and which lecture goes deep on it.
API server
The OpenAI-compatible frontend. It validates the request, applies the chat template, tokenizes, and streams tokens back as they are produced. Structured outputs and tool-call parsers live here.
Three components you will read, measure and tune
Simplified views of three components, drawn the vLLM way. The ideas behind them are assumed knowledge. The lectures are about how vLLM and SGLang each implement them, where they live in the code, and which settings change the numbers.
The KV cache: blocks or a radix tree
vLLM keeps a block pool with block tables and shares prefixes by hash. SGLang keeps a radix tree of token sequences and shares any common prefix. Lectures 1, 2, 6 and 7 cover what each buys you.
The scheduler's step, in each engine
Both engines refill a freed slot at the next step within a token budget. vLLM tunes it with max_num_seqs and max_num_batched_tokens; SGLang adds an overlap scheduler that prepares the next batch while the GPU runs.
Speculative decoding in both engines
A drafter (EAGLE, MTP, n-gram or a draft model) proposes, the target verifies in one pass, and the accepted prefix is kept. Both engines expose the acceptance rate, and lecture 10 measures whether it pays on each.
Twelve lectures over six weeks
Two lectures a week. Most topics are taught as a pair: how vLLM does it, how SGLang does it, and what that changes in the numbers. Details may change as the material is finalized.
Inside both engines
vLLM architecture and the request lifecycle
What problem vLLM solves, the V1 engine (frontend, EngineCore, scheduler, KV cache manager, model runner, workers), PagedAttention, and how vLLM implements continuous batching and chunked prefill.
Build vLLM from source, trace one request through the code with logging, and run your first model.
SGLang architecture: frontend language and the SRT runtime
What SGLang adds on top of serving: the frontend language for structured programs, the SRT runtime, its scheduler, RadixAttention and the radix cache, and how requests move from the HTTP server through the tokenizer manager to the workers.
Build SGLang from source, trace the same request through its runtime, and compare the two paths side by side.
Serving on each
Serving with vLLM
vllm serve and the OpenAI-compatible API, chat, completions and streaming, sampling parameters, structured outputs, tool-call and reasoning parsers, multi-LoRA serving, multimodal models, and serving Hugging Face checkpoints.
Deploy a server with several LoRA adapters and constrained JSON output, then build a small application on top of it.
Serving with SGLang
sglang.launch_server and the OpenAI-compatible API, SGLang programs with gen, select and fork, structured outputs with xgrammar and llguidance, multi-turn and agentic workloads that reuse the radix cache, LoRA and multimodal serving.
Rebuild the lecture 3 application as an SGLang program and measure what the shared prefix cache saves on a multi-turn workload.
Measure, then tune
Benchmarking both engines with one harness
TTFT, TPOT, inter-token latency and throughput, request throughput versus token throughput, vllm bench and sglang.bench_serving, concurrency and load testing, GPU utilization and memory measurement, profiling with torch profiler and Nsight.
Design and run one benchmarking experiment against both engines, whose numbers you can defend, and keep it as the harness for the rest of the course.
vLLM performance engineering
max_num_seqs, max_num_batched_tokens, gpu_memory_utilization, max_model_len, chunked prefill tuning, automatic prefix caching (hashing, eviction, hit rate), KV cache tuning, attention backends (FlashAttention, FlashInfer, Triton), CUDA graphs and torch.compile.
Tune a vLLM deployment against a latency target step by step, measuring every knob against the lecture 5 harness.
Tune, quantize, fit
SGLang performance engineering
The radix cache and its eviction, chunked prefill, the overlap scheduler and zero-overhead batch scheduling, mem-fraction-static and max-running-requests, attention backends (FlashInfer, FlashAttention 3, Triton), CUDA graphs and torch.compile in SGLang.
Tune the same deployment on SGLang against the same target, and explain every place the two engines needed different settings.
Quantization and memory engineering on both engines
Which formats each engine supports on which hardware, the kernels behind them (Marlin, Machete, FP8 paths, W8A8), producing a checkpoint both engines load with llm-compressor, KV cache precision, memory calculations, context length versus concurrency.
Quantize a model to FP8 and INT4, serve it on both engines, and compare accuracy, memory and latency with the BF16 baseline.
Scale and speculate
Multi-GPU and mixture-of-experts serving
How each engine implements tensor, pipeline, data and expert parallelism, MoE kernels and DeepSeek-class models, multi-node serving with Ray and with SGLang's multi-node launch, and how to measure communication overhead.
Serve a large model across 1, 2, 4 and 8 GPUs on both engines, compare parallelism plans, and explain where the scaling stops.
Speculative decoding and disaggregated prefill and decode
EAGLE, MTP, n-gram and draft-model speculation in vLLM and in SGLang, acceptance rate as the deciding metric, disaggregated prefill and decode: vLLM's KV connectors (LMCache, NIXL) and SGLang's PD disaggregation with Mooncake, and when either pays off.
Add speculative decoding on both engines, measure acceptance rate and speedup on a chat workload, and run one disaggregated deployment.
Production and the choice
Production: routers, Kubernetes, metrics and cost
Dockerizing each engine, Kubernetes and replicas, SGLang's cache-aware router versus vLLM's production stack and prefix-aware routing, autoscaling, Prometheus metrics and observability, failure handling, cost per token and capacity planning.
Deploy replicas of both engines behind a router on Kubernetes with dashboards and an autoscaling policy.
Choosing, migrating, extending, and the capstone
Which engine for which workload, migrating a deployment between them, how to add a model, a plugin or a custom kernel to each, and how to read both codebases as they change.
Capstone: serve one model on vLLM and on SGLang, benchmark both, deploy the winner, and defend the choice with numbers.
What you have built by the end
Everything lives in one repository that grows across the lectures. The benchmark harness from lecture 5 measures every change you make after it, on both engines, up to the capstone endpoint.
A benchmark harness that runs against both engines
TTFT, TPOT, and throughput under load, built in lecture 5 and run against every change you make afterwards, on vLLM and on SGLang.
A tuned single-GPU deployment on each engine
Scheduler budgets, cache settings and attention backends chosen from your own measurements, with a written account of where the two engines needed different settings.
A quantized model in production form
An FP8 or INT4 checkpoint you produced with llm-compressor, served by both engines, with the accuracy and latency difference documented.
A production endpoint, and a defended choice
Served across GPUs, containerized, behind a router on Kubernetes with metrics and autoscaling, on the engine your capstone benchmarks picked.
The stack you work in, as it is used in production
Nothing here is a teaching substitute. These are the tools the labs run on.
vLLM
Engine one, V1
SGLang
Engine two, SRT runtime
FlashAttention / FlashInfer
Attention backends
llm-compressor
FP8 and INT4 checkpoints
xgrammar / llguidance
Structured outputs
LMCache / NIXL / Mooncake
KV transfer
SGLang router / vLLM production stack
Cache-aware routing
Docker + Kubernetes
Deployment
Prometheus + Grafana
Metrics
torch profiler + Nsight
Profiling
Ship an endpoint and defend the numbers
The last lecture is the capstone. You bring two deployments, one benchmark report, and the reasoning behind the engine you chose.
One model, both engines, one defended choice
Pick an open-weight model and a latency target. Serve it on vLLM and on SGLang, tune each one the way the lectures taught, decide whether quantization and speculative decoding pay off on each, and ship the winner with a router, metrics and autoscaling.
- A written service level target: p50 and p99 latency, throughput, and cost per million tokens
- A benchmark report that compares the two engines, baseline and tuned, on the same workload
- A Kubernetes deployment of the chosen engine with Prometheus metrics and an autoscaling policy
- A short presentation of what you tried on each engine, what helped, and why you chose one
Who this course is for
- →Engineers who serve open-weight models and want to stop guessing which engine and which flags to use
- →ML engineers moving from training into inference and deployment
- →Platform and infrastructure engineers who own the GPU bill
- →Anyone who has learned inference fundamentals and wants to see those ideas inside two real engines
What you will be able to do
- →Explain how a request moves through vLLM and through SGLang, and where the time goes in each
- →Benchmark both engines properly and tune either one to a latency or throughput target
- →Quantize a model, serve it on both, and measure what changed
- →Scale to multiple GPUs and choose the right parallelism plan on each engine
- →Deploy either engine on Kubernetes with a router, metrics, autoscaling, and a cost model
- →Choose between the two for a given workload, and defend the choice with numbers
You should be comfortable with Python and the command line and have run a model on a GPU before. Labs run on rented cloud GPUs; a budget guide is shared before the cohort starts.
Dr. Sreedath Panat
MIT PhD · Vizuara AI Labs
Taught by Dr. Sreedath Panat
Dr. Sreedath holds a PhD from MIT and is the co-founder and director of Vizuara AI Labs. An IIT Madras graduate and department gold medalist, he has built a 200K+ subscriber YouTube channel and co-authored a Manning book on building DeepSeek from scratch. He teaches every concept from first principles.
- All 8 lectures personally delivered
- PhD from MIT
- IIT Madras graduate and department gold medalist
- Winner of the Langmuir Award
- 200K+ YouTube subscribers, 115K+ LinkedIn followers
Build a DeepSeek Model from Scratch
Raj Dandekar, Rajat Dandekar, Sreedath Panat, Naman Dwivedi
View on manning.comQuestions? Write to sreedath@vizuara.com
Start your research with a head start.
Do not start from scratch. Tell us your topic of interest and we will generate a personalised research roadmap and an initial version of your research paper, delivered asynchronously, so you can hit the ground running from day one.
What is in the kit
Personalised research roadmap (PDF)
You tell us your topic. We produce an 8-week plan with milestones, deliverables, and acceptance criteria for your inference or serving-systems research area: literature scope, experiment matrix, benchmark design, and manuscript timeline.
Initial research paper draft
A 6 to 8 page scaffold with the research questions framed, the method outlined, related work surveyed, and the experiment setup defined, so you never start from a blank page.
Curated paper reading list
12 to 15 papers chosen for your topic, with a reading order, key takeaways, and the connections between them, plus a literature matrix template.
Starter code template
A clean, documented codebase for an inference-systems research project: model loading, a serving harness for vLLM and SGLang, benchmark and trace collection, evaluation, and experiment config. Ready to run on day one.
Example research topics
Your roadmap is personalised to your background and goals. These are the kinds of topics the kit is built for.
Scheduling policies for mixed prefill and decode workloads
KV cache compression, quantization, and eviction strategies
Radix-cache and prefix-cache hit rates on agentic workloads
Speculative decoding drafters for domain-specific models
Disaggregated prefill and decode across heterogeneous GPUs
Quantization accuracy versus latency on small and mid-sized models
Cache-aware routing for multi-replica serving
Serving mixture-of-experts models with expert parallelism on few GPUs
Build your workshop
Select what you need. Everything adjusts instantly.
Step 1: choose your program
Step 2: or pick a bundle and save
Select a program to get started.
EMI available at checkout. All sales are final.
Learn vLLM and SGLang from the source code to a production endpoint.
Twelve live lectures, a benchmark harness and deployments you build yourself, and recordings you keep.
Starts Tuesday, 10 November 2026 · Tuesdays and Thursdays, 9 to 11 am IST, for six weeks