$devvkit learn --librarie vllm-guide

vLLM Guide

[llm][inference][gpu][serving]

AI / LLM Tools

Install

pip install vllm
# or: uv add vllm
# Requires: NVIDIA GPU with CUDA, or AMD ROCm
# Docker:
docker run --gpus all -p 8000:8000 vllm/vllm-openai:latest

vLLM is the production-grade LLM serving engine. Its key innovation is PagedAttention — managing KV cache in non-contiguous memory blocks (like OS paging), eliminating fragmentation and enabling 2-4x more concurrent requests than naive implementations.

vLLM serves an OpenAI-compatible API out of the box. It supports continuous batching (group incoming requests into optimal batches), tensor parallelism (spread one model across multiple GPUs), and prefix caching (reuse KV cache for common prefixes).

Start with `vllm serve meta-llama/Llama-3.2-3B-Instruct`. Key flags: `--tensor-parallel-size 4` (4 GPUs), `--max-model-len 8192` (reduce for speed), `--gpu-memory-utilization 0.9` (how much VRAM to use). vLLM requires significant GPU memory — not for consumer GPUs with large models.

Basic Serving

Start server— Serve a model.

vllm serve meta-llama/Llama-3.2-3B-Instruct
vllm serve mistralai/Mistral-7B-Instruct-v0.3 --port 8000
vllm serve TheBloke/Llama-2-7B-GGUF  # GGUF also supported

OpenAI API

Chat completion— OpenAI-compatible chat.

from openai import OpenAI

client = OpenAI(base_url='http://localhost:8000/v1', api_key='none')

response = client.chat.completions.create(
    model='meta-llama/Llama-3.2-3B-Instruct',
    messages=[
        {'role': 'system', 'content': 'You are a code assistant.'},
        {'role': 'user', 'content': 'Write a Python function to reverse a linked list'}
    ],
    temperature=0.7,
    max_tokens=1024
)
print(response.choices[0].message.content)

Streaming response— Real-time token output.

from openai import OpenAI
client = OpenAI(base_url='http://localhost:8000/v1', api_key='none')

stream = client.chat.completions.create(
    model='meta-llama/Llama-3.2-3B-Instruct',
    messages=[{'role': 'user', 'content': 'Write a haiku'}],
    stream=True
)
for chunk in stream:
    print(chunk.choices[0].delta.content or '', end='')

Batching & Performance

Clients: performance test— Load test the server.

# Install benchmarks:
pip install aiohttp

# Python benchmark script:
vllm serve meta-llama/Llama-3.2-3B-Instruct &
python benchmark_serving.py \
  --backend vllm \
  --model meta-llama/Llama-3.2-3B-Instruct \
  --request-rate 10 \
  --num-prompts 200

Multi-GPU

Multi-GPU (tensor parallel)— Split model across GPUs.

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4
# Requires 4 GPUs with enough VRAM
# Works with any model > GPU memory

# Pipeline parallel (for even larger models):
  --pipeline-parallel-size 2 --tensor-parallel-size 4

Advanced

Prefix caching— Reuse KV cache for common prefixes.

vllm serve meta-llama/Llama-3.2-3B-Instruct \
  --enable-prefix-cache
# Speeds up: chatbot with long system prompts
# or few-shot examples repeated across requests

# Also: automatic prefix detection based on
# hash of previous tokens

Quantization in vLLM— Use AWQ/GPTQ quantized models.

# AWQ quantization (best quality/speed trade-off):
vllm serve TheBloke/Llama-2-7B-AWQ --quantization awq

# GPTQ:
vllm serve TheBloke/Llama-2-7B-GPTQ --quantization gptq

# FP8 (Hopper GPUs only):
vllm serve meta-llama/Llama-3.2-3B-Instruct --dtype half

Speculative decoding— Use draft model for speed.

# Speeds up large model inference by 2-3x
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --speculative-model meta-llama/Llama-3.2-3B-Instruct \
  --num-speculative-tokens 5

# Draft model runs on the same GPU(s) —
# predicts tokens, target model verifies