Skip to content
>_devvkit
$devvkit learn --librarie ollama-guide

Ollama Guide

[llm][local][inference][cli]
AI / LLM Tools
Install
# macOS
brew install ollama
# Linux
curl -fsSL https://ollama.com/install.sh | sh
# Windows: download from ollama.com
# Docker:
docker run -d -v ollama:/root/.ollama -p 11434:11434 ollama/ollama

Ollama wraps llama.cpp and other runtimes into a dead-simple CLI. `ollama pull llama3.2` downloads a 3B-parameter model and `ollama run llama3.2` starts an interactive chat. Models are cached in `~/.ollama` and served on `localhost:11434`.

The REST API is OpenAI-compatible — any tool that works with OpenAI's API can use Ollama by changing the base URL to `http://localhost:11434/v1`. This means you can use Ollama as a drop-in replacement for OpenAI with Cursor, Continue.dev, Open Interpreter, and more.

Ollama supports GGUF models from Hugging Face. Use `Modelfile` to customize system prompts, temperature, and context length. Pull models in parallel with `ollama pull llama3.2 & ollama pull mistral &`. GUI: Ollama Web UI (open-webui), ChatGPT-style interface with local models.

Basic Usage

Pull and run— Download and chat.
ollama pull llama3.2            # Download model (3B params)
ollama run llama3.2              # Interactive chat
ollama run llama3.2 "Explain quantum computing"  # One-shot prompt
ollama run mistral "Write a poem"                # Mistral model

Model Management

List models— Show downloaded models.
ollama list                     # Show all downloaded models
# Output: NAME, ID, SIZE, MODIFIED
ollama rm llama3.2               # Delete a model
ollama cp llama3.2 my-model       # Copy/rename model
Pull multiple models— Parallel downloads.
# Download several models at once:
ollama pull llama3.2 &
ollama pull mistral &
ollama pull codellama:7b &
ollama pull nomic-embed-text &
wait
echo "All models downloaded!"

REST API

REST API — chat— OpenAI-compatible endpoint.
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.2",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Hello!"}
    ]
  }' | jq '.choices[].message.content'
REST API — generate— Simple text completion.
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Why is the sky blue?",
  "stream": false
}' | jq -r '.response'
OpenAI compatibility— Use with any OpenAI client.
import openai

client = openai.OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama"  # Required but not checked
)

response = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Hello"}]
)
print(response.choices[0].message.content)

Modelfile

Custom Modelfile— Custom system prompt and params.
# Modelfile
FROM llama3.2
# Set parameters
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER num_ctx 4096       # Context window
# System prompt
SYSTEM "You are a helpful coding assistant. Provide concise answers with code examples."

# Build:
ollama create my-coder --file Modelfile
ollama run my-coder

Advanced

Run with GPU— Enable GPU acceleration.
# Check GPU support:
ollama run llama3.2 --verbose  # Shows if GPU is used

# Force GPU layers:
ollama run llama3.2 --num-gpu-layers 999

# CPU only:
OLLAMA_NO_GPU=1 ollama run llama3.2

# Set concurrent requests:
OLLAMA_NUM_PARALLEL=4 ollama serve
Server mode— Run as background service.
ollama serve                    # Start server (daemon by default)
# Custom port:
OLLAMA_HOST=0.0.0.0:8080 ollama serve
# Allow network access:
OLLAMA_HOST=0.0.0.0 ollama serve  # Listen on all interfaces

# Set model directory:
export OLLAMA_MODELS=/mnt/bigdrive/ollama
ollama serve