Local LLMs: Ollama, vLLM, and Self-Hosted AI

Descrizione del corso

The complete guide to self-hosted LLMs. Covers model formats (GGUF, GGML, safetensors), quantization techniques (GPTQ, GGUF Q4/Q8, AWQ), Ollama setup and modelfile customization, vLLM for high-throughput serving, llama.cpp compilation and optimization, LM Studio for desktop use, OpenAI-compatible API servers, model benchmarking (MMLU, HumanEval), privacy-first RAG with local models, and hardware sizing for 7B/13B/70B models.

Cosa imparerai

• Curated from public resources. Run large language models locally and on private servers with Ollama, vLLM, LM Studio, and llama.cpp.

Programma del corso

1. GGUF, GGML, safetensors: model format guide (28 min)
2. Quantization: GPTQ, AWQ, GGUF Q4/Q5/Q8 tradeoffs (35 min)
3. Ollama setup: models, Modelfile, and API (30 min)
4. Ollama custom models with Modelfile FROM/SYSTEM (25 min)
5. Open WebUI: ChatGPT-like interface for local models (22 min)
6. vLLM: PagedAttention and continuous batching (38 min)
7. OpenAI-compatible server with vLLM (20 min)
8. Hardware sizing: GPU VRAM requirements per model size (25 min)

Struttura del corso

3 sezioni, 8 lezioni, 11h 0m di durata complessiva. Il corso si segue interamente online, al proprio ritmo.

Domande frequenti