Category: LLMs from Scratch

LLM_log #008: CLIP — Understanding Multimodal AI Through Step-by-Step Experiments

13.02.2026

  Highlights: In this post, you’ll learn how CLIP connects images and text in a shared embedding space — enabling zero-shot image classification, semantic search, and visual perception scoring without any task-specific training. We start from the ground up with Vision Transformers, walk through CLIP’s contrastive learning architecture, run hands-on embedding experiments, and then push CLIP to its limits with a real-world challenge: can it tell cheap bedrooms from expensive ones using actual house sale…
Read more

LLM_log #007: From Random Text to Coherent Language – Pretraining Your First Large Language Model

11.02.2026

Highlights: In this guide, you’ll learn how to pretrain a large language model from scratch — implementing training loops, evaluation metrics, and advanced text generation strategies. We’ll build a complete GPT-style training pipeline, watch it evolve from random gibberish to coherent text, and explore techniques like temperature scaling and top-k sampling. By the end, you’ll load professional pretrained weights into your own architecture. Source: This is part of our ongoing “Building LLMs from Scratch” series…
Read more

LLM_log #006: Implementing ChatGPT 2.0 from scratch – Rashchka

10.02.2026

Highlights: In this post, we build a complete GPT-2 model (124 million parameters) from scratch in PyTorch. We implement every component — layer normalization, GELU activations, the feed forward network, shortcut connections — and wire them into a transformer block that we stack 12 times to create the full architecture. By the end, you will have a structurally complete GPT model that can generate text token by token. We also weave in key insights from…
Read more

LLMs Scratch #004: Mixture of Experts (MoE) Models: The Architecture Powering 2025’s Best AI Systems

25.11.2025

🎯 What You’ll Learn This comprehensive guide takes you from MoE fundamentals to state-of-the-art implementations like DeepSeek V3. You’ll understand why sparse architectures outperform dense models at every compute scale, master the critical routing mechanisms that determine expert selection, and learn the training techniques that make these complex systems work. We’ll examine real benchmark results from Llama 4, Grok, and DeepSeek, explore load balancing challenges and solutions, and walk through the complete evolution of DeepSeek’s…
Read more

LLMs from Scratch #003: Modern Transformer Architectures: A Deep Dive into Design Principles and Training

23.11.2025

Modern Transformer Architectures: A Deep Dive into Design Principles and Training 🎯 What You’ll Learn In this comprehensive guide, we’ll explore the evolution of transformer architectures from the original “Attention is All You Need” paper to modern implementations. You’ll discover why today’s language models use specific design choices like RoPE position embeddings and SwiGLU activations, understand the trade-offs between serial and parallel layer arrangements, and learn how to make informed decisions about hyperparameters like head…
Read more

©  2026 Master Data Science. Built using WordPress and the Mesmerize Theme