Long Context in Practice — Training Recipes (RoPE, YaRN, NoPE), KV and Prompt Caching, and Context Compaction Highlights: In this second part of the long-context lecture from CMU’s 11-768 AI Agents course (Fall 2026, Graham Neubig), the focus shifts from model architecture to everything that surrounds it: how long-context models are trained, served, and kept affordable. We walk through staged short-to-long training pipelines, positional encoding choices from RoPE interpolation to NoPE, the serving stack built…
Read more

Why Agent Context Explodes — Quadratic Growth, Serving Costs, and Efficient Attention (Linear, DeltaNet, KDA, MLA) Highlights: Graham Neubig’s lecture in the CMU 11-768 AI Agents course (Fall 2026) looks at why agent context grows quadratically and what architectures make long context affordable to serve. We walk through the token composition of real OpenHands sessions, the prefill/decode split and its serving metrics, and the local, linear, sparse and KV-compression mechanisms used in current frontier models.…
Read more

Exposing and Evaluating Tools — REST APIs, MCP, Parallel Tool Calls, and Tool-Use Benchmarks Highlights: This session of CMU’s 11-768 AI Agents course (Fall 2026, Graham Neubig) moves tool calling out of the notebook and into the network, tracing the path from in-process Python libraries to REST endpoints, MCP servers, and the serving stacks underneath them. We walk through wrapping tools with FastAPI and Pydantic, the two-tier credential split that MCP introduces, concurrent execution with…
Read more

How Language Models Call Tools — Tool Primitives, CodeAct, Chat Templates, and Constrained Decoding Highlights: Graham Neubig’s lecture on tool use in the CMU 11-768 AI Agents course (Fall 2026) treats tools as the interface through which a language model invokes an external program, and code as a meta-tool that collapses long chains of single API calls into one action. We walk through why agents need tools, the five foundational tool primitives, the CodeAct paradigm…
Read more

Agent Capabilities and Harness Engineering — Why Agents Are Systems, Not Just Models Highlights: In Graham Neubig’s segment of CMU 11-768: AI Agents (taught with Daniel Fried), the argument is that assembling an agent is easy while making one work reliably is not. We walk through the capabilities that separate the two — tool calling, long-context coherence, customizability, task management, environment understanding, and safety — and the two levers available for each: harness engineering and…
Read more

What Are AI Agents and How Do They Work? — From Tool Calls to the ReAct Loop Highlights: The opening lecture of CMU’s 11-768: AI Agents (Fall 2026), taught by Daniel Fried and Graham Neubig, sets out what an agent actually is and where current systems break down. We walk through the anatomy of an agent — environment, observations, actions, reward — and follow the path from plain next-token prediction to tool calls, chat templates,…
Read more

Fame and Uniqueness: Measuring What Makes a Cover Memorable Highlights: Romaniuk’s Distinctive Brand Asset framework (2018) splits brand memorability into two axes — Fame (exposure history) and Uniqueness (visual distinctiveness) — but only one of them is recoverable from a picture. We build a full measurement pipeline on top of Gemini 2.5 Pro: genuine pairwise visual-distinctiveness judgments aggregated with the Bradley–Terry model (not a single holistic LLM call), a 13-family literature-grounded absolute-scoring taxonomy, and real…
Read more

Visual Complexity Without an LLM Judge — Head-to-Head Against Gemini Highlights: LLM/VLM pairwise judges have a structural position bias — the first-shown candidate wins more often than it should, worst-case exactly on close calls. We replace the LLM judge with a transparent, CPU-only alternative: handcrafted complexity features + logistic regression on the signed difference \( \mathbf{d} = f(A) – f(B) \). The same fit gives you a per-image score \( \text{score}(x) = \mathbf{w} \cdot f(x)…
Read more

Highlights: In this post we try to answer one practical question: can we score the visual complexity of an image without humans in the loop? We grab the SAVOIAS dataset (1,420 images with human-rated complexity), focus on the 200 advertisements, and try five different recipes — zero-shot CLIP prompts, Ridge regression on CLIP embeddings, pairwise ranking on embedding differences, kNN retrieval on cosine similarity, and finally Gemini 2.5 Pro as a chain-of-thought judge. The headline:…
Read more

Highlights: An image is just a matrix — but the Transformer eats sequences of vectors. The whole “vision” trick of ViT lives in how that matrix is turned into a sequence. We cut a 224×224 image into a fixed 14×14 grid of 16×16 patches, flatten each, and project it through ONE learned Linear layer. Patches are not sampled — every cell of the grid becomes a token, in fixed order. We prepend a learnable [CLS]…
Read more

Highlights: Jason Weston traces the arc from early neural language models to self-improving LLMs that generate their own training data and evaluate their own reasoning System 1 vs System 2: fixed-compute pattern-matching vs deliberate multi-step reasoning — and why the same LLM implements both Chain-of-Thought prompting: adding “Let’s think step by step” jumps GSM8K accuracy from ~10% to 40–50%; few-shot CoT hits 90%+ on MultiArith CoVe + S2A: Chain-of-Verification reduces hallucinations 3× on knowledge list…
Read more

Highlights: Yu Su’s guest lecture in the UC Berkeley CS294-280 course argues language agents are not “LLM + tools” but a new evolutionary stage of machine intelligence. We walk through the agent-first framing and three concrete research pillars — long-term memory (HippoRAG), implicit reasoning (Grokked Transformers), and model-based planning (WebDreamer) — that map directly to classical AI problems re-examined through the lens of LLMs. Agent-first framing: token generation is itself an action; the inner monologue…
Read more