Learning Skills from Experience — Inducing Reusable Workflows and Code Skills for AI Agents Highlights: Daniel Fried’s lecture in Carnegie Mellon University’s CMU 11-768 AI Agents course (Fall 2026) examines how autonomous agents discover, verify, and maintain reusable skills across sequential web tasks. We walk through the representational trade-offs between natural language workflows and executable Python tools, automated verification and sandboxing, failure-driven strategy induction, and usage-based memory pruning. Task compression: Converting atomic browser interactions into…
Read more

Beyond the Context Window — Authored Skills and External Memory for AI Agents Highlights: Daniel Fried’s lecture in CMU 11-768 AI Agents (Fall 2026) examines how autonomous agents extract reusable operational skills and manage structured memory across multi-step tasks. We walk through the AgentSkills specification standard, progressive disclosure mechanisms that protect context windows, and benchmarked gains on SkillsBench. Finally, we explore tiered storage in MemGPT, memory reconciliation in Mem0, and trajectory reflection frameworks like Reflexion…
Read more

Long Context in Practice — Training Recipes (RoPE, YaRN, NoPE), KV and Prompt Caching, and Context Compaction Highlights: In this second part of the long-context lecture from CMU’s 11-768 AI Agents course (Fall 2026, Graham Neubig), the focus shifts from model architecture to everything that surrounds it: how long-context models are trained, served, and kept affordable. We walk through staged short-to-long training pipelines, positional encoding choices from RoPE interpolation to NoPE, the serving stack built…
Read more

Why Agent Context Explodes — Quadratic Growth, Serving Costs, and Efficient Attention (Linear, DeltaNet, KDA, MLA) Highlights: Graham Neubig’s lecture in the CMU 11-768 AI Agents course (Fall 2026) looks at why agent context grows quadratically and what architectures make long context affordable to serve. We walk through the token composition of real OpenHands sessions, the prefill/decode split and its serving metrics, and the local, linear, sparse and KV-compression mechanisms used in current frontier models.…
Read more

Exposing and Evaluating Tools — REST APIs, MCP, Parallel Tool Calls, and Tool-Use Benchmarks Highlights: This session of CMU’s 11-768 AI Agents course (Fall 2026, Graham Neubig) moves tool calling out of the notebook and into the network, tracing the path from in-process Python libraries to REST endpoints, MCP servers, and the serving stacks underneath them. We walk through wrapping tools with FastAPI and Pydantic, the two-tier credential split that MCP introduces, concurrent execution with…
Read more

How Language Models Call Tools — Tool Primitives, CodeAct, Chat Templates, and Constrained Decoding Highlights: Graham Neubig’s lecture on tool use in the CMU 11-768 AI Agents course (Fall 2026) treats tools as the interface through which a language model invokes an external program, and code as a meta-tool that collapses long chains of single API calls into one action. We walk through why agents need tools, the five foundational tool primitives, the CodeAct paradigm…
Read more

Agent Capabilities and Harness Engineering — Why Agents Are Systems, Not Just Models Highlights: In Graham Neubig’s segment of CMU 11-768: AI Agents (taught with Daniel Fried), the argument is that assembling an agent is easy while making one work reliably is not. We walk through the capabilities that separate the two — tool calling, long-context coherence, customizability, task management, environment understanding, and safety — and the two levers available for each: harness engineering and…
Read more

What Are AI Agents and How Do They Work? — From Tool Calls to the ReAct Loop Highlights: The opening lecture of CMU’s 11-768: AI Agents (Fall 2026), taught by Daniel Fried and Graham Neubig, sets out what an agent actually is and where current systems break down. We walk through the anatomy of an agent — environment, observations, actions, reward — and follow the path from plain next-token prediction to tool calls, chat templates,…
Read more

Fame and Uniqueness: Measuring What Makes a Cover Memorable Highlights: Romaniuk’s Distinctive Brand Asset framework (2018) splits brand memorability into two axes — Fame (exposure history) and Uniqueness (visual distinctiveness) — but only one of them is recoverable from a picture. We build a full measurement pipeline on top of Gemini 2.5 Pro: genuine pairwise visual-distinctiveness judgments aggregated with the Bradley–Terry model (not a single holistic LLM call), a 13-family literature-grounded absolute-scoring taxonomy, and real…
Read more

Visual Complexity Without an LLM Judge — Head-to-Head Against Gemini Highlights: LLM/VLM pairwise judges have a structural position bias — the first-shown candidate wins more often than it should, worst-case exactly on close calls. We replace the LLM judge with a transparent, CPU-only alternative: handcrafted complexity features + logistic regression on the signed difference \( \mathbf{d} = f(A) – f(B) \). The same fit gives you a per-image score \( \text{score}(x) = \mathbf{w} \cdot f(x)…
Read more

Highlights: In this post we try to answer one practical question: can we score the visual complexity of an image without humans in the loop? We grab the SAVOIAS dataset (1,420 images with human-rated complexity), focus on the 200 advertisements, and try five different recipes — zero-shot CLIP prompts, Ridge regression on CLIP embeddings, pairwise ranking on embedding differences, kNN retrieval on cosine similarity, and finally Gemini 2.5 Pro as a chain-of-thought judge. The headline:…
Read more

Highlights: An image is just a matrix — but the Transformer eats sequences of vectors. The whole “vision” trick of ViT lives in how that matrix is turned into a sequence. We cut a 224×224 image into a fixed 14×14 grid of 16×16 patches, flatten each, and project it through ONE learned Linear layer. Patches are not sampled — every cell of the grid becomes a token, in fixed order. We prepend a learnable [CLS]…
Read more