Coding Agents — Localize, Edit, Verify

Highlights: Coding agents now fix real repository issues: Hybrid-Gym lifts SWE-bench Verified from 7% to 32.4%. Graham Neubig’s lecture in the CMU 11-768 AI Agents course (Fall 2026) examines the transition from one-pass code prompting to iterative agentic software engineering. We walk through the localize-edit-verify repair loop, diff format tradeoffs, synthetic task generation via mutation, and multi-harness reinforcement learning. We also cover multimodal frontend testing and expanding agent evaluation across the software development lifecycle.

• Iterative repair loop: Autonomous coding agents replace single-turn prompts with an iterative localize-edit-verify cycle guided by runtime execution feedback.
• Diff representation impact: Removing line-number coordinates from unified diffs eliminated arithmetic calculation failures and lifted GPT-4 Turbo refactoring pass rates from 20% to 61%.
• Synthetic task generation: SWE-smith procedurally injects defects into passing codebases to harvest inexhaustible, verified repair tasks with automated test oracles.
• Multi-harness robustness: Training across multiple scaffolding frameworks prevents models from overfitting to specific prompt formats and tool schemas.
• Lifecycle task scope: Because developers spend only 15% of their workday writing code, benchmarks must expand to cover continuous integration repair, test synthesis, and library generation.

Tutorial Overview:

1. Transitioning from Single-Step Prompts to Iterative Agentic Coding
2. SWE-agent Loop: Localize, Edit, Verify
3. Fixed-Workflow Alternatives: The Agentless Pipeline
4. Shell-Only Toolsets and Command-Line Manipulation
5. File Editing Paradigms and Diff Format Performance
6. Codebase Localization and Graph-Based Navigation
7. SWE-bench Pipeline and Test Harness Evaluation
8. Synthetic Task Scaffolding and Procedural Code Mutation
9. Multi-Language Scaffolding and Multi-Harness Robustness
10. Frontend Agent Workflows and UI Bug Resolution
11. Visual Verification and Multimodal Evaluation Benchmarks
12. Software Lifecycle Tasks and Developer Time
13. Advanced Engineering Benchmarks Across Lifecycle Roles
14. Synthetic Project Generation for Practice Environments
15. Cross-Domain Skill Transfer and Agent Architecture
16. Summary

---

1. Transitioning from Single-Step Prompts to Iterative Agentic Coding

In short
– Coding agents replace one-pass generation with an iterative inspect, edit, and test cycle.
– Models must balance functional correctness with non-functional constraints like speed, security, and maintainability.
– Reinforcement learning from execution test feedback scales accuracy through expanded thinking token budgets.

1.1 Shift to Iterative Agent Cycles

Coding moves from single-step completions to dynamic cycles driven by runtime diagnostics.

Approach	Context Scope	Execution Step	Diagnostic Feedback
Prompt completion	Single prompt	One forward pass	None
Agentic coding	Multi-file repository	Iterative tool calls	Dynamic runtime signals

• Inline comments: models emit code comments as an implicit reasoning buffer during planning.
• Reward model: post-training balances human readability against syntactic correctness using specialized quality classifiers.
• Multi-file edits: agents inspect repository structure, apply focused file modifications, and observe tool outputs.
• Localize–edit–verify loop: an iterative cycle where an agent inspects files to find faulty logic, applies a code modification, and tests the change against automated test suites.

Code repair proceeds through three steps:
1. Localize faulty logic across repository files.
2. Edit source code to satisfy behavioral specifications.
3. Verify modifications against automated test suites.

Fig 1. Iterative localization, modification, and test verification resolve codebase defects from runtime diagnostics.

In the lecture’s running example, a client defaults retry counts incorrectly when a caller explicitly passes zero. The agent localizes the fallback assignment, inserts an explicit check, and executes tests to verify the fix.

1.2 Reasoning and Non-Functional Requirements

Algorithmic problems require deductive reasoning rather than memorized pattern matching.

• 0 to 10^12: calculating integer sums exposes severe flaws across naive reasoning pathways.
• Loop iteration: summing integers sequentially exhausts execution time limits on large inputs.
• Formula recall: memorizing closed-form equations risks sign errors like n(n - 1) // 2.
• Endpoint pairing: deriving n(n + 1) // 2 guarantees efficiency while verifying base cases 0 and 1.
• NoFunEval benchmark: evaluates code optimization tasks without altering external program behavior.
• DiffBLEU score: quantifies latency and resource changes by comparing edits against reference solutions.
• CodeQL analysis: verifies code maintainability and security standards through automated static checks.

Fig 2. Endpoint pairing derives an efficient closed-form equation, avoiding slow loops and erroneous memory recall.

Systems must satisfy operational constraints alongside functional accuracy. Unit tests confirm outputs while static analysis catches maintainability flaws.

1.3 Reinforcement Learning from Execution Feedback

Post-training on programmatic test outcomes produces verifiable problem-solving policies.

• Binary rewards: isolated test suites evaluate generated programs to supply automated training signals.
• LiveCodeBench gains: execution feedback trains models to replace brute-force guessing with principled derivations.
• 16K to 32K: doubling response token limits grants models computational room for complex proofs.
• 64K tokens: expanding test-time evaluation budgets matches top frontier model accuracy without retraining.

Fig 3. Evaluation accuracy improves consistently as reinforcement learning and test-time token budgets expand.

Execution feedback removes human labeling bottlenecks entirely. Direct compiler signals force models to verify edge cases internally before emitting code.

---

2. SWE-agent Loop: Localize, Edit, Verify

In short
– Agentic coding uses an iterative loop to localize, edit, and verify repository defects.
– Targeted unit tests expose semantic root causes like improper falsy value handling.
– Test execution provides immediate validation and scalar rewards for policy training.

2.1 Structure of the Repair Loop

Neubig contrasts single-step coding prompts with agentic repository repair across an iterative tool cycle.

• SWE-agent loop: SWE-agent runs a localize–edit–verify loop (iterative inspection, editing, and automated test execution).
• Dynamic recovery: test failure outputs guide next iterations until candidate patches pass.
• Agentless alternative: Chunqiu Steven Xia et al. design a fixed, non-iterative repair workflow.
• Single-step tuning: Agentless suited older models, but lacks dynamic feedback recovery.

Fig 4. Iterative tool execution localizes bugs, applies edits, and verifies fixes against tests.

1. Localize: inspect files, navigate dependencies, and isolate faulty code.
2. Edit: modify target lines to implement the repair.

3. Verify: execute automated tests or linters to check regressions.

4. Predict files needing modification.

5. Identify target classes and functions.
6. Select exact lines to edit.
7. Synthesize patch via single-step language model.

System	Mechanics	Patch generation	Feedback recovery
SWE-agent	Iterative loop	Multi-turn tool calls	Dynamic test failure feedback
Agentless	Fixed sequential phases	Single-step LM	Lacks dynamic recovery

Iterative feedback enables agents to recover from errors. Fixed pipelines cannot adjust when tools fail.

2.2 Bug Localization and Root Cause

Agents trace observed symptoms back through the codebase to pinpoint and reproduce the underlying defect.

• Running example: a client defaults retry counts incorrectly when a caller explicitly passes zero.
• Localization tool: rg locates return config.get("retries") or 3 in buggy_config.py.
• 3 != 0 failure: running unittest reproduces the bug via AssertionError: 3 != 0.
• 0 evaluates to false: Python falsy zero evaluation mistakenly selects default value 3.
• Targeted edit: set default 3 only when the value is explicitly None.

$ rg 'retries' buggy_config.py
return config.get("retries") or 3
$ RETRY_MODULE=buggy_config python3 \
  -m unittest test_config.RetryTests.test_zero
AssertionError: 3 != 0
FAILED (failures=1)

# Before
return config.get("retries") or 3

# After
value = config.get("retries")
return 3 if value is None else value

Explicit checks preserve valid zero values. Missing and None inputs still receive their intended defaults.

2.3 Verification and Execution Feedback

Automated test suites confirm defect repair and guarantee that modifications introduce no regressions.

• 4 unit tests passed: missing and None yield 3, zero yields 0, positive yields 2.
• Granular feedback: test execution reports per-condition status across all tested inputs.
• Iterative revision: agents use test failure traces to revise faulty code hypotheses.
• Reinforcement learning: pass-or-fail test signals provide scalar rewards to train repair policies.

$ python3 -m unittest -v test_config
missing   → 3   PASS
None      → 3   PASS
zero      → 0   PASS
positive  → 2   PASS
Ran 4 tests: OK

Automated checks provide objective correctness signals. They guide model updates without requiring manual code review.

---

3. Fixed-Workflow Alternatives: The Agentless Pipeline

In short
– Agentless replaces autonomous agent loops with rigid hierarchical localization and single-step patching.
– Structured phases prevent compounding errors common in free-form tool use.

3.1 Fixed Pipelines and Agentless Architecture

Chunqiu Steven Xia introduced Agentless [A fixed, non-iterative workflow that isolates repair into sequential localization steps and single-step patch generation without dynamic agent loops.] to eliminate interactive tool loops.

1. Predict candidate files to edit.
2. Predict relevant classes and functions within candidate files.
3. Select exact lines requiring repair.

• Patch generation: a single-step model prompt generates the final repair patch directly.
• Eliminated feedback: the pipeline discards interactive tool feedback and autonomous execution loops.
• Running example: locating the client directly fixes zero-retry handling without interactive shell navigation.

Confining models to isolated localization phases prevents compounding errors common in free-form tool use.

3.2 Why Fixed Pipelines Outperformed Agents

Early benchmarks favored structured pipelines over autonomous tool-using agents.

Feature	Agentless Pipeline	Autonomous Agent
Workflow	Fixed sequential phases	Iterative tool loops
Training match	Single-step code generation	Multi-turn tool calling
Error risk	Mitigates compounding errors	Unguided loop failures

• Benchmark advantage: fixed pipelines outperformed early tool-using agents on repository repair.
• Training orientation: base models were trained heavily for single-step code resolution.
• Verification loops: avoiding unguided tool manipulation prevented compounding interactive errors.

Fig 5. Structured localization phases prevent compounding tool errors common in free-form agents.

Base models excelled at single-turn generation because training rarely optimized interactive tool calling.

---

4. Shell-Only Toolsets and Command-Line Manipulation

In short
– Interactive shell tools provide full repository access but trigger frequent command-line escaping errors.
– Simplified unified diffs eliminate line-number calculation errors, outperforming raw search-and-replace formats.
– Dedicated file editing and dependency-graph localization tools improve autonomous bug repair accuracy.

4.1 Coding via Shell-Only Environments

Autonomous agents inspect repositories, modify source files, and execute test suites through an interactive bash terminal.

• Terminal interface: the harness captures terminal output, exit statuses, and execution timeout signals during command execution.
• Interactive shell primitive: mini-SWE-agent solves SWE-bench [A benchmark evaluating coding agents on real-world GitHub issues by executing isolated test suites against candidate repository patches] via bash commands.
• Inspection utilities: agents scan source trees using cat, find, grep, and rg (ripgrep).
• Editing commands: models alter files by choosing between sed, awk, ad-hoc Python scripts, and patch.
• Verification commands: agents run automated validation checks using pytest or repository build scripts.
• Debugging tools: unrestricted terminal access allows models to execute interactive utilities like GDB.

Fig 6. A single bash interface handles code navigation, stream editing, and automated test execution.

Terminal strings often break on unbalanced quotes or regular expression backslashes during routine code modifications. Specialized tools shield models from syntax idiosyncrasies and conserve reasoning tokens.

4.2 Editing Code and Localizing Defects

Structured editing formats and dependency navigation tools prevent common terminal errors during repository modifications.

sed 's/^RETRIES = 3$/RETRIES = 5/' \
  settings.py > settings.new
mv settings.new settings.py

1. Run stream substitutions matching regex patterns into a temporary file.
2. Move the temporary file to overwrite the original target script.

Edit Style	Context Cost	Line Numbers Required	Primary Failure Mode
Whole-file rewrite	High	No	Context exhaustion
Standard diff	Low	Yes	Hunk offset calculation
Simplified diff	Low	No	Repeating pattern ambiguity

• Unified diff [A structured format displaying code modifications alongside surrounding context lines, often simplified by removing line numbers to avoid model calculation errors]: line-numbered hunks like @@ -1,2 +1,2 @@ trigger model calculation errors.
• Simplified diff formats: Aider and Codex omit line coordinates to prevent calculation failures.
• Search-and-replace ambiguity: repeated lines like retries = 3 require surrounding context for unique matching.
• 20% -> 61%: GPT-4 Turbo refactoring score on 89 tasks; Gemini drops on mismatched schemas.
• Code localization: tracing caller-callee relationships finds bugs when retries default incorrectly on explicit zero.
• Graph-based localization: LocAgent links config.py modules, functions, and unit tests across dependency edges.

Omitting line numbers prevents patch calculation errors without sacrificing necessary local modification context. Graph navigation connects modules directly to tests, eliminating full-repository context dumps.

---

5. File Editing Paradigms and Diff Format Performance

In short
– Whole-file overwrites repeat code, whereas search/replace blocks need context lines to disambiguate.
– Omitting line counts from diffs prevents frequent model arithmetic errors.
– Simplified unified diffs increased GPT-4 Turbo refactoring pass rates from 20% to 61%.

5.1 Editing Formats and Patch Blocks

Agents modify repository code using whole-file overwrites, snippet replacements, or diff patch commands.

Editing Format	Mechanism	Tradeoff
Whole-file overwrite	Replaces full file	Repeats unchanged code
Search/replace block	Matches snippet lines	Needs surrounding context lines
Patch command	Specifies line edits	Line numbers cause errors

• Unified diff: structured format (displays modifications alongside context lines, omitting fragile line counts).
• Search/replace blocks: exchange targeted snippets, requiring surrounding context lines to disambiguate identical matches.
• Patch blocks: update code via boundary markers without fragile line-count accounting.

--- a/settings.py
+++ b/settings.py
@@ -1,2 +1,2 @@
-RETRIES = 3
+RETRIES = 5
 TIMEOUT = 30

*** Begin Patch
*** Update File: settings.py
@@
-RETRIES = 3
+RETRIES = 5
 TIMEOUT = 30
*** End Patch

Models routinely miscalculate line counts during editing. Stripping line numbers prevents automated patch application failures.

5.2 Diff Formats and Benchmark Accuracy

Editing syntax strongly affects model capabilities on complex code refactoring tasks.

• 89 Python tasks: evaluated GPT-4 Turbo moving class methods to top-level functions in Aider.
• Preserved syntax: required methods to convert correctly while preserving valid code execution.
• 20% success rate: scored by GPT-4 Turbo when using search/replace editing blocks.
• 61% success rate: scored when using a simplified unified diff without line counts.
• Pre-training distribution: models encounter unified diffs far more frequently across pre-training corpora.

Models succeed more often when editing formats match pre-trained distributions. Unfamiliar replacement blocks cause avoidable syntax and parsing errors.

Fig 7. Simplified unified diffs triple refactoring success by matching pre-trained code distributions.

---

6. Codebase Localization and Graph-Based Navigation

In short
– Codebase search traces bug symptoms directly to faulty logic without reading every file.
– Dependency graph navigation adds overhead without outperforming plain keyword search and test checks.

6.1 Finding Context with Search Tools

Agents run ripgrep (rg 'retries') or grep to trace bug symptoms without reading entire repositories.

1. Ingest input: locate where configuration values enter the system.
2. Trace defaulting: find the logic where zero incorrectly becomes three.
3. Identify tests: locate relevant unit checks in test_config.py.

Targeted searches isolate code lifecycles immediately. Tracing symptoms avoids filling model context windows with unrelated files.

6.2 Navigating Code via Dependency Graphs

LocAgent traverses structural dependency paths across connected files rather than relying solely on flat search.

• Trace definitions: inspect configuration declarations inside config.py.
• Track callers: follow downstream call paths across cli.py and client.py.
• Inspect checks: locate behavioral test assertions within test_config.py.
• Navigation overhead: Neubig notes graph traversal rarely outperforms keyword search and execution checks.

Fig 8. LocAgent traverses call graphs, but simple search combined with execution checks works better.

Graph traversal introduces significant tool overhead. Simple keyword filtering paired with execution feedback repairs bugs faster.

---

7. SWE-bench Pipeline and Test Harness Evaluation

In short
– SWE-bench tests repository patches against isolated pre-merge unit tests.
– Split test suites isolate bug repairs while preventing unintended functional regressions.
– Runnable execution environments enable reinforcement learning and verifier-based inference reranking.

7.1 Evaluating Tasks with Test Harnesses

SWE-bench (a benchmark evaluating coding agents on real-world GitHub issues by executing isolated test suites against candidate repository patches) tests code patches.

• Pre-merge snapshots: creators pair pre-merge codebases with issue descriptions and developer-written pull request tests.
• Binary scoring: patches earn a 1 if all target tests pass and 0 otherwise.
• Test harness: the test harness (the runtime environment and test runner responsible for executing code checks to award pass-or-fail evaluation signals) isolates checks from the agent harness (the scaffolding framework that structures an agent’s prompt templates, tool definitions, context management, and operational loop).
• 5 to 10 minutes: full repository test runs take too long per candidate patch.
• Targeted subsets: evaluation extracts small test slices to keep evaluation tractably fast.

Test Category	Pre-patch State	Post-patch Target	Primary Purpose
Fail-to-pass	Fails on buggy code	Must pass	Confirms targeted bug repair
Pass-to-pass	Passes on buggy code	Must continue passing	Prevents functional regressions

In the lecture’s running example, a client defaults retry counts incorrectly when a caller explicitly passes zero. Fail-to-pass tests (test checks that fail on the buggy codebase and must pass after patching to confirm the targeted defect was repaired) check explicit zero handling. Pass-to-pass tests (pre-existing test checks that pass before patching and must continue passing to ensure a change causes no functional regressions) ensure standard positive retry counts still succeed.

Runnable benchmark tasks require three packaged components:
1. Starting repository state containing source code, pinned dependencies, and isolated execution commands.
2. Issue description stating reproduction steps, requested behavior, and expected outcomes.
3. Automated test checks that return deterministic binary signals.

Automated grading requires hermetic environments to prevent spurious failures. Baseline checks confirm failures stem strictly from candidate patches.

Fig 9. Isolated evaluation containers test candidate patches against fail-to-pass and pass-to-pass subsets.

7.2 Training and Inference Compute Scaling

Executable software engineering environments unlock policy optimization through direct unit test feedback.

• Objective terminal rewards: models update weights directly from binary unit test pass-fail signals.
• Multi-step trajectories: agents explore directories, edit source files, and execute tool commands iteratively.
• 500 environments: fine-tuning a 32B model on 500 runnable SWE-Gym environments beat its 11% baseline.
• Inference-time scaling: sampling multiple trajectories and reranking with learned verifiers outperforms greedy generation.
• December 2024 to June 2026: SWE-Rebench continuously extracts freshly merged pull requests to prevent leakage.

Multi-step interactions teach agents effective repository navigation. Fresh benchmark splits reflect contemporary libraries and prevent test contamination.

---

8. Synthetic Task Scaffolding and Procedural Code Mutation

In short
– Training coding agents requires containerized environments with deterministic automated test checks.
– Procedural code mutation generates endless verified repair tasks from existing test suites.

8.1 Scalable Environment Creation

Real pull requests evaluated on SWE-bench (evaluating coding agents on GitHub issues with isolated test suites) scale slowly.

• December 2024 through June 2026: SWE-rebench harvests pull requests across this timeframe to prevent benchmark contamination.
• 3 core elements: configure starting codebases, issue descriptions, and automated execution test checks.
• Pre-validated containers: verifying environments beforehand avoids uninformative trajectories and noisy reinforcement learning rewards.
• 500 verified tasks: SWE-Gym training boosted models from an 11% baseline and enabled reranking.

Fig 10. Runnable training environments require codebases, issue descriptions, and automated execution checks.

Human pull request volume limits training scale, forcing environments to rely on synthetic task generation. Pre-validating container environments ensures execution checks produce reliable reinforcement learning reward signals.

8.2 Procedural Bug Injection via Mutation

SWE-smith synthesizes repair tasks by inverting mutation testing (procedurally injecting code defects to harvest verified repair tasks).

1. Prepare baseline: verify that all pre-existing unit tests pass on the codebase.
2. Mutate code: procedurally alter codebase source files while preserving the original test suite.
3. Filter tasks: keep instances where previously passing tests fail to produce bug reports.

value = config.get("retries")
return 3 if value is None else value

value = config.get("retries")
return 3 if not value else value

• Running example defect: in the lecture’s running example, a client defaults retry counts incorrectly when passing zero.
• 4 baseline tests: all four unit tests pass cleanly before procedural bug injection.
• 3 of 4 tests pass: integer zero evaluates to falsy, triggering the default fallback erroneously.
• All 4 tests restored: fixing the defaulting rule allows all four unit tests to pass cleanly.

Procedural code mutations supply inexhaustible verified training tasks with automated ground-truth test oracles. Using existing test suites eliminates human curation bottlenecks in reinforcement learning data pipelines.

---

9. Multi-Language Scaffolding and Multi-Harness Robustness

In short
– Multi-step RL code repair optimizes trajectories, but sparse terminal rewards complicate credit assignment.
– Agents read root configuration files to adapt across Python, TypeScript, and Rust repositories.
– Training across multiple agent harnesses and synthetic mutations prevents interface and prompt overfitting.

9.1 Reinforcement Learning for Code Repair

Reinforcement learning frames repository repair as a sequential decision process over trajectory actions.

1. Search: locate relevant source files like config.py using repository inspection tools.
2. Edit: modify code logic, such as preserving explicit zero values.
3. Test / finish: execute test suites to verify repairs and check for regressions.

[
\max_\theta \mathbb{E}{\tau \sim \pi\theta} [R(\tau)]
]

• R2E-Gym: optimizes policy parameters (\theta) across sampled multi-step repair trajectories (\tau).
• Terminal reward: the environment provides intermediate observations but returns only final execution feedback.
• Credit assignment (identifying the failing edit): sparse terminal feedback obscures which intermediate tool call failed.

Terminal test rewards conceal intermediate execution errors. Agents cannot easily pinpoint single flawed code edits across trajectories.

9.2 Handling Multi-Language Repositories

Agents inspect repository configuration files to determine project directory structures and test runners.

Language	Config file	Directories	Test command
Python	pyproject.toml	src/, tests/	pytest
TypeScript	package.json	src/, tests/	npm test
Rust	Cargo.toml	src/, tests/	cargo test

• Multi-SWE-bench: evaluates coding agents across diverse programming language ecosystems.
• SWE-bench Multilingual: tests repository repairs against varied package managers and conventions.
• Tool adaptation: agents parse project configuration files before calling build and test commands.

Directory conventions and test runners differ across languages. Reading configuration files ensures agents invoke correct project tooling.

9.3 Multi-Harness Training Prevents Overfitting

Training across varied scaffolding frameworks stops agents from memorizing specific prompts and tool signatures.

• SWE-smith: generates synthetic training data using mutation testing (procedurally injecting deliberate defects).
• In the running example: a client defaults retry counts incorrectly when callers pass zero.
• 4 pass -> 3 pass, 1 fail: mutating value is None to not value fails zero-retry tests.

# Working baseline: 4 pass
value = config.get("retries")
return 3 if value is None else value

# Mutated condition: 3 pass, 1 fail
value = config.get("retries")
return 3 if not value else value

• Agent harness (framework structuring prompts, tools, and loops): single harnesses cause interface and prompt overfitting.
• >= 2 harnesses: NVIDIA Nemotron models train across five harnesses per task distribution.
• Polar and multimodal tasks: Polar links OpenHands, OpenCode, and Codex; SWE-bench Multimodal tests front-end issues.

Fig 11. Multi-harness training shares model parameters across diverse agent environments to prevent overfitting.

Agents easily memorize single-harness prompt formats. Multi-harness training forces models to learn general repair logic instead.

---

10. Frontend Agent Workflows and UI Bug Resolution

In short
– Frontend agents resolve repository UI defects by pairing code edits with live browser execution.
– Verification combines deterministic Playwright scripts with adaptive autonomous browser exploration agents.
– Visual inspection catches rendered layout bugs that internal state checks miss.

10.1 Visual UI Bug Resolution

SWE-bench Multimodal (a benchmark evaluating coding agents on real-world GitHub issues) repairs existing repository interfaces rather than greenfield code.
– Visual inputs: inspects problem screenshots, component layout, font sizing, and visual error banners.
– Interactive verification: confirms dynamic browser behavior, interactive event handling, and DOM updates.
– Visual repair loop: runs a localize–edit–verify loop (iterative fault localization, code editing, and test verification).
– 0 length inputs: modifies value.length to value.trim().length, rejecting blank form submissions with errors.
– GLM-5.2 arena lead: topped Front-End Design Arena without vision, though visual layout inspection remains critical.

Fig 12. Agent frontend loop localizing UI components, editing logic, and verifying in live browsers.

Visual verification detects rendered layout flaws that internal assertions miss. Browser execution ensures DOM events trigger state updates properly.

10.2 Browser Automation and Testing Patterns

Developers verify UI repairs using deterministic Playwright automation scripts or adaptive autonomous browser exploration agents.
– Playwright scripting: executes deterministic interaction sequences to verify state changes and visibility.
– Autonomous browser agents: cycle continuously through observation and action phases to navigate interfaces.
– Interactive exploration: inspects DOM state and display to uncover unexpected visual anomalies.

1. Populate input fields using fill() with test whitespace characters.
2. Trigger form submissions and event handlers using click().
3. Validate application state changes and UI visibility with expect().
4. Capture a screenshot() for manual review or baseline comparisons.

Pattern	Execution Style	UI Interaction	Primary Verification Role
Playwright Scripts	Deterministic scripted steps	Fixed sequential calls	Repeatable regression testing
Browser Agents	Dynamic observation-action cycles	Adaptive interface exploration	Uncovering unexpected anomalies

Deterministic scripts guarantee reliable regression passes along known paths. Autonomous exploration agents discover unexpected edge-case failures across complex dynamic pages.

---

11. Visual Verification and Multimodal Evaluation Benchmarks

In short
– Deterministic Playwright scripts pair programmatic state checks with rendered screenshots.
– Multimodal benchmarks grade patches via execution and regression checks, not pixel similarity.

11.1 Assertion-based frontend verification

Playwright scripts verify frontend repairs by combining deterministic user actions with programmatic assertions.
1. fill(): enters test inputs like whitespace strings into targeted form fields.
2. click(): triggers user events such as pressing the submit button.
3. expect(): validates DOM state changes and confirms database records remain zero.
4. screenshot(): captures the rendered browser interface for visual inspection against baselines.

Fig 13. Programmatic assertions and visual inspection verify interactive frontend repairs together.

Scripting browser actions prevents flaky tests from autonomous agent variance during verification. Testing value.trim().length ensures whitespace submissions correctly fail instead of creating blank records.

11.2 Evaluating visual issues beyond pixels

Benchmarks grade patches using behavioral execution instead of raw image pixel similarity.
– Multimodal inputs: provide an issue description, a problem image, and starting repository files.
– SWE-bench Multimodal (a benchmark evaluating agent patches): runs repairs in a live browser test harness (runtime environment executing test checks).
– Execution checks: verify the defect is repaired so whitespace input displays visible errors.
– Regression checks: ensure valid submissions still save and all pre-existing tests continue passing.

Raw pixel matching penalizes harmless visual rendering differences across operating systems. Behavioral test suites isolate genuine functional bug fixes from non-functional styling noise.

---

12. Software Lifecycle Tasks and Developer Time

In short
– Developers spend only 15% of their working hours writing or reading code.
– Software engineering requires managing both the inner loop and outer loop.
– Agent benchmarks must expand from isolated repairs to full lifecycle tasks.

12.1 Where developers spend their time

Meyer et al. showed developers spend most working hours coordinating, debugging, and testing rather than writing code.

• 15% of workday: writing or reading code and tests according to Meyer et al.
• 25% of workday: attending meetings and processing email communications.
• 14% of workday: debugging software and fixing defects.
• 11% of workday: helping teammates, syncing up, and general networking.
• 8% of workday: running automated test suites.
• GLM 5.2: won Front-End Design Arena despite lacking multimodal image processing.

Fig 14. Developers spend most workday hours on collaboration, debugging, and testing rather than typing code.

Agents focused purely on code generation miss most developer responsibilities. Automating auxiliary workflows yields greater productivity gains.

12.2 Expanding agents across the lifecycle

Agents must expand beyond immediate code fixes to automate the complete software development lifecycle.

Workflow	Focus	Typical activities
Inner loop (immediate implementation cycle)	Code changes	Reproducing failures, editing files, running tests
Outer loop (broader lifecycle beyond direct coding)	System operations	Requirements, code review, CI, deployment

• Outer loop bottlenecks: less structured specifications and scarce training trajectories hinder agent learning.
• Production release needs: diff reviews, operational telemetry, and rollback paths exceed simple unit test passes.
• Broader benchmarks: synthesizing libraries from scaffolds, generating discriminative tests, and repairing CI workflows.
• Multi-issue milestones: maintaining repositories across multi-issue tasks tests long-term coherence.
• CodeScout: a reinforcement learning framework that trains agents on sub-tasks like localization.

Passing isolated tests does not guarantee operational safety in production. Agents must master complete lifecycle coordination to deliver reliable real-world systems.

---

13. Advanced Engineering Benchmarks Across Lifecycle Roles

In short
– Developers spend only 15% of their working time actively writing or modifying code.
– Specialized benchmarks evaluate localization, test synthesis, continuous integration repair, and whole-library implementation.
– Multi-milestone benchmarks test whether autonomous agents sustain long-term repository quality without regressions.

13.1 Comparative Lifecycle Benchmarks

Developers dedicate most work hours to collaborative and operational duties beyond the inner loop [the immediate, day-to-day implementation cycle of reproducing bugs, editing source files, and running test suites].

• 15% coding: developers spend only this fraction actively writing or editing source code.
• 25% communication: time consumed by meetings and email correspondence.
• 27% governance: time spent on code review, administrative planning, and breaks.
• 14% debugging: time devoted to active debugging and bug fixing.
• 11% coordination: time spent on helping, sync-ups, and networking.
• 8% pipeline operations: time allocated to running tests and builds.

Fig 15. Evaluation tasks across engineering artifacts, verification environments, and multi-step planning horizons.

Benchmarks must expand beyond single-file patches to reflect realistic engineering distributions. Agents require distinct operational competencies for library synthesis, continuous integration, and long-horizon evolution.

13.2 Localization, Test Generation, and CI Repair

Modular benchmarks isolate discrete maintenance skills rather than requiring full repository modifications at once.

• F1 overlap rewards: CodeScout trains agents to pinpoint candidate files, modules, and functions without code edits.
• Mutation testing validation: SWT-Bench requires agents to synthesize discriminative unit tests using mutation testing [a method of procedurally injecting deliberate defects into code while checking against existing test suites to harvest verified repair tasks].
• Discriminative test behavior: tests must fail on the unfixed baseline and pass once repaired.
• Running example defect: in the lecture’s running example, a client defaults retry counts incorrectly when a caller explicitly passes zero.
• Log-driven environment diagnosis: LCA CI Builds Repair tests fixing pipeline crashes such as an EBADENGINE failure when Node 16 runs a Node $\ge 18$ app.

# setup-node configuration
with:
  node-version: '20'

Fig 16. Continuous integration repair benchmarks diagnose runtime logs and patch pipeline configuration files.

Isolating single operational stages simplifies reinforcement learning rewards. Subtask evaluation measures precise failure modes before running unconstrained agent loops.

13.3 App Modification, Libraries, and Evolution

Broad-scope benchmarks assess full system generation, binary reverse engineering, and multi-version repository evolution.

1. Zero-to-One: ViBench tasks agents with building a complete application from a product specification.
2. Vibe-on-Ref: agents extend an existing reference minimum viable product with new features.
3. Vibe-on-Vibe: agents iteratively expand an application previously synthesized by the agent itself.

• Browser-based evaluation: ViBench validates progressive full-stack application updates against human-authored browser test plans.
• Black-box library synthesis: ProgramBench agents probe uninstrumented binaries with inputs, observe outputs, and reimplement packages from scratch.
• Scaffolded module completion: Commit0 and ProgramBench test filling interface stubs like $\text{parse}(\dots)$ and $\text{evaluate}(\dots)$ to pass package test suites.
• Iterative milestone maintenance: SWE-Milestone assesses sequential feature milestones while preventing architectural decay and backwards compatibility regressions across releases.
• Automated benchmark gyms: SWE-Playground and Hybrid-Gym procedurally generate environments to train multi-role agents.

Iterative milestones ensure patches do not degrade structural maintainability. Evaluating whole packages reveals errors that single-file issue benchmarks overlook.

---

14. Synthetic Project Generation for Practice Environments

In short
– SWE-Playground procedurally generates verified coding environments from scratch.
– A six-stage pipeline builds projects, tasks, and test suites automatically.
– Synthetic practice expands agent skills without manual repository curation bottlenecks.

14.1 Procedural Project Generation

Neubig presents SWE-Playground (Zhu, Gandhi & Neubig) to generate complete practice environments from scratch.

1. Project proposal: defines the repository concept and high-level architecture.
2. Task proposal: creates specific engineering problems and feature requests.
3. Repository setup: builds the directory structure and dependency files.
4. Unit test generation: writes automated test suites for verification.
5. Functionality implementation: writes baseline code implementing requested features.
6. Capability-specific adaptation: adjusts challenge difficulty for targeted training goals.

Procedural creation replaces scarce real-world repositories with reproducible, automated environments.

14.2 Scalable Environment Generation

Playgrounds expand agent training across broader software engineering activities.

• Zero curation bottlenecks: generates tasks procedurally without scraping public repositories.
• Executable verification: provides automated test suites for instant code feedback.
• Benchmark readiness: builds problem-solving skills before downstream benchmark evaluations.

Fig 17. Synthetic playgrounds provide scalable task generation with automatic execution feedback.

Synthetic practice allows agents to build capabilities across diverse engineering challenges.

---

15. Cross-Domain Skill Transfer and Agent Architecture

In short
– Modular sub-skill training transfers across unfamiliar repositories and diverse tasks.
– Hybrid-Gym boosts a 32B model across repair, testing, and library building.
– Complete systems unite execution rewards, editing loops, and lifecycle tools.

15.1 Acquiring Modular Engineering Skills

Agents learn reusable programming sub-skills rather than monolithic end-to-end patch generation.

• Targeted sub-skills: agents search code dependencies, implement functions from specs, or author executable tests.
• Training methods: models train using supervised fine-tuning or reinforcement learning.
• SWE-Playground: generates synthetic environments for deliberate coding practice.
• Budget-matched testing: held-out evaluations test generalizability under identical inference compute budgets.

Modular practice lets an agent locate retry logic without needing full issue supervision. Specialized sub-tasks generalize better than end-to-end issue repairs.

15.2 Transferring Across Development Tasks

Multi-task training environments allow models to transfer skills across distinct software challenges.

• Hybrid-Gym: trains a 32B parameter model on diverse development tasks beyond issue repair.
• 7% to 32.4%: success jump on SWE-bench Verified [a benchmark evaluating coding agents on real-world GitHub issues] over baseline.
• SWT-Bench Verified: evaluates cross-task transfer gains on automated test generation.
• Commit0 Lite: assesses multi-file repository generation for entire library building.

Fig 18. Hybrid-Gym elevates SWE-bench Verified issue resolution from 7% to 32.4%.

Fig 19. Multi-task fine-tuning lifts accuracy across test generation and library building.

15.3 Architectural Synthesis of Complete Agents

Designing practical coding agents requires combining models, scaffolds, and verification environments.

• Localize–edit–verify loop: inspects files, modifies lines, and tests changes against automated test suites.
• Execution rewards: code models train directly against test suite execution signals.
• Agent harness: scaffolding framework structuring prompts, tools, context management, and loop state.
• Frontend verification: visual checks validate user interfaces alongside unit test suites.
• Outer loop coverage: extends repairs to continuous integration failures and code maintenance.
• World models: internal simulations predict code execution paths and runtime errors.

---

16. Summary

• Iterative repair loop: Agents resolve defects by repeatedly localizing logic, editing code, and verifying runtime test execution.
• Simplified diff editing: Removing line numbers from unified diffs prevents calculation errors and improves refactoring accuracy.
• Procedural synthetic scaffolding: Mutating codebases yields verified training data, while multi-harness practice prevents interface overfitting.
• Multimodal visual verification: Browser automation and visual inspection uncover interface defects that internal state assertions miss.
• Full lifecycle engineering: Multi-task training across CI repair and library synthesis transfers robust developer capabilities.

Mastering iterative execution feedback, diverse tooling harnesses, and full lifecycle workflows transforms language models into reliable autonomous software engineers.

---

Source

Lecture: Agents for Coding and Software Development by Graham Neubig, Carnegie Mellon University.
Course: CMU 11-768 AI Agents (Fall 2026), Lecture 6, second half. The first half is covered in LLM_log #036.
Video: YouTube · Slides: PDF · Course page: cmu-agents.com

Further reading referenced in the lecture:

• Singhal et al., 2024 · NoFunEval · 397 code-editing tasks
• Luo et al., 2025 · DeepCoder · execution-reward RL
• Interaction-loop concepts: SWE-agent · original teaching diagram
• Xia et al. · Agentless, original overview figure
• mini-SWE-agent · shell-only interface and execution model
• Aider · formats
• OpenHands
• Pi
• OpenCode
• Aider · unified diffs
• Codex · parser
• OpenCode · selection
• OpenHands · GPT-5 preset
• Chen et al., 2025 · LocAgent
• Jimenez et al. · SWE-bench evaluation pipeline, original source figure
• SWE-Gym · executable training environments
• Jain et al., 2025 · R2E-Gym
• Yang et al., 2025 · SWE-smith, original pipeline figure
• Multi-SWE-bench
• SWE-bench Multilingual
• Nemotron 3 Super
• Nemotron 3 Ultra
• Polar
• Harness implementations
• Yang et al., 2024 · SWE-bench Multimodal · browser and screenshot tools for SWE-agent
• Koh et al., 2024 · VisualWebArena
• Playwright actions
• Assertions
• Screenshot comparisons
• Meyer et al., 2019 · Table 2 · 5,928 self-reported Microsoft workdays
• CodeScout, 2026
• ViBench · collection and evaluator definitions checked 2026-09-07
• Commit0
• ProgramBench
• SWE-Milestone, 2026
• Mündler et al., 2024 · SWT-Bench
• LCA CI Builds Repair · dataset revision ebf12dad
• Zhu, Gandhi & Neubig · SWE-Playground
• Hybrid-Gym, 2026 · selected 32B results

---

dataHacker.rs — LLM_log #037: Agentic Software Engineering — Toolsets, Harnesses, and Environments. By Vladimir Matic.