Training Code LLMs — Data, Infilling, Tests

Highlights: Train only on code and MATH scores collapse from 33.2 to 10.3. The right data mix, infilling and test feedback make code models work. Graham Neubig’s lecture in CMU 11-768 AI Agents (Fall 2026) outlines the technical foundations required to build autonomous code generation systems and repository-level software engineers. We walk through the progression across three capability tiers, data curation and whitespace-preserving tokenization, bidirectional causal infilling, and evaluation frameworks spanning static code representations to execution-guided reinforcement learning.

• Capability tiers: Neubig categorizes coding systems across three operational levels: single-file autocompletion, multi-file repository modification, and full software development lifecycle management.
• Tokenization and data curation: Specialized byte-pair encoding compresses whitespace runs to preserve indentation syntax, while multi-stage pipelines filter credentials, licenses, and benchmark contamination.
• Bidirectional infilling: Sentinel tokens reorder prefix and suffix contexts into standard autoregressive sequences, allowing causal language models to synthesize code edits without architectural modifications.
• Multi-faceted evaluation: Program assessment combines static structural metrics like CodeBLEU and CodeBERTScore with isolated sandboxed execution and non-functional efficiency tests.
• Execution-guided RL: Binary test feedback and execution timeouts incentivize reinforcement learning policies to discover computationally optimal algorithms rather than brute-force solutions.

Tutorial Overview:

1. Progression from Autocompletion to Repository-Level Agents
2. Data Sourcing, Selection, and Cleaning
3. Tokenization and Corpus Scaling Across Languages
4. Data Mixtures and Context Window Extension
5. Infilling Objectives and Advanced Learning Paradigms
6. Reference-Based and Representation-Level Metrics
7. Dynamic Execution and Functional Correctness
8. Assessing Non-Functional Requirements
9. Reinforcement Learning with Execution Feedback
10. Summary

---

1. Progression from Autocompletion to Repository-Level Agents

In short
– Coding agents advance across three tiers from isolated completions to entire project lifecycles.
– Models require language mastery, bidirectional context editing, and verifiable reasoning paths.
– Three training stages combine balanced data mixtures, custom tokenizers, and execution feedback.

1.1 Three Tiers of Coding Capability

Neubig categorizes modern software engineering agents into three escalating levels of operational capability.

Capability Tier	Scope	Typical Input	Primary Action
Single completion	Standalone file	Isolated prompt, CSV	Generates scripts or plots
Repository modification	Multi-file codebase	Issue report, directory tree	Edits modules and dependencies
Full development	Software lifecycle	Feature specification	Manages CI, tests, and builds

• 3 capability tiers: single completion, multi-file repository modification, and full software development lifecycle management.
• Single completion: generates a self-contained function, script, or isolated chart from an input prompt.
• CSV to visualization: single-step tasks prompt models to transform raw CSV data into bar charts.
• Repository modification: inspects directories, traces imports, and edits multiple files without breaking module dependencies.
• Full software development: handles environment setup, test generation, continuous integration, and version control.
• Developer work distribution: software engineers spend most time debugging, reviewing pull requests, and configuring builds.

Fig 1. Coding capabilities progress from isolated code completions to full software development lifecycles.

Real-world software engineering requires maintaining existing systems rather than authoring isolated scripts. Autonomous agents must navigate multi-file dependencies to resolve realistic project issues.

1.2 Core Components of Coding Agents

Three coordinated elements enable autonomous systems to execute iterative problem-solving loops across codebases.

• 3 agent components: prompt instructions, external execution tools, and the underlying large language model.
• The prompt: specifies task requirements, output formatting rules, and high-level operational instructions.
• The tools: provide shell interfaces, syntax search utilities, file editing commands, and test runners.
• The language model: supplies underlying semantic comprehension, program synthesis, and logical reasoning.
• Technical bottleneck: tool design and model grounding matter far more than prompt engineering alone.
• Iterative problem-solving: agents navigate directories, formulate hypotheses, apply edits, and execute test suites.

Prompts alone cannot resolve complex software bugs without deep program structure understanding. Models must interpret runtime failure traces directly within their iterative execution loops.

1.3 Single-Step Code Generation Requirements

Single-step code generation measures program synthesis from prompts without interactive environment feedback.

• 3 core competencies: programming language mastery, bidirectional editing capability, and specification reasoning.
• Language mastery: understands concrete syntax, standard library APIs, and idiomatic programming patterns.
• Bidirectional editing: conditions on preceding prefix code and succeeding suffix code when modifying implementations.
• Program reasoning: translates specifications into executable computational logic:
  [ \text{specification} \to \text{behavior} ]
• Verifiable parity check: Neubig illustrates this with a simple parity check function, is_positive(x).
• Verifiable semantics: execution against automated test assertions provides objective verification of intermediate reasoning steps.

Verifiable test feedback makes programming an ideal domain for evaluating multi-step reasoning capabilities. Most production frontier models undergo intensive code training to sharpen logical deduction.

1.4 The Three-Stage Training Pipeline

Building competitive coding models requires a deliberate progression across three successive training stages.

1. Pre-training: learns syntax, natural language reasoning, and software semantics over broad token distributions.
2. Mid-training (A continual pre-training stage that adapts an already trained general language model to code by feeding it large code mixtures and longer sequences): exposes models to curated code mixtures and extends sequence context lengths.
3. Reinforcement learning post-training: optimizes reasoning trajectories and implementation accuracy using automated execution rewards.

Fig 2. Specialized coding models evolve across pre-training, mid-training, and reinforcement learning stages.

Base language models do not automatically acquire complex repository engineering capabilities from web text alone. Staged pipelines systematically adapt models from general text comprehension to execution-grounded reasoning.

1.5 Pre-Training Mixture and Objectives

Models optimize autoregressive objectives over heterogeneous token mixtures balancing technical depth and language fluency.

• Autoregressive language modeling (A training process where a model learns by predicting each subsequent token based on the tokens that came before it): optimizes context prediction likelihood:
  [ \mathcal{L}{\text{LM}} = -\mathbb{E}{x \sim \mathcal{D}{\text{mix}}} \sum_t \log p\theta(x_t \mid x_{<t}) ]
• 5 data sources: synthesizes web text, technical prose, mathematics, source code, and API documentation.
• Web text: establishes broad general knowledge and conversational English fluency.
• Technical prose and math: explains algorithms and systems while reinforcing structured symbolic reasoning.
• Code and documentation: demonstrates syntax, implementation idioms, and explicit functional usage patterns.

Training exclusively on source code damages natural language comprehension and instruction-following abilities. Balanced multi-domain mixtures ensure agents comprehend both developer instructions and technical codebases.

1.6 Tokenization and Tool Grounding

Specialized tokenizers and execution environments preserve code structure and prepare models for agent tooling.

• Whitespace preservation: standard text whitespace normalization breaks indentation-sensitive languages like Python.
• Byte-pair encoding (BPE) (A subword tokenization method that compresses common text and whitespace patterns into single token units): learns dedicated tokens for repeated spaces to prevent sequence inflation.
• 4-space indentation tokens: compressed into single token units to preserve structural hierarchy.
• Boundary preservation: preserves Markdown formatting and explicit file boundary markers across contexts.
• Operating system utilities: exposes models to grep, find, package managers, and directory hierarchies.
• Functional coding evaluation: monitors execution benchmarks alongside domain-specific held-out perplexity during pre-training.

Collapsing whitespace invalidates Python syntax because indentation directly defines programmatic execution blocks. Grounding models in shell utilities during pre-training ensures seamless interaction with agent harnesses later.

---

2. Data Sourcing, Selection, and Cleaning

In short
– Pre-training blends natural language, mathematical reasoning, and indentation-preserving source code.
– Multi-stage filtering purges autogenerated boilerplate, leaked credentials, proprietary licenses, and benchmark contamination.
– Code-specific tokenizers and deduplication compress whitespace and prevent security bug propagation.

2.1 Joint Code and Language Learning

Effective code models require pre-training corpora that unify natural language explanations with executable source code.

• Data mixture: combine web text, technical prose, mathematical derivations, source code, and API documentation.
• Whitespace preservation: retain exact indentation in functions like is_positive(x) instead of collapsing spaces.
• Objective loss: optimize autoregressive language modeling [A training process where a model learns by predicting each subsequent token based on the tokens that came before it] over mixed corpus $\mathcal{D}_{\mathrm{mix}}$ (Groeneveld et al., 2024).
• Dual tracking: monitor held-out validation loss across domains alongside periodic functional execution checks.

[\mathcal{L}{\mathrm{LM}} = -\mathbb{E}{x \sim \mathcal{D}{\mathrm{mix}}} \sum_t \log p\theta(x_t \mid x_{<t})]

Raw code lacks the communicative grounding needed to interpret developer instructions. Whitespace carries critical syntactic meaning in languages such as Python.

2.2 Filtering Corrupted Repository Data

Raw public repositories contain noise, compiler artifacts, and sensitive credentials that require multi-stage filtering protocols.

1. Source filtering: discard unsupported file extensions, non-UTF-8 encodings, and machine-generated files.
2. Content cleaning: eliminate compiler artifacts, minified scripts, and low-utility repetition.
3. Provenance recording: log open-source licenses, repository creation dates, and contributor opt-outs.
4. Secret detection: scrub private APIs, access tokens, and credentials using TruffleHog.
5. Decontamination: isolate repositories and problem families to protect downstream evaluation benchmarks.

Fig 3. Multi-stage filtering removes low-quality files, sensitive secrets, and benchmark contamination.

Commercial models risk copyright infringement if trained on proprietary code reproducing licensed logic. Repository-level isolation prevents benchmark score inflation caused by leaked utility functions.

2.3 Deduplication and Code Tokenization

Deduplication and specialized subword tokenization prevent severe context bloat and repetitive security vulnerabilities.

• Duplicate removal: purge identical files, vendored directories, and near-duplicate implementations from GitHub forks.
• Indentation compression: merge contiguous whitespace runs and newlines into single unified tokens.
• Identifier isolation: separate whitespace tokens from alphanumeric characters to avoid partial-word fragmentation.
• Subword tokenization: apply byte-pair encoding (BPE) [A subword tokenization method that compresses common text and whitespace patterns into single token units] to preserve verbatim formatting.
• 80+ languages: training StarCoderBase (Li et al., 2023) across languages improved pass@1 [pass@k: A metric that calculates the probability that at least one program passes all unit tests when sampling a given number of candidate solutions] on high-resource languages.
• Generation bias: models default to Python for general requests, but select React for dashboards.

Standard tokenizers assign separate tokens to each space, wasting context window capacity. Cloned repositories overfit models to boilerplate and replicate known security flaws.

---

3. Tokenization and Corpus Scaling Across Languages

In short
– Code tokenizers compress whitespace runs while strictly preserving indentation scope and syntactic boundaries.
– Multilingual pre-training across 1T tokens steadily increases pass@1 accuracy across 80+ programming languages.
– Models exhibit strong inductive biases toward Python and React unless explicitly constrained.

3.1 Code-Aware Tokenization and Whitespace Preservation

Standard tokenizers collapse spaces, but code models require syntax-bearing indentation to preserve execution scope.

Tokenizer Type	Whitespace Handling	Indentation Preservation	Autocompletion Safety
Standard text	Collapses space runs	Drops syntax scope	Merges across boundaries
Code-aware BPE	Compresses space runs	Preserves indent blocks	Isolates whitespace tokens

• Byte-pair encoding (BPE): BPE (A subword tokenization method that compresses common text and whitespace patterns into single token units) trains on code and technical prose.
• Indentation as syntax: Python uses whitespace indentation to define code blocks and execution scope.
• Run compression: tokenizers compress consecutive spaces and indentation blocks into specialized single tokens.
• StarCoder2 tokenization: splits if · ready : ↵ · · · · return · value ↵ into if | · ready | : | ↵ · · · | · return | · value | ↵.

if ready:
    return value

• Indent token representation: compresses newline and three indentation spaces into ↵ · · ·, attaching leading spaces to identifiers.
• Boundary isolation: isolating whitespace avoids partial-word syntax errors and preserves multi-word autocompletion.

Merging characters across spaces creates syntax errors when generation stumbles. Preserving whitespace boundaries prevents frequent phrases like import numpy as np from forming monolithic tokens.

3.2 Multilingual Scaling and Compute Allocation

Massive multilingual pre-training scales functional programming capabilities monotonically across compute budgets and token counts.

• 15.5B parameters, 1T tokens: StarCoderBase trains across 80+ programming languages from diverse developmental artifacts.
• Four data sources: mixes GitHub source code, Git commits, GitHub issues, and Jupyter notebooks.
• Corpus sanitization: teams deduplicate forks, vendored libraries, and copied solutions, while TruffleHog filters API keys.
• License compliance: curators track open-source licenses and provide opt-out mechanisms to prevent copyright infringement claims.
• Monotonic pass@1 gains: pass@k (A metric that calculates the probability that at least one program passes all unit tests when sampling a given number of candidate solutions) rises with compute.
• Cross-lingual transfer: heavily represented languages achieve higher absolute performance while shared vocabularies lift low-resource languages.

Fig 4. Pre-training compute steadily increases pass@1 performance across more than 80 programming languages.

Compute scaling expands syntactic mastery across diverse runtimes. Shared subword vocabularies transfer representations from high-resource languages into low-resource codebases.

3.3 Large-Scale Multilingual Code Corpus Pre-Training

Pre-trained code models exhibit persistent inductive biases that favor specific programming languages when prompts lack explicit constraints.

• Python default: models overwhelmingly write Python when given ambiguous instructions like “write an app to process data”.
• React frontend exception: models default almost universally to the React JavaScript library when prompted for a dashboard.
• Neubig pipeline experiment: an optimized Python pipeline converted to Rust repeatedly reverted to Python on helper scripts.
• Persistent prompt constraints: enforcing Rust required repeatedly appending explicit “in Rust” constraints across every interaction turn.
• Three bias factors: syntactic conciseness, corpus dominance in public repositories, and execution alignment in Python runtimes drive defaults.

Models minimize generation risk by choosing dominant training distributions. Python provides compact syntax and matches standard reinforcement learning execution environments.

---

4. Data Mixtures and Context Window Extension

In short
– Mid-training on balanced code, text, and math mixtures preserves general reasoning abilities.
– Dependency graphs and structured file ordering enable effective repository-level context learning.
– Causal masked infilling improves both bidirectional editing and standard code generation.

4.1 Balancing Code, Text, and Math

Mid-training adapts general base models to coding through targeted code and technical text mixtures.

• Mid-training [continual pre-training stage adapting models to code via code mixtures and longer sequences]: Code Llama continues Llama 2 pre-training.
• 500 billion tokens: added across 7B, 13B, and 34B parameter variants.

Fig 5. Mid-training adapts pre-trained base checkpoints with specialized code mixtures.

• 100/0/0 mixture: code-only data triggers severe performance degradation across MATH and MMLU benchmarks.
• 70/20/10 mixture: balances code, text, and math to safeguard general reasoning.

Fig 6. Balanced code, text, and math mixtures protect critical multi-step reasoning performance.

Software engineering requires natural language specifications and multi-step algorithmic reasoning. Dropping text and math ruins these essential capabilities.

4.2 Managing Cross-File Repository Context

Context extension allows coding models to inspect definitions and shared structures across multi-file repositories.

• 1 million tokens: maximum supported context length, far below enterprise monorepos spanning billions of lines.
• 8K to 32K tokens: context window scaled during Qwen2.5-Coder mid-training.
• 128K tokens: context window reached using YaRN positional interpolation.

Structured multi-file training sequence order:
1. Group related files that reside within the same directory.
2. Trace import statements and repository dependency graphs to link definitions with usages.
3. Order dependent files sequentially, placing types.py before worker.py and caller.py.

Fig 7. Dependency ordering packs multi-file repository context into manageable training sequences.

Concatenating entire repositories exceeds training compute budgets. Structured dependency ordering resolves cross-file references cleanly.

4.3 Bidirectional Code Infilling

Infilling trains models to predict masked code spans using both surrounding prefix and suffix context.

• Code infilling [synthesizing missing code using preceding and following context]: resolves ambiguities invisible to left-to-right generation.
• is_positive(x) example: def is_positive(x: int) -> ____: requires looking ahead to return x > 0.
• InCoder causal masking: conditions code generation symmetrically on both prefix and suffix tokens.
• 52 billion tokens: causal masking achieves higher pass@1 [test pass probability] on HumanEval and MBPP.

Fig 8. Causal masked infilling improves standard left-to-right generation pass rates across benchmarks.

Conditioning on future context exposes structural representations across entire files. This bidirectional training strengthens standard left-to-right generation.

---

5. Infilling Objectives and Advanced Learning Paradigms

In short
– Sentinel tokens turn bidirectional code infilling into standard autoregressive sequence modeling without changing architectures.
– Causal masking beats left-to-right generation on edits and improves normal left-to-right synthesis benchmarks.
– Commit diffs, compiler diagnostics, and execution traces provide real-world supervision for software repair agents.

5.1 Sentinel-based autoregressive code infilling

Standard language models generate code left to right, but software engineering requires editing and infilling existing code files.

• Edit operations: developers routinely update type declarations, insert intermediate logic, and refactor existing functions.
• Prefix ambiguity: Neubig illustrates this with a simple parity check function, is_positive(x), where signatures omit types.
• Suffix proof: the return type remains unknown until inspecting the downstream suffix return x > 0.
• InCoder formulation: Fried et al. cast bidirectional code infilling [Synthesizing a missing segment of code in the middle of a file by looking at both the preceding code and following code.] into standard autoregressive language modeling [A training process where a model learns by predicting each subsequent token based on the tokens that came before it.].
• Sentinel markers: replaces removed code spans with a sentinel token [A special placeholder marker inserted into code to indicate where a missing piece of code was removed and where it needs to be generated.] denoted as [M0].

def is_positive(x: int) -> bool:
    return x > 0

Left-to-right conditioning fails when crucial syntax clues appear only later in the file. Rearranging text lets standard causal models read both contexts before generating edits.

Fig 9. Conditioning on prefix and suffix contexts enables exact return-type inference.

5.2 Rearranging documents for causal infilling

InCoder transforms training documents into causal sequences by shifting masked target spans to the document end.

• Sequence format: training formats text as prefix [M0] suffix [M0] span [END].
• Inference generation: prompts contain prefix [M0] suffix [M0] so the model emits the span until [END].
• Symmetric context: conditions causal attention across both the left prefix and the right suffix simultaneously.
• Single-span focus: pre-training samples single-span gaps most frequently, with two or three spans appearing under a long-tail distribution.
• Edit tasks: infilling handles missing function bodies, type inference, docstring generation, and identifier renaming across files.

Rearranging multi-gap documents follows four sequential ordering steps:

1. Insert [M0] at the first gap location and [M1] at the second gap location.
2. Concatenate prefix, intermediate context, and suffix together into one prompt.
3. Append [M0] followed by the first target code span.
4. Append [M1] followed by the second target code span.

Fig 10. Document rearrangement packs multiple non-contiguous gaps into one autoregressive training sequence.

Standard causal attention processes these rearranged sequences without requiring any modifications to the transformer architecture. Appending target spans at the end preserves left-to-right training while supplying bidirectional conditioning.

5.3 Evaluating bidirectional context in editing

Conditioning directly on right-hand context delivers higher functional correctness than left-to-right generation across code editing benchmarks.

• Single-pass synthesis: causal masking synthesizes full edits in one pass without candidate re-ranking.
• Single and multi-line: infilling evaluations on HumanEval test suites show large gains across gap sizes.
• 52 billion tokens: matched training budgets demonstrate causal masking improves both editing and standard left-to-right synthesis.
• 8.0% vs 6.0%: HumanEval pass@1 [A metric that calculates the probability that at least one program passes all unit tests when sampling a given number of candidate solutions.] for causal masking against standard models.
• 10.9% vs 8.9%: MBPP pass@1 for causal masking against standard left-to-right models.

Generation Approach	Right Context Available	Single-Pass Inference	Relative Synthesis Accuracy
Standard Left-to-Right	No	Yes	Baseline
Candidate Re-ranking	Suffix used for scoring	No (generates many)	Moderate
Causal Masked Infilling	Suffix used in prompt	Yes (direct generation)	Highest

Infilling objectives force the network to connect cross-span syntactic structures and data dependencies. Learning mid-file completions improves general program understanding across all generation directions.

Fig 11. Causal masking outperforms standard generation and expensive re-ranking across edit benchmarks.

5.4 Supervision from real development workflows

Beyond synthetic spans, models acquire practical software engineering skills directly from continuous integration artifacts and repository histories.

• Commit diffs: git version history pairs broken implementations with clean human patches.
• CI diagnostics: continuous integration pipelines capture compiler error messages, test failures, and stack traces automatically.
• Test rewards: automated test suites provide binary or scalar execution signals to guide reinforcement learning.
• Runtime states: execution traces supply line-by-line variable mutations, pointer aliasing, and active control branches.
• Code world models: learning execution dynamics lets models anticipate test outcomes before running external tools.

Fig 12. Four training signals beyond raw code: commit diffs, diagnostics, tests and execution traces.

Mining public pull requests pairs failed test logs with developer fixes at massive scale. Tracing runtime execution variables teaches models how source statements alter program state during execution.

---

6. Reference-Based and Representation-Level Metrics

In short
– Surface token overlap fails because trivial edits alter execution semantics.
– CodeBLEU augments n-grams with syntax trees and data-flow graphs.
– CodeBERTScore softly aligns contextual embeddings to capture functional equivalence.

6.1 Surface-Level Reference Matching

Reference matching compares model-generated code directly against human reference solutions without executing either program.

• BLEU-4 metric: measures token n-gram overlap alongside a brevity penalty for short code.
• Exact match: penalizes harmless variable renames and correct syntactic variations.
• Surface discrepancy: awards high scores to broken implementations matching reference phrasing.
• Parity check example: Neubig illustrates this with a simple parity check function, is_positive(x).

Candidate	Code	Overlap	Behavior
Reference	return x > 0	100%	Correct
Off-by-one	return x >= 0	High	Fails at 0
Semantic match	return not (x <= 0)	Low	Correct

Programming languages follow rigid execution semantics. Small textual changes break program logic, while different syntax can preserve behavior.

6.2 Structural Matching With CodeBLEU

CodeBLEU [An evaluation metric combining surface word overlap with syntax trees and data-flow graphs to measure code similarity] analyzes static structure without dynamic execution.

CodeBLEU = alpha * BLEU + beta * Weighted N-Gram + gamma * AST Match + delta * Data-flow Match

• 4 weighted components: balances BLEU, keyword n-grams, AST match, and data-flow match.
• Standard BLEU: measures surface token n-gram overlap between candidate and reference code.
• Weighted n-grams: weights critical language keywords higher than generic syntax tokens.
• AST [A tree representation of source code that captures its syntactic structure and grammatical hierarchy] match: checks candidate subtrees against reference grammar across functions.
• Data-flow graph [A structural map tracking how data and variables move and get used from where they are defined to where they appear across execution paths]: tracks variable propagation and value lifecycles across execution paths.

Fig 13. CodeBLEU combines token overlap, keyword weights, syntactic trees, and data-flow graphs.

Data-flow graphs trace variable lifecycles across execution paths that syntax trees miss. This combination aligns static evaluation with human functional judgments.

6.3 Representation Similarity via CodeBERTScore

CodeBERTScore [An evaluation method that uses neural network embedding vectors to softly compare candidate code against a reference without requiring exact matching text] evaluates code in continuous vector spaces.

1. Jointly encode prompt and code with a pretrained model into contextual vectors.
2. Build a pairwise cosine similarity matrix between candidate and reference tokens.
3. Compute directional precision and recall from maximum similarities to produce an F-score.

• Continuous embeddings: measures program similarity in vector space instead of using symbolic parsers.
• Soft alignment: matches operations functionally when identifier names and syntax differ.
• Syntax tolerance: scores incomplete code and minor formatting issues without crashing a parser.
• Prompt conditioning: contextualizes vector representations using the natural language prompt and surrounding code.

Fig 14. CodeBERTScore pipeline computes soft token alignments and F-scores from contextual embeddings.

Contextual token embeddings capture operation intent across varied implementations. Soft vector matching avoids the brittleness of static compilers and token overlap.

---

7. Dynamic Execution and Functional Correctness

In short
– Dynamic execution verifies code behavior against test suites inside isolated runtime environments.
– Benchmarks range from function completion to full competitive programming and repository modifications.
– Metrics balance pass@k functional correctness with non-functional efficiency, maintainability, and resource utilization.

7.1 Evaluating Behavior via Sandboxed Execution

Dynamic execution runs candidate programs against test suites in isolated environments to verify semantic behavior directly.

Benchmark	Scope	Input	Test Harness
HumanEval	Single function	Docstrings	Hidden unit tests
CodeContests	Standalone program	Problem statements	Compilation and I/O limits
Agentic suites	Repository patch	Multi-file edits	Continuous integration tests

• 1 or 0: functional score assigns binary success only if all tests pass.
• Neubig’s parity example: is_positive(x) with x >= 0 fails edge case 0 while passing others.
• 164 Python problems: HumanEval measures zero-shot function completion using docstring prompts.
• Zero-shot solving: CodeContests requires standalone algorithmic solutions without human-like iterative debugging.

Fig 15. Isolated sandboxes execute candidate code against test suites to verify runtime behavior.

Surface syntax matching misses semantic bugs that isolated runtime execution catches. Zero-shot competitive programming tests direct behavioral correctness without human-like debugging loops.

7.2 Strengths and Pitfalls of Execution

Execution checks accommodate diverse algorithmic logic but encounter test suite flaws and container drift.

• Implementation flexibility: accepts valid recursion, iteration, or closed-form math that passes assertions.
• False positives: incomplete test suites let buggy code pass, as exposed by EvalPlus.
• False negatives: underspecified prompts omit formatting rules, failing valid logic on strict assertions.
• Environment drift: container updates and external library shifts break older evaluation suites.
• Docker sandboxing: virtualized containers isolate untrusted model code and enforce strict resource caps.
• Repository latency: full continuous integration runs on large codebases create significant evaluation bottlenecks.

Flawed test assertions distort accuracy by penalizing valid code or accepting broken logic. Sandboxed virtual containers shield local infrastructure from dangerous or runaway executions.

7.3 Statistical Sampling and Non-Functional Metrics

Generative evaluation balances probabilistic multi-sample scoring against operational software engineering constraints.

• pass@k (A metric that calculates the probability that at least one program passes all unit tests when sampling a given number of candidate solutions.): evaluates probability that at least one sampled candidate succeeds.
• pass@1 vs pass@10: compares strict zero-shot reliability against expanded sampling search budgets.
• Non-functional requirements (Program quality attributes beyond basic correctness, such as execution speed, memory efficiency, and maintainability.): evaluate operational efficiency and quality beyond basic unit testing.
• 397 editing tasks: NoFunEval tests speed improvements, loop optimizations, and memory reduction.
• DiffBLEU (A metric that scores the quality of a code edit by comparing generated modifications against a reference implementation patch.): scores edit distance against reference patches to evaluate resource utilization.
• CodeQL checks: inspects structural code maintainability and security vulnerabilities through static analysis.

Fig 16. Sampling more candidate completions increases the probability of finding a working program.

Binary correctness tests fail to show if generated code runs efficiently at scale. Non-functional metrics ensure models produce maintainable, secure, and resource-efficient software implementations.

---

8. Assessing Non-Functional Requirements

In short
– Non-functional benchmarks evaluate code efficiency, latency, and maintainability alongside correctness.
– Execution timeouts in reinforcement learning incentivize models to derive optimal algorithms.

8.1 Evaluating Non-Functional Code Properties

Models must optimize code to reduce runtime and resources while preserving identical functional outputs.

• Optimization goal: refactor code to improve non-functional requirements (program quality attributes beyond basic correctness, like efficiency).
• Correctness bias: historical benchmarks evaluated functional correctness rather than execution speed or code maintainability.
• 397 tasks: NoFunEval (Singhal et al., 2024) systematically grades code edits against non-functional criteria.

Fig 17. NoFunEval measures code refactoring across runtime speedup, resource use, and maintainability.

Property	Verification Protocol	Metric
Runtime efficiency	Direct test suite execution	Empirical speedup
Latency and resources	Reference implementation comparison	DiffBLEU (metric scoring edit quality against reference patches)
Maintainability and security	Static analysis rule checks	CodeQL × DiffBLEU

Programs must consume minimal compute and memory while maintaining strict functional equivalence.

8.2 Runtime Efficiency and Timeout Penalties

Reinforcement learning from test results trains models to discover computationally optimal algorithms instead of brute-force solutions.

• Pre-training bias: standard generation models favor any code passing unit tests regardless of efficiency.
• Execution timeouts: Codeforces competitive programming benchmarks fail and penalize slow, unoptimized solutions.
• (10^{12}) integer sum: iterative loops time out, but reasoning yields the closed-form formula (n(n + 1) // 2).
• DeepCoder (Luo et al., 2025): reinforcement learning scales reasoning tokens to avoid brute-force implementations.

1. Sample reasoning trajectories and candidate code completions.
2. Execute completions in isolated test harnesses with strict timeouts.
3. Update policy weights based on execution test rewards.

Timeout penalties force models to reason algorithmically before emitting inefficient code.

---

9. Reinforcement Learning with Execution Feedback

In short
– Unit test execution provides binary verification rewards to guide code generation models.
– Reinforcement learning enforces both algorithmic correctness and execution speed constraints.
– Progressive token budget scaling enables extended reasoning required for competitive programming.

9.1 Test Results as Policy Rewards

RL trains reasoning models using functional unit test verification instead of predicting reference code tokens.

1. Sample solutions: generate reasoning steps paired with executable code candidates.
2. Execute tests: run programs in isolated environments to assign binary pass or fail rewards.
3. Update policy: reinforce successful reasoning trajectories to increase their generation probability.

Algorithm	Implementation	Execution Outcome	RL Reward
Direct iteration	sum(range(n + 1))	Timeouts on large inputs ($n \le 10^{12}$)	0 reward
Endpoint pairing	Deriving $n(n + 1) // 2$	Solves boundary cases 0 and 1 efficiently	Full reward
Formula recall	Recalling $n(n – 1) // 2$	Fails immediately on test $n = 1$	0 reward

Binary execution rewards penalize slow and incorrect code, guiding models toward efficient algorithmic solutions.

Fig 18. Sandboxed test execution generates binary rewards that directly optimize model reasoning policies.

9.2 Progressive Token Scaling Post-RL

Execution feedback after mid-training (continual pre-training that adapts a general model to code) boosts benchmark accuracy.

• DeepCoder benchmark gains: execution feedback drives substantial performance leaps on LiveCodeBench challenges.
• 16K token limit: train the initial RL policy with response limit $K = 1{,}000$ tokens.
• 16K to 32K tokens: expand training budgets to accommodate longer step-by-step reasoning traces.
• 64K token evaluation: test without RL post-training to approach competitive o3-mini benchmark results.

Fig 19. Progressively expanding token limits during RL equips models to solve harder reasoning benchmarks.

Extended chain-of-thought generation is vital for exploring search spaces and establishing correct mathematical bounds.

---

10. Summary

• Capability progression: Coding models advance from single completions to repository engineering across pre-training, mid-training, and reinforcement learning stages.
• Tokenization and context: Code-aware BPE preserves indentation syntax, while dependency-ordered multi-file training enables extended repository context learning.
• Bidirectional infilling: Sentinel tokens rearrange documents to teach standard causal transformers prefix-and-suffix code editing without architecture modifications.
• Static evaluation: CodeBLEU and CodeBERTScore assess syntactic trees, data-flow graphs, and contextual embeddings beyond superficial token overlap.
• Dynamic verification: Sandboxed test harnesses evaluate functional pass rates alongside non-functional efficiency, latency, and code maintainability metrics.
• Execution-guided RL: Binary test feedback and timeout penalties reinforce extended reasoning trajectories that yield correct and computationally optimal implementations.

Combining grounded pre-training with execution-based reinforcement learning transforms language models into robust, autonomous software engineering agents.

---

Source

Lecture: Agents for Coding and Software Development by Graham Neubig, Carnegie Mellon University.
Course: CMU 11-768 AI Agents (Fall 2026), Lecture 6, first half.
Video: YouTube · Slides: PDF · Course page: cmu-agents.com

Further reading referenced in the lecture:

• Groeneveld et al., 2024 · OLMo, §2
• StarCoder · data selection, filtering, and decontamination
• Chen et al., 2021 · whitespace-run tokens
• StarCoder2 tokenizer
• Rozière et al., 2023 · Code Llama, §2
• Hui et al., 2024 · Qwen2.5-Coder, Table 3 (selected columns)
• Fried et al. · InCoder, §2 and §4 · illustrative example
• Related scaling study: Bavarian et al., 2022
• Muennighoff et al., 2023 · OctoPack
• Yasunaga & Liang, 2020 · DrRepair
• Le et al., 2022 · CodeRL
• Liu et al., 2023 · CodeExecutor
• Yin & Neubig, ACL 2017 · §5.1
• Ren et al., 2020 · CodeBLEU
• Zhou et al., EMNLP 2023 · CodeBERTScore · original similarity heatmap
• Liu et al., 2023 · EvalPlus / test adequacy
• CodeContests / AlphaCode
• Singhal et al., 2024 · NoFunEval · 397 code-editing tasks
• Luo et al., 2025 · DeepCoder · execution-reward RL

---

dataHacker.rs — LLM_log #036: Training and Evaluating Foundation Models for Code — Data Mixtures, Infilling, Execution Feedback, and Metrics. By Vladimir Matic.