Exposing and Evaluating Tools — REST APIs, MCP, Parallel Tool Calls, and Tool-Use Benchmarks

Highlights: This session of CMU’s 11-768 AI Agents course (Fall 2026, Graham Neubig) moves tool calling out of the notebook and into the network, tracing the path from in-process Python libraries to REST endpoints, MCP servers, and the serving stacks underneath them. We walk through wrapping tools with FastAPI and Pydantic, the two-tier credential split that MCP introduces, concurrent execution with asyncio.gather, and why the same model weights produce wildly different tool-call error rates across hosting providers.

• From libraries to endpoints: A local get_weather function becomes an authenticated HTTP route in a few lines of FastAPI, and Pydantic annotations yield an OpenAPI JSON Schema you can hand straight to a model as a tool definition.
• Shell-native invocation: Coding agents with bash access often skip custom clients entirely and hit REST APIs with curl, passing keys through headers like X-API-Key.
• Two-tier credentials: MCP splits authentication into an MCP key given to the agent runtime and an upstream key held only by the server, so the agent never sees your GitHub token and cannot push it to a public repo.
• Where the protocols diverge: OpenAPI and MCP share names, descriptions, and JSON Schema, but differ on discovery (tools/list versus fetching a spec), invocation (tools/call over MCP transport versus an HTTP verb), authentication target, and scope.
• Parallel calls and dependencies: Bundling three independent calls collapses a ~6.4-second serial timeline to roughly 2.8 seconds via asyncio.gather, but chained steps like customer ID → orders → refund must stay sequential.
• Provider variance is real: OpenRouter’s tracking shows the same model served at tool-call error rates from 15% down to 0.01%, driven by FP4 versus FP8 quantization, lossy speculative decoding, missing constrained decoding, and plain unreliability.

Tutorial Overview:

1. Direct REST API Invocation and Shell Execution
2. Model Context Protocol (MCP) Architecture and Tool Discovery
3. Schema Contracts, Response States, and Error Handling
4. Serial Latency Bottlenecks and Concurrent Tool Execution
5. Dependency Analysis and Execution Efficiency
6. Whole-Stack Evaluation, Serving Stacks, and Provider Variance
7. Summary

---

1. Direct REST API Invocation and Shell Execution

1.1 From In-Process Libraries to HTTP REST APIs

There is an alternative to schema-based tool calling: programmatic tool calling. Here we hand the language model an entire Python script and let it call the functions defined inside whenever it needs to act.

Consider an in-process library named weather_lib.py. We define the data models with Pydantic so the types are strict:

from typing import Literal
from pydantic import BaseModel

class Weather(BaseModel):
    summary: str
    temperature: float

def get_weather(city: str, units: Literal["C", "F"] = "F") -> Weather:
    return provider.lookup(city, units)

An agent running in an environment with Python execution can import get_weather and run it locally to fetch observations on demand.

Fig 1. Libraries expose in-process APIs

This works well in a self-contained local setup, but it breaks down in production. A local in-process library cannot be queried across a network by remote clients, and it does not map cleanly onto the non-programmatic tool-calling schemas that frontier models support.

The fix is to wrap those functions in an HTTP REST API. With FastAPI, a production-ready web backend takes only a few lines:

from typing import Annotated, Literal
from fastapi import FastAPI, Security
from fastapi.security import APIKeyHeader
from weather_lib import Weather, get_weather

app = FastAPI(title="Weather API")
key_header = APIKeyHeader(name="X-API-Key")

@app.get("/weather", operation_id="get_weather")
def weather(
    city: str,
    key: Annotated[str, Security(key_header)],
    units: Literal["C", "F"] = "F"
) -> Weather:
    verify(key)
    return get_weather(city, units)

Your local Python library is now an authenticated web endpoint.

Fig 2. REST APIs expose functions over HTTP

Why bother with the wrapper? The first payoff is access control. Guarding a weather lookup may not matter much, but private databases, proprietary internal APIs and corporate resources certainly do. Requiring an API key in the header lets the backend authenticate every request and reject unauthorized callers before any tool logic runs.

The second payoff is that FastAPI generates a formal OpenAPI specification for free, with a complete JSON Schema for every route and payload. Because it inspects your Pydantic annotations, the input and output contracts are already machine-readable.

You can feed that generated JSON Schema straight to a foundation model as a tool definition. When the model picks the tool, it formats the call over standard HTTP, sends it to the remote server and waits for the response — a clean, decoupled boundary between the agent’s reasoning loop and the remote execution environment.

Fig 3. REST API servers publish OpenAPI descriptions

Fig 4. Why Wrap Tools in a REST API

1.2 REST Calls from Agent Shell Environments

How do coding agents reach REST endpoints when they live inside a shell? Models can emit structured JSON tool calls, but REST remains the paradigm software engineers use every day for web services. An autonomous agent with bash access can talk to remote services through the same command-line utilities we do.

Fig 5. Agent Calls a REST API from the Shell

1.3 Executing REST Calls with cURL

Rather than writing a custom Python client, a shell-enabled coding agent can build and run HTTP requests on the fly with curl:

curl -sS -G https://weather.example/weather \
  --data-urlencode "city=Pittsburgh" \
  --data-urlencode "units=F" \
  -H "X-API-Key: $WEATHER_API_KEY"

The pattern is transparent and effective if you are comfortable with REST conventions and shell scripting. If you have not built services with FastAPI yet, it is worth trying — exposing endpoints and inspecting the auto-generated documentation is painless.

Fig 6. Coding agents often call REST APIs with curl

Calling APIs with curl over bash is the traditional paradigm of software integration, predating specialized agent protocols. MCP, introduced by Anthropic roughly two years before the CMU lecture, promised a standardized protocol for supplying language models with external context and tool integrations — and Graham Neubig’s first reaction to it was skepticism.

Fig 7. The skeptic’s view of MCP: what the architecture adds beyond plain REST calls.

---

2. Model Context Protocol (MCP) Architecture and Tool Discovery

2.1 Connecting Hosts to MCP Servers

Building tool backends from scratch gets tedious fast, and FastMCP exists to remove that friction. Think of it as the FastAPI of the Model Context Protocol (MCP) ecosystem: to expose an application or service as an MCP server, you swap FastAPI out and FastMCP in. The result plugs into the wider architecture, where a host application reaches local or distributed MCP servers through dedicated client interfaces.

Fig 8. MCP connects a host to servers

You may not need new endpoints at all. FastMCP can ingest an existing specification from a standard REST API — an OpenAPI description, for instance — and convert that surface into a working MCP server, mapping the HTTP operations and schemas into callable tool signatures your agent can invoke.

Fig 9. FastMCP projects OpenAPI operations as MCP

Why add an MCP server instead of letting the agent talk to web APIs directly? Mostly security. Authentication splits into two tiers: an MCP API key and an upstream API key. The MCP API key goes to the runtime process running your agent so it can authenticate with the intermediate server, while the upstream key — a personal GitHub token, say — is given only to the MCP server and withheld from the agent.

Fig 10. Two-Tier Credentials in MCP

In code, that split is just two environment variables: the upstream client carries UPSTREAM_API_KEY in its header, while a StaticTokenVerifier guards the MCP endpoint with MCP_API_KEY.

Fig 11. Start MCP with separate credentials

The separation matters once you think about how models fail. Hand a raw GitHub token to an LLM agent and nothing stops it from deciding that committing and pushing that secret to a public repository is a good idea. At that point your account is compromised and attackers can inject malware across all your codebases. With an MCP server in between, the agent only ever holds an MCP-scoped key that is comparatively harmless if leaked, and your production credentials stay behind the server boundary.

Fig 12. The MCP client uses its own API key

This is credential brokering: the server accepts one credential from the agent harness and uses a completely different credential upstream, so the model sees neither secret.

Fig 13. Direct API credential exposure versus two-tier isolation via an MCP server

2.2 Finding MCP Servers: The Official Registry

To find a server for a specific task, start with the official registry — an indexed catalog of standardized tools and a reliable way to extend your agent’s capabilities.

The registry indexes a very large number of MCP servers, and it is not the whole ecosystem. Community-run directories rank servers by popularity and similar metrics, curating what other developers have built so you can spot the most useful integrations.

Fig 14. The official Model Context Protocol registry interface at registry.modelcontextprotocol.io

---

3. Schema Contracts, Response States, and Error Handling

3.1 Shared Schemas, Different Execution Contracts

Both approaches give you the same fundamental capability: wiring an entire set of APIs into your agent. You can do it with a plain JSON Schema description, or you can hook into the Model Context Protocol (MCP). The objective is identical; the day-to-day mechanics are not.

Tool discovery is the clearest example. In a REST workflow, the agent fetches an OpenAPI document and inspects it to learn which endpoints exist. MCP standardizes that step with a dedicated API that lets the client list every available tool directly.

Invocation diverges just as sharply. REST means your agent fires an HTTP call over the web, while MCP routes the call through its own transport, which can run locally over STDIO or across a socket. Beyond tool calling and authentication, MCP also bundles first-class support for resources, prompts, and extensions that REST setups do not provide natively.

Fig 15. Where REST and MCP Diverge

3.2 Protocol Boundaries and Contract Enforcement

Modeling domain states accurately requires formal interfaces. HTTP APIs described by OpenAPI and MCP build their contracts on exactly the same primitives: names, descriptions, and JSON Schema for structured inputs.

Fig 16. REST APIs and MCP: shared schemas, different execution contracts

From that shared foundation the two split across discovery, invocation, authentication, and scope. OpenAPI discovery means retrieving an external specification document; an MCP client discovers tools at runtime with a tools/list request. For invocation, an HTTP API expects a verb sent to a path with parameters, while an MCP agent dispatches tools/call over its transport.

The security models follow the same split. An OpenAPI client authenticates directly against the target API endpoint, whereas an MCP client authenticates to the MCP server and leaves any upstream API authentication as a separate concern handled behind the scenes. OpenAPI is scoped strictly to HTTP operations; MCP covers resources, prompts, and extensions as well.

3.3 Response-State Edge Cases

Real data punishes simplistic categorical assumptions about response states. In the CMU lecture, Graham Neubig used Pittsburgh weather as the running example throughout
tag and broke his own weather API live: the city was rainy and sunny at once, a torrential downpour under bright sunshine.

The interface could not return two states at the same time. It had never been designed to deliver multiple concurrent states, so the moment that edge case appeared, the call failed unexpectedly.

---

4. Serial Latency Bottlenecks and Concurrent Tool Execution

4.1 The Cost of Serial Tool Execution

When tool calls run serially, generation steps and tool calls strictly alternate, and every step stacks onto the timeline. A typical trace looks like generation phases of 0.8 seconds interleaved with a Weather call at 1.2 seconds, a Calendar call at 0.9 seconds, and a Flight search at 1.1 seconds.

Forcing this back-and-forth pattern turns the dependency between generation and execution into a compounding latency bottleneck.

Fig 17. Serial tool-call cost

4.2 Bundling Independent Calls into One Round Trip

The fix that has emerged recently is parallel tool calling, and it addresses the same problem we saw with CodeAct and programmatic tool calling: calling tools one at a time stretches even simple operations. Querying the weather, then waiting to query a calendar, then searching for a flight adds delay between requests that are completely independent of each other.

Fig 18. Serial vs. Parallel Tool Calls

Bundling the three independent calls into one round trip collapses the timeline to two generation phases plus the slowest tool: 2 × 0.8 + max(1.2, 0.9, 1.1) ≈ 2.8 seconds, down from roughly 6.4 seconds.

Fig 19. Parallel tool calls

4.3 Running Calls Concurrently

Independent tool calls can be issued at the same time instead of one after another, and the mechanics are simple: add multiple tool call blocks to a single agent response.

On the execution side, ordinary Python is enough to collect the calls and run them together. Write a small coroutine that parses a call’s arguments, awaits the corresponding tool, and returns the call ID along with the value, then hand all of those coroutines to asyncio.gather so the results come back once the calls have finished running concurrently:

async def execute(call):
    args = json.loads(call.arguments)
    value = await TOOLS[call.name](**args)
    return call.call_id, value

results = await asyncio.gather(
    *(execute(call) for call in function_calls),
    return_exceptions=True,
)

Pairing each result with its call ID is what lets you match the gathered results back to the individual parallel calls that produced them.

Fig 20. Concurrent execution

---

5. Dependency Analysis and Execution Efficiency

5.1 Parallel Calls vs. Enforced Sequencing

Before firing off several tool calls at once, check one thing: dependencies. You can only parallelize calls whose inputs do not depend on another call’s result. Looking up the weather for Pittsburgh, reading today’s calendar, and checking a flight status can all run concurrently, because none of them needs the others’ output.

Contrast that with finding a customer ID, using that ID to fetch their orders, and issuing a refund on one of them. Each step supplies the input for the next, so the calls have to resolve in order. The same applies if the calendar lookup or flight status depended on the weather result. Dependency constraints are the real limit on when parallel tool calling applies.

Fig 21. Dependency Check Before Parallelizing

Fig 22. Parallelize only without dependencies

When calls are genuinely independent, should you run them in parallel? Usually yes. Standard Python is enough to manage a whole set of tool calls, and asyncio.gather is the common pattern: fire them off concurrently and collect the results once they all finish.

5.2 Why Expensive Models Can Be Cheaper

Parallelization also explains a gap between efficient and inefficient models. With today’s agentic language models there is a paradox: on a task-by-task basis, the more expensive model often costs less.

Two reasons drive this. Smarter models reach an appropriate solution faster instead of burning cycles on failed attempts. And more recent models are much better at parallel tool calling, issuing a batch of calls at the same time.

Fig 23. Why Expensive Models Can Be Cheaper

Run a coding agent or a similar workflow with the better models and you notice this right away — and the reason is training, not just capability: reinforcement learning forced the behaviour.

Fig 24. The path from a long-available mechanism to models that routinely batch tool calls.

5.3 Benchmarking Tool Use: the BFCL

How do we evaluate all of this? Rather than jumping to end-to-end agentic tasks, it pays to measure the foundational tool use ability of a model first, since that is a prerequisite for being effective at agentic tasks at all. Several datasets benchmark this, and the most famous is the Berkeley Function Calling Leaderboard (BFCL).

Fig 25. Berkeley Function Calling Leaderboard

BFCL has gone through four versions. Version one assessed single-turn tool calling; by V4 the benchmark also covers multi-turn tool calling.

It includes agentic tool calling as well, tailored to tool-calling agents rather than end-to-end coding agents. It also measures robustness through failure modes such as hallucinated arguments, plus format sensitivity across different schema formats and parameter configurations.

Fig 26. What BFCL Measures

---

6. Whole-Stack Evaluation, Serving Stacks, and Provider Variance

Benchmarking tool-calling systems cannot stop at isolated prompt tests. To know whether a system will survive production, you have to evaluate the entire stack end to end — the metrics, the operational variables, and the infrastructure quirks that decide whether tool calls work or fail.

6.1 Tool Selection, Trajectory, and Task Success

The first dimension is tool selection: did the model choose an appropriate tool for the task? A common failure mode is picking a tool that cannot accomplish the goal at all. Next to it sits argument generation — whether the model supplied the right values and respected the expected parameter schema.

Fig 27. Evaluate the whole stack

Once you move into agentic tool calling, single-step checks stop being enough. You also need to evaluate the trajectory: were the tools invoked in the correct order across multiple turns? And beyond sequencing, end-to-end task accuracy tells you whether the agent actually reached the objective. Tool selection accuracy, argument validity, execution trajectory, and task completion are the foundational criteria behind benchmarks like the Berkeley Function Calling Leaderboard (BFCL).

6.2 Latency, Token Overhead, and Retries

Correct final output is only half the story; there is an operational dimension centred on efficiency. Efficiency tracks how fast a model solves a task and how many tokens it burns doing so. Since tokens translate directly into cost, a trajectory that wastes thousands of them is impractical in production even when it eventually succeeds.

Fig 28. Quality has operational dimensions

Reliability is the close companion of efficiency. If a third-party call times out or hits network latency, does the model retry appropriately? If a tool is out of order or returns an explicit error, can the model recognise the failure and switch to an alternative? A brittle agent crashes or hallucinates; a reliable one adapts its plan to recover from runtime faults.

Fig 29. Operational Reliability Checks

In production these failures come from three distinct places: the model (wrong tool, bad arguments, ignored results), the harness (parse, auth, validation, or call-ID mismatch), and the provider or tool itself (timeout, rate limit, execution error).

Fig 30. Operational tool failures

Safety is the third operational pillar the leaderboard has grown to cover. When a user prompts the model to invoke a tool in an inappropriate or malicious way, can the system refuse or handle the request properly? That question leads to one of the most surprising realities of deployment: the same model can behave radically differently across inference providers.

6.3 Provider Discrepancies: Serving Stacks and Quantization

A striking data point comes from OpenRouter‘s tool-call error rate tracking. It evaluates one model — GLM 5.3 in this snapshot — across different commercial hosting providers, so the only variable is who serves the weights. You would expect identical performance from identical weights.

Fig 31. Provider tool-call error rates

The reality is staggering: error rates range from an enormous 15% down to 0.01% or 0.05%. Several engineering choices hidden inside the serving stack explain that spread.

First, inference and decoding algorithms differ from provider to provider, and quantization is a major factor. Models served at FP8 — roughly today’s default — make far fewer mistaken tool calls than models compressed to FP4. FP4 saves the provider substantial memory and hardware cost, but it usually yields a measurably worse model, and very few FP4 quantizations preserve baseline tool-calling performance.

Second, speculative decoding, including variants tailored for agents. The engine drafts predictions with a cheaper, smaller model and verifies them against the larger one to speed up generation. Truly lossless speculative decoding should not change outputs at all — but a lossy implementation tuned for throughput drifts the output distribution, breaks tool calls, and marketing it as the original model is misleading.

Fig 32. Inference and serving stack factors driving provider tool-call variance

Third, providers differ in whether they enforce constrained decoding. Grammar-based constraints make structured tool outputs far more dependable. Many providers write custom serving engines from scratch; others build on vLLM or SGLang. Because these stacks support different grammar-based decoding algorithms — or none — some providers leave the model to produce syntactically valid JSON unaided, and their syntax failure rate climbs immediately.

Finally, the boring explanation: plain provider unreliability. Intermittent service drops, dropped connections, or internal gateway timeouts during tool execution all register as failed tool calls.

Fig 33. Four Causes of Provider Tool-Call Variance

6.4 Tool Use as a Layered System

Stepping back, tool use is fundamentally a multi-layered system. Tools supply the capabilities that make an agent an agent: retrieval, execution, generation, and application interaction.

Those capabilities do not come free by pointing a model at an API. The architecture has to inject structured schemas into the prompt so the model can emit well-formed calls, and the orchestration layer then validates the payload, dispatches it to the target service, and matches results back to the original call by ID.

Fig 34. Tool use is a layered system

Grammar-based decoding at the inference layer makes generation more reliable, but remember what it buys you: structural and syntactic validity, not semantic correctness. A model constrained to emit valid Python can produce a script that parses perfectly and still implements completely wrong logic.

Connecting models to tools then spans a spectrum, from direct REST calls to standardised protocols like the Model Context Protocol (MCP). Around all of it, orchestration frameworks manage dependencies, scheduling, and concurrency — and give you the visibility needed to benchmark the full stack.

6.5 What to Check Before You Ship

Here is the short checklist to run over any tool-calling deployment:

• Tool selection accuracy — does the model pick a tool that can actually accomplish the task?
• Argument validity — are the values right and do they match the expected parameter schema?
• Trajectory order — in multi-turn and agentic settings, are the tools invoked in the correct sequence?
• End-to-end task success — did the agent reach the real objective, not just produce plausible intermediate steps?
• Latency, tokens, and cost — how long and how expensive is a typical trajectory?
• Retry and recovery behaviour — on timeouts, errors, or an out-of-order tool, does the agent retry or switch tools instead of crashing or hallucinating?
• Provider quantization and constrained-decoding support — which precision level (FP8 vs FP4) is served, is speculative decoding lossless, and does the serving stack implement grammar-based decoding at all?

---

7. Summary

Connecting language models to external services means replacing fragile in-process scripts with real network boundaries. Wrapping Python functions in a REST API with FastAPI and Pydantic gives you auto-generated OpenAPI contracts, and moving up to the Model Context Protocol (MCP) adds a standardized runtime layer: dynamic tool discovery, multiple transports, and two-tier credential isolation. Keep the upstream API key behind the MCP server and hand the agent only an intermediate key, and a hallucination or a stray line of executed code can no longer leak your real token.

On the execution side, dependent workflows need strictly sequenced calls, while independent operations can be dispatched together through parallel tool calling so latency stops accumulating. Models trained with reinforcement learning and brevity penalties have become good at batching several calls at once, which is why a seemingly more expensive model often finishes a complex agentic task faster and cheaper — exactly the behavior tracked by the Berkeley Function Calling Leaderboard.

Reliability, though, is a property of the whole serving stack, not of your prompt. OpenRouter’s tracking shows that identical model weights can produce error rates anywhere from 15% down to a fraction of a percent depending on the hosting provider. Quantization choices such as FP4 versus FP8, lossy speculative decoding implementations, and missing grammar-based constrained decoding each add their own failure mode. Dependable agents come from disciplined choices across schemas, transport layers, and inference runtimes.

---

Source

Lecture: Tool Use for Language Model Agents by Graham Neubig, Carnegie Mellon University.
Course: CMU 11-768 AI Agents (Fall 2026), Lecture 2, second half. The first half is covered in LLM_log #028.
Video: YouTube. Slides: PDF. Course page: cmu-agents.com.

Further reading referenced in the lecture:

• FastAPI — security, OpenAPI
• FastMCP — OpenAPI integration, token verification, bearer client auth
• Model Context Protocol — modelcontextprotocol.io
• Berkeley Function Calling Leaderboard (BFCL) V4 — gorilla.cs.berkeley.edu
• OpenRouter provider analytics — openrouter.ai

---

dataHacker.rs — LLM_log #029: Exposing and Evaluating Tools — REST APIs, MCP, Parallel Tool Calls, and Tool-Use Benchmarks. By Vladimir Matic.