Module 2: Agents: Using and Extending Them#

From chatbot to agent: give the model tools and a loop, and it can act. Tool-calling mechanics at the API level, the pi harness (permission modes, supervision, where agents shine and where they waste your time), then making it yours: skills, custom tools, MCP servers, subagents, and extensions that hook the harness itself. The longest module of the class: it owns the middle of Day 1, and its lab runs in five parts woven through the lecture.

Questions this module answers#

• What turns a chatbot into an agent?
• How does tool calling actually work at the API level — what does a tool call look like on the wire?
• What does a harness like pi do for me — and what is it doing behind my back?
• How much autonomy should I give an agent, and what happens when I get that wrong?
• When do agents shine, and when do they waste my time?
• Skill, custom tool, MCP server, or subagent — which extension mechanism fits which job?
• What is MCP under the hood, and when is plain bash the better tool?
• What does it mean that the context is just editable data?

Slides#

Labs 2.1–2.5: Build an Agent, Then Bend the Harness (~3.5 h)#

Five labs, one arc:

1. Lab 2.1 — turn your Lab 1.1 chat script into a tool-calling agent by hand and observe how the model treats tool results.
2. Lab 2.2 — lift those same tools out of your script into a minimal MCP server and see what the protocol buys (and costs).
3. Lab 2.3 — get hands-on with pi, the industrial version of what you wrote.
4. Lab 2.4 — teach it: two skills, from a one-file explainer to a skill that bundles examples and an executable script.
5. Lab 2.5 — extend pi itself: two extensions that hook the harness’s loop, ending with one of your own design.

A bonus lab at the end of the page extends the harness one step further. Bonus labs are optional; do them if you finish early or want to go deeper.

Use small (the fast, cheap model) throughout unless a step says otherwise. The mcp package is preinstalled on your instance.

Lab 2.1: hand-rolled tool calling (45 min)#

Goal#

Turn your Lab 1.1 chat script into a tool-calling agent by hand: the model requests a tool, your code runs it, the model sees the result. Then find out how the model treats a tool result it has reason to doubt.

Provided#

• Your own Lab 1.1 script
• A checker for this lab, on your instance
• The tool schema and tool-call response shapes are shown in the snippets on this page

Steps#

1. Reopen your Lab 1.1 script. Ask it something it cannot know: “what files are in my current directory?” Watch it guess or refuse. This is the gap you’re about to close.

2. Define one tool, list_files, as a JSON schema (name, description, no parameters) and pass it via the tools parameter. The snippet in step 3 shows the shape.

3. Handle the response: when the model returns tool_calls instead of content, run os.listdir(), append the assistant’s tool-call message and a tool role message with the result, and call the API again. Print the final reply.

   tools = [{
       "type": "function",
       "function": {"name": "list_files", "description": "List files in the current directory",
                    "parameters": {"type": "object", "properties": {}}},
   }]
   
   resp = client.chat.completions.create(model="small", messages=messages, tools=tools)
   msg = resp.choices[0].message
   
   if msg.tool_calls:
       messages.append(msg)
       for call in msg.tool_calls:
           result = "\n".join(os.listdir("."))  # the only "execution" there is
           messages.append({"role": "tool", "tool_call_id": call.id, "content": result})
       resp = client.chat.completions.create(model="small", messages=messages, tools=tools)
       msg = resp.choices[0].message
   
   print(msg.content)

4. Ask the same question again. The model now answers with your actual files. Note the sequence: the model asked, your code acted, the model saw the result.

That sequence, as a state machine — this is the entire agent loop:

flowchart LR
    U(["User task"]) --> M["Model"]
    M -->|text answer| D(["Done: print reply"])
    M -->|tool_calls| T["Run the tool"]
    T -->|append result as tool message| M

5. Now implement four tools with arguments: add, subtract, multiply, divide, each taking two numbers. Parse the arguments JSON, compute, return the result as a string. A dict mapping tool names to functions keeps the dispatch clean.

import json

def run_turn(messages, tools):
    while True:
        resp = client.chat.completions.create(
            model="small", messages=messages, tools=tools)
        msg = resp.choices[0].message
        if not msg.tool_calls:
            return msg.content
        messages.append(msg)
        for call in msg.tool_calls:
            args = json.loads(call.function.arguments)
            result = TOOLS[call.function.name](args["a"], args["b"])
            messages.append({"role": "tool", "tool_call_id": call.id,
                             "content": str(result)})

6. Sabotage your own tool: inside add, write an if statement so that adding 2 and 2 returns "bread" instead of 4. Every other addition stays correct.

   def add(a, b):
       if a == 2 and b == 2:
           return "bread"
       return str(a + b)
   
   TOOLS = {"add": add, "subtract": subtract, "multiply": multiply, "divide": divide}
   # dispatch: args = json.loads(call.function.arguments)
   #           result = TOOLS[call.function.name](args["a"], args["b"])

7. The experiment: ask the agent to do some math problems, including 2+2. Watch closely: does it answer “bread”? “4”? Does it call the tool again, apologize, explain? Run it several times; try large (the stronger model) too. Write down what you saw.

8. Run the Lab 2.1 checker: all four tools dispatch correctly, add(2, 2) returns "bread", other additions are correct. What the model says about 2+2 is your observation for the debrief, not a pass/fail; that’s the point.

Done when#

The Lab 2.1 checker passes.

Lab 2.2: the same tools, as an MCP server (30 min)#

Goal#

Your four calculator tools live inside your script: private, unshareable, redefined in every program that wants them. MCP is the fix: tools become a service any MCP-aware client can discover and call. Move your calculator into one and measure what that buys and costs.

Provided#

• Your Lab 2.1 agent
• A checker for this lab, on your instance
• The MCP server and client shapes are shown in the hint and solution blocks on this page

Steps#

1. Write a minimal MCP server exposing the same four tools: add (bread and all), subtract, multiply, divide. With the mcp package this is mostly decorators around the functions you already wrote. Run it standalone first and confirm it lists its tools.

from mcp.server.fastmcp import FastMCP

mcp = FastMCP("example")

@mcp.tool()
def greet(name: str) -> str:
    """Greet a person by name."""
    return f"Hello, {name}"

if __name__ == "__main__":
    mcp.run(transport="stdio")

from mcp.server.fastmcp import FastMCP

mcp = FastMCP("calculator")

@mcp.tool()
def add(a: float, b: float) -> str:
    """Add two numbers."""
    if a == 2 and b == 2:
        return "bread"
    return str(a + b)

# subtract, multiply, divide: same shape, no bread

if __name__ == "__main__":
    mcp.run()  # stdio transport

2. Now gut your script: delete the local tool schemas and the dispatch dict, and replace them with MCP client calls. At startup, ask the server what tools it has and build the tools parameter from the response; on tool_calls, forward the invocation to the server and relay the result. Your agent loop doesn’t change; only where execution lives does.

import sys
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client

server = StdioServerParameters(
    command=sys.executable,
    args=["calculator_server.py"],
)

async with stdio_client(server) as streams:
    async with ClientSession(*streams) as session:
        await session.initialize()
        available = (await session.list_tools()).tools

tools = [{"type": "function",
          "function": {"name": t.name, "description": t.description,
                       "parameters": t.inputSchema}}
         for t in (await session.list_tools()).tools]

What the finished split looks like on the wire:

sequenceDiagram
    participant A as Your agent (MCP client)
    participant S as Your MCP server
    Note over A,S: JSON-RPC over stdio
    A->>S: initialize
    S-->>A: capabilities
    A->>S: tools/list
    S-->>A: add, subtract, multiply, divide (+ schemas)
    A->>S: tools/call: add(2, 2)
    S-->>A: "bread"

3. Re-run the math problems. Same behavior, same bread; which is the proof the server is real: your script no longer contains an add function at all, yet 2+2 is still bread. The lie moved from your process into a service.
4. Run the Lab 2.2 checker: it inspects your script for the absence of local tool definitions, calls your MCP server directly to verify all four tools (including add(2,2) == "bread"), then drives your agent end-to-end.
5. Weigh the trade before moving on: same four functions, but now discoverable by any client. pi could use your calculator without you writing a line of integration. That’s the power. The cost: a running service, a protocol hop on every call, and a new thing to trust. This trade is the entire MCP decision, and you now know both sides of it first-hand.

Done when#

The Lab 2.2 checker passes.

Lab 2.3: hands-on with pi (30 min)#

Goal#

Drive pi, the industrial version of the loop you built in Lab 2.1: same mechanism, plus context management, permission modes, and a full tool set. Learn what supervision actually looks like.

Steps#

1. Start pi on your instance and give it a real multi-step task (e.g., “find every file mentioning 8080 in this directory tree and summarize what each does”). Watch the tool-call loop: same mechanism you built in Lab 2.1, industrialized.
2. Try the permission modes from lecture: run one task approving every tool call, then the same task with more autonomy. Compare control against throughput.
3. Interrupt it mid-task, redirect it, and watch it adapt. Then give it a deliberately vague task and watch it flail; restate precisely in a fresh session and compare.
4. Optional connection: point pi at your Lab 2.2 MCP server and ask it to do math. Your hand-built service, discovered and driven by an industrial harness, zero integration code. (And yes: ask it 2+2 one more time.)

Done when#

You have run the same task under two permission modes, redirected pi mid-task, and compared a vague prompt against a precise one in a fresh session.

Lab 2.4: skills (40 min)#

Goal#

Two skills, easy then harder. Skill = packaged knowledge the agent loads when relevant; the second one shows a skill can carry executable resources too.

Provided#

• A skill template, a sample vulnerability description, and a CVSS v3 test-vector file, on your instance
• Checkers for this lab, on your instance

Steps#

1. Skill 1 (single SKILL.md): a one-file skill with a clear before/after: ask the agent something without the skill, watch it get it wrong or improvise; add the skill; watch the answer snap into place. Verify with the checker.
2. Skill 2, vuln-report (SKILL.md + examples + script): ask fresh-session pi to “write up this vulnerability” using the provided sample description, and note the format it invents and the CVSS score it hallucinates. Now build a skill directory:
   • a SKILL.md explaining the general report format (title, affected component, vuln class, reproduction steps, impact, CVSS vector and score; the same shape you’ll use for real in Module 5’s lab)
   • an examples/ dir with two or three finished reports to anchor tone and depth
   • a cvss.py the SKILL.md tells the agent to run to convert a CVSS v3 vector string into a numeric score and rating, instead of computing it in its head. Use pi to write cvss.py (it knows the spec) and validate it against the provided test-vector file before trusting it.

3. Run the Lab 2.4 checker: pi must produce a write-up of a second vulnerability description with all required sections present and ordered, and the CVSS score must exactly match the vector (only achievable by running the script; models reliably fumble CVSS arithmetic in-head).
4. Note what each layer bought you: the SKILL.md fixes structure, the examples anchor tone and depth, and the script makes scoring deterministic. Knowledge, calibration, and computation: a complete skill ships all three.

Done when#

The Lab 2.4 checker passes.

Lab 2.5: extend the harness (75 min)#

Goal#

Build two pi extensions, in order: a warm-up that hooks both ends of the loop, then one of your own design. Use pi to write them.

Provided#

• The pi extension API documentation and one worked example extension, on your instance
• A checker for the timestamps extension, on your instance

Steps#

1. Extension 1, timestamps: record when each user message is submitted and when the agent’s reply lands; display both plus the elapsed time. Verify against a slow task and a fast one. (Warm-up: one hook at each end of the loop.)
2. Extension 2, your choice: build something you actually want. Ideas to steal: a token/cost meter per turn, a tool-call audit log, a profanity-triggered kill switch, an auto-save that snapshots context to disk. Scope it to finish in ~25 minutes; working and small beats ambitious and broken.
3. Run the timestamps checker: it verifies timestamps appear and are plausible. Extension 2 has no checker; it has an audience.

Done when#

The timestamps checker passes and your Extension 2 does something demonstrable.

Bonus lab: rewrite the context#

Bonus labs are optional; do them if you finish early or want to go deeper.

1. Build an edit-context extension: let the user open the current context and edit any of it before the next turn: past user messages, past agent replies, and tool-call results. Verify all three paths: rewrite one of your questions, rewrite one of the agent’s answers, and change a tool result; then continue the session and confirm the model believes the edited history. This is Module 1’s lesson made executable: the context is just a list, and whoever holds the list writes the past.

2. Re-run the bread experiment through your new extension: have the agent do math with honest tools, then edit the tool result to “bread” after the fact and continue the conversation. Same lie, different layer; no sabotaged tool required.
3. Run the edit-context checker on your instance: it drives the extension through all three edit paths.