Module 6: Capstone: Implement Foundry-lite#

The capstone: build the machine that does the auditing. Using spec-kit and pi, you implement Foundry-lite — a sanctioned subset of Foundry, Cisco’s open specification for an agentic security-evaluation system — and run your implementation against the Lab 5 vulnbox to find real vulnerabilities. The cut is not a compromise the class made behind the spec’s back: the spec’s §4.2 explicitly asks which roles you will merge, split, or omit, and answering that question through the clarify step is the spec-kit lesson. This module doubles as Day 2’s schedule sponge, and it ends the class: a room-wide walk comparing what each implementation caught, the class debrief, and a short closing lecture: fine-tuning vs. context engineering, taking agentic workflows to production, and what to build first back at your desk.

Questions this module answers#

• What is the Foundry security spec, and what problem does it solve?
• What does spec-driven development change about building with agents?
• Why does a trustworthy evaluation system need roles and quality gates instead of one “find bugs” prompt?
• How do I cut a seed spec down to my own situation — and where does the spec itself tell me I may?
• Why does a mechanical citation check beat asking the model to be careful?
• When should I fine-tune versus prompt and context-engineer?
• What does it take to run agentic workflows in production — cost, logging, evaluation?
• What should I build first when I’m back at my desk?

Slides#

Lab 6.1 (Capstone): Implement Foundry-lite (~3 h)#

Goal#

Implement Foundry-lite — the full Foundry value chain, run sequentially, with every quality gate intact — using spec-kit and pi, then point it at the Lab 5 vulnbox and make it produce a finding it can prove. This is the post-lunch block and the rest of the day belongs to it; it’s also the Day 2 sponge, so the time box flexes with how the day has gone.

Success criterion: the spec’s own SC-001, verbatim: “On a target with at least one known seeded vulnerability, an end-to-end run produces a published true-positive finding for it with evidence satisfying §7.3, with no operator intervention between up and publication.” Your seeded target is the vulnbox — you hold this morning’s flags as ground truth.

The spec, in one paragraph#

Foundry distills Cisco ASIG’s internal agentic security-evaluation systems into an organization-neutral specification: eight core agent roles (indexer, cartographer, detector, triager, validator, reporter, and friends — each catching the previous role’s failure mode), a finding lifecycle with an evidence gate, a coordination substrate, and ~130 functional requirements, plus a constitution of eleven inviolable principles, each encoding a production failure. There is deliberately no code in the repo: the spec is the deliverable, and it’s written to be consumed by GitHub’s spec-kit — you clarify its open questions against your environment, then plan, task, and implement with your agent. Your environment is this classroom, and it demands a much smaller system than Cisco’s.

The cut: Foundry-lite#

Five roles, run in order, one process:

flowchart LR
    M["Cartographer-lite<br/>security map"] --> D["Detector<br/>rule sweep + exploratory"]
    D -->|"candidates<br/>(never surfaced)"| T["Triager<br/>three-leg evidence gate"]
    T -->|"true-positive"| V["Validator-lite<br/>live PoC vs. vulnbox"]
    T -.->|"citation fails to resolve:<br/>auto-demote"| N["needs-review<br/>(stays internal)"]
    V -->|"exploited?"| R["Reporter-lite<br/>vuln-report skill → markdown"]

• Cartographer-lite — one security-map doc: endpoints, auth model, trust boundaries (FR-031/FR-032). Kept because the spec’s §7.3 rationale says the trust-boundary map is what makes the evidence gate tractable.
• Detector — rule sweep (FR-037) over the class rule corpus (FR-041, provided) plus exploratory hunting (FR-040). Candidates carry location, class, and reasoning (FR-043) and go to the finding store only — never to a human (FR-044).
• Triager — implemented in full: verdicts (FR-050), real investigation (FR-051), the three-leg evidence gate — reachability, trust-boundary crossing, concrete impact (FR-052/FR-087) — with mechanical citation resolution and auto-demotion to needs-review (FR-088), the honesty rule (FR-053), reasoning recorded (FR-054).
• Validator-lite — reproduce each true-positive’s headline impact with a runnable curl PoC against your live instance; only this role sets exploited (FR-089).
• Reporter-lite — render true-positives, and only true-positives (FR-057), to markdown through your own Lab 2.4 vuln-report skill.

Dropped, with the spec’s blessing (each is an answer to a [NEEDS CLARIFICATION] it asks): the Orchestrator (a ~20-line sequential runner — the throttled proxy forbids fleets anyway), the Indexer (the target fits in context), the Coverage-Guide (the timebox is the done signal), dependency/secret scanning, the entire §8 substrate, all five extension roles, observability, integrations. The finding store is a directory of JSON files with the lifecycle candidate → triaged → validated → published, keeping the FR-090 fingerprint (hash of path + symbol + class) so a second run files nothing twice.

What no cut may touch: the gates. Merging and omitting roles is sanctioned; dropping a responsibility is not.

Provided#

• The Foundry spec repo: README.md, spec.md, constitution.md, and docs/ — read the README’s “Getting started” section carefully; it is the lab manual
• spec-kit, preinstalled on your instance
• The starter detection-rule corpus: labs/06-capstone/rules/rules.md — eight rules derived from the Lab 5 checklist, versioned per FR-041
• The target: the Lab 5 vulnbox source (labs/05-audit/vulnbox/), plus your hosted instance for the Validator (address in the class portal tab, next to your $CLASS_API_URL and $CLASS_API_KEY values)
• The classroom clarification key: labs/06-capstone/clarifications.md — handed out after your own clarify pass
• Two helpers in labs/06-capstone/provided/ so your 90 minutes go into the pipeline, not plumbing: llm.py (a chat-completions client that survives 429s and connection resets) and citation_gate.py (the FR-088 mechanical citation resolver). Read them — the gate especially is the point of the module.
• A raised token throttle: the class proxy’s per-student rate limit goes up for this lab. Implementing a system like this is token-hungry and that’s expected; your spend cap still applies.

One constraint to design around from the start: the throttle counts estimated prompt tokens + your max_tokens, per request. Whole-file prompts will never fit — chunk your sweeps by function — and a max_tokens set too low silently truncates replies mid-JSON, which is the most confusing failure you can build for yourself.

Ground rules#

Build lab: attack nothing, with one exception — your Validator may reproduce findings against your own hosted vulnbox instance. No parallel agent fleets: the proxy is throttled per key, so run your stages sequentially and back off when it pushes back. Findings stay on your instance and are never published anywhere; “published” in the lifecycle means “written to your local findings directory”. Full text: labs/06-capstone/rules-of-engagement.md.

Steps#

1. Read the constitution end to end (it’s short), then the README’s getting-started flow. Set up spec-kit in a fresh project, install the Foundry constitution, and seed spec.md per the README.

2. The clarify pass (~20 min): run /speckit.clarify and answer the markers yourself, cutting toward the Foundry-lite shape above. This is the exercise: the spec’s §4.2 asks you which roles to merge or omit, and each role’s section prices the omission. Then /speckit.specify, and clarify again if new markers appear — do not carry a [NEEDS CLARIFICATION] into planning. Compare against the classroom key when it’s handed out.

   Hint: how to answer three dozen markers in twenty minutes (step 2)

   Most markers are integration questions with one-line classroom answers: one VM, sequential runner, JSON files on disk, local git, no issue tracker, class proxy as the only provider, no extensions. Spend your thinking on the two that matter: the §4.2 role cut (keep the value chain, drop the scale) and the §5.6/§11.12 testbed question — you have a live testbed, your own vulnbox instance, which is what makes the Validator and the exploited flag worth building.

3. Build (~90 min): /speckit.plan, /speckit.tasks, /speckit.implement. Supervise the way Module 2 taught you: watch the loop, interrupt early, keep tasks small. If the build eats the run window, cut breadth (fewer rules swept, a thinner map) — never a gate.

   Hint: the mechanical citation check is small (step 3)

   FR-088 sounds heavyweight and isn’t: for each cited location (file, function, line-or-symbol), check the file exists, the symbol is defined in it, and the quoted line actually occurs. Thirty lines of Python, no LLM involved. Wire it to run after the Triager writes its verdict and before the verdict is accepted into the store — demote to needs-review on any failure. The point of the lab is watching that demotion fire on a confident, fluent, fabricated finding.

4. Run (~40 min): point it at the vulnbox source; give the Validator your hosted instance’s address. One command, no hand-feeding: candidates accumulate, verdicts land, the gate demotes, the Validator fires curls, reports render through your vuln-report skill.

5. Run it a second time on the unchanged target: zero duplicate findings, or your FR-090 fingerprint has a bug.

6. Score it against your Lab 5 ground truth and bring the numbers to the room walk: caught, missed, hallucinated (split into “the gate demoted it” and “it got through and died against the live instance”), plus any rule-gap — a confirmed exploratory finding no rule in the corpus would have produced (FR-042).

Done when#

An end-to-end run publishes at least one true-positive on the vulnbox with gate-satisfying evidence and no operator intervention — SC-001, on a target you personally proved is vulnerable this morning.

Stretch rungs, in order: (1) the exploited flag set by a live PoC your Validator ran itself; (2) a second target — your own Day-1 ShareBox, or trade source with another student and audit each other’s (source only; attack nothing).

Then the class debrief — the room-wide caught/missed/hallucinated table — and the closing lecture: fine-tuning vs. context engineering (almost always the latter; LoRA in one slide), what it takes to run agentic workflows in production — cost, logging, evaluation at scale — and what to build first when you’re back at your desk.