{ "module": "SDD-B03 — InjecAgent: The Bridge Benchmark", "course": "2B — Securing & Attacking Harnesses and LLMs", "version": "1.0.0", "duration_minutes": 30, "total_questions": 15, "bloom_distribution": { "target": "20% recall / 40% application / 40% analysis-design", "actual": { "recall": 3, "application": 6, "analysis": 6 } }, "passing_score_percent": 70, "questions": [ { "id": "Q01", "bloom": "recall", "type": "multiple_choice", "prompt": "What does the ~50% InjecAgent finding actually measure?", "options": [ "A constant of nature — roughly half of all agents, defended or not, will always be injectable.", "A population baseline for UN-DEFENDED agentic harnesses with no injection defenses — roughly half of agentic tasks fail when a tool output is adversarial. It is the starting line, not a constant or a target.", "The expected injection rate for any production-grade defended harness.", "The target injection rate that a hardened harness should aim to reach." ], "answer_index": 1, "rationale": "The ~50% is a baseline for un-defended harnesses. It is NOT a constant, NOT a defended harness's expected rate, and NOT a target. A harness scoring 50% has measured a vulnerability, not managed one. The managed version is the defended delta (before/after). Misreading the baseline as a target is the most common InjecAgent misuse." }, { "id": "Q02", "bloom": "recall", "type": "multiple_choice", "prompt": "Which attack vector does InjecAgent measure, and why is it the right surface?", "options": [ "Direct prompt injection in the user's prompt, because it is the most common attack.", "Indirect prompt injection via TOOL OUTPUTS, because the agent must read tool outputs to function and cannot structurally distinguish tool-output data from instruction policy — both arrive as text in the context window. Direct injection is largely solved; indirect via tool outputs is the unsolved, structural surface.", "Model weight extraction, because weights are the highest-value target.", "Sandbox escape, because runtime isolation is the weakest layer." ], "answer_index": 1, "rationale": "InjecAgent measures indirect injection via tool outputs. Direct injection is largely closed by input sanitization and system-prompt primacy. Indirect via tool outputs is structural: the agent must read tool outputs to function, so there is no 'refuse attacker data' option — the attacker's data is indistinguishable from legitimate data. A defense that holds under direct injection but folds under indirect-via-tool-output holds on the test and fails in production." }, { "id": "Q03", "bloom": "recall", "type": "multiple_choice", "prompt": "What are the four roles InjecAgent plays across the two courses?", "options": [ "Scanner, fuzzer, explainer, reporter.", "Quality gate (2A: is my harness injectable?), measurement instrument (2B: did this defense hold, and by how much?), regression gate (both: did this change open a surface?), engagement scoper (B12: which chains do I run?).", "Builder tool, attacker tool, defender tool, auditor tool.", "Input filter, output filter, egress gate, ingress gate." ], "answer_index": 1, "rationale": "The benchmark code does not change between roles. What changes is the question, the lens, and the authorization. A 2A graduate who only knows the quality-gate role will under-use the benchmark in 2B. SDD-B03 upgrades the role to measurement instrument, with the before/after delta as the unit." }, { "id": "Q04", "bloom": "application", "type": "multiple_choice", "prompt": "Your team adds a structured-output sanitizer and reports 'we're now at 9% injection on InjecAgent.' What is missing, and why does it matter?", "options": [ "Nothing — 9% is a strong result and speaks for itself.", "The BASELINE (pre-sanitizer) number. Without it the 9% is meaningless: if the baseline was 14%, the sanitizer did almost nothing; if it was 50%, it did serious work. You cannot tell from the 9% alone. Every claim must be a (before, after, configuration) triple.", "The number of developers who worked on the sanitizer.", "The cost of the sanitizer in API calls per run." ], "answer_index": 1, "rationale": "This is the most common InjecAgent anti-pattern. A defended number without its baseline is a claim with no evidence. The honest effectiveness metric is the delta — (before, after, configuration) — with the model version, sampling params, and defense settings pinned so the triple is reproducible. No triple, no claim." }, { "id": "Q05", "bloom": "application", "type": "multiple_choice", "prompt": "You run InjecAgent against a harness and a tool scores 60% injectable. What additional breakdown turns this headline into a defense prescription?", "options": [ "The tool's name and version number.", "The ATTACK-TYPE taxonomy: how much of the 60% is credential-exfil vs disallowed-tool vs scope-escape vs policy-override vs action-redirection. Each type maps to a different defense (egress gate vs tool-call policy vs scope gate vs instruction hierarchy vs session-level intent detection).", "The number of lines of code in the tool.", "The programming language the tool was written in." ], "answer_index": 1, "rationale": "A single 60% number is a headline; the prescription comes from the attack-type breakdown. A tool failing on credential-exfil needs a deterministic egress gate; one failing on action-redirection needs session-level intent detection. The taxonomy is mandatory in 2B because a B12 engagement prescribes defenses, not headlines." }, { "id": "Q06", "bloom": "application", "type": "multiple_choice", "prompt": "You are comparing a baseline run (defenses off) at temperature 0.2 to a defended run (defenses on) at temperature 0.0, on the same task set. Is the resulting delta valid? Why or why not?", "options": [ "Yes — the task set is the same, so the delta is valid.", "No — the SAMPLING PARAMETER changed between runs. Injection success is sampling-dependent. The delta is only valid if EVERYTHING is constant except the defense under test: same task set, same pinned model version, same sampling params. This delta is noise, not a measurement.", "Yes — temperature only affects creativity, not injection resistance.", "No — you should never change temperature for any reason." ], "answer_index": 1, "rationale": "InjecAgent's stochasticity means every comparison must hold everything constant except the variable under test. The three pinned variables are model version, sampling parameters, and defense configuration. A defended number measured under different sampling than the baseline is noise. The configuration card is what makes the triple reproducible." }, { "id": "Q07", "bloom": "application", "type": "multiple_choice", "prompt": "A vendor says 'our product blocks prompt injection.' What evidence do you demand?", "options": [ "A marketing whitepaper describing the approach.", "An InjecAgent (or equivalent) (before, after, configuration) triple: baseline rate without the product, defended rate with it, same task set + pinned model + same sampling, plus the failure transcripts for review. 'Blocks prompt injection' without a measured delta against a named benchmark is a claim with no evidence.", "A list of customers who deployed the product.", "A certification from a standards body." ], "answer_index": 1, "rationale": "The discipline from B03.2: never accept a defense claim without the delta. The triple is the evidence; the transcripts are the proof. A vendor reporting only the defended number, or a claim with no benchmark, is unfalsifiable. Demand the triple and the transcripts." }, { "id": "Q08", "bloom": "application", "type": "multiple_choice", "prompt": "You read an InjecAgent failure transcript: an injected tool output steered the agent to call a privileged tool it should not have called. Reading this through the SDD-B01 (OWASP) lens, what chain is it?", "options": [ "A single ASI01 goal-hijacking bug — no chain.", "ASI07 (insecure output handling — the tool output was not sanitized/structured) → ASI01 (goal hijacking — the injection overrode the task) → ASI05 (tool abuse — the privileged call). The transcript is a cross-row chain, not a single bug.", "ASI08 supply chain — the tool was compromised.", "ASI09 resource exhaustion — the tool consumed too many tokens." ], "answer_index": 1, "rationale": "Every transcript is a partial chain. The injected output is ASI07, the agent following it is ASI01, the off-task privileged call is ASI05 (or ASI03 if beyond scope). The benchmark produces the raw chain material; the taxonomy produces the finding. Reading transcripts in isolation (as single bugs) is the anti-pattern." }, { "id": "Q09", "bloom": "application", "type": "multiple_choice", "prompt": "Why do deterministic defenses (structured outputs, deterministic egress, scope gates) produce larger InjecAgent deltas than probabilistic ones (LLM-as-judge, sanitization models)?", "options": [ "They do not — probabilistic defenses always produce larger deltas because they understand semantics.", "Deterministic defenses deny the injection a place to live (structured outputs) and a path out (deterministic egress) — they move the needle toward single digits. Probabilistic defenses carry their OWN attack surface (the judge is a model, so it is injectable — cf SDD-B04) and catch only the novel patterns the deterministic rules don't enumerate. They are the SECOND line, not the first.", "Deterministic defenses are cheaper to run, so they are always preferred.", "Probabilistic defenses are illegal in production." ], "answer_index": 1, "rationale": "Defense-in-depth is measurable via InjecAgent. Deterministic-first harnesses sit well below the baseline; probabilistic-only harnesses sit near it. The probabilistic layer (CrabTrap's judge) catches novel patterns but is itself injectable (the judge reads text). The right architecture is deterministic-first, probabilistic-second. InjecAgent shows this clearly." }, { "id": "Q10", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why is a measured-but-not-gated injection threshold a documented vulnerability rather than a managed one?", "options": [ "Because measurement itself introduces risk.", "Because a threshold that is not enforced by the gate erodes the moment a deadline arrives — the team raises it 'just this once' and the injection surface reopens silently. The gate turns a measurement into a CONTROL: binary at merge (below threshold = allow, above = block until fixed OR threshold consciously raised with sign-off). Threshold = policy; enforcement = mechanical.", "Because thresholds are always wrong and should never be used.", "Because only the model provider can set thresholds." ], "answer_index": 1, "rationale": "The regression gate is the only honest enforcement of an injection-resistance standard. A standard measured but not gated erodes under deadline pressure. The gate is mechanical: it runs every merge, compares to threshold, and blocks. The threshold is a policy choice (5%? 10%?); the enforcement is not." }, { "id": "Q11", "bloom": "analysis", "type": "multiple_choice", "prompt": "How does running InjecAgent first scope a B12 engagement faster than jumping straight to the OWASP/Microsoft taxonomies?", "options": [ "It does not — the taxonomies are always run first.", "The per-tool per-attack-type scorecard identifies the HIGH-SCORE tools (entry points) and HIGH-SCORE attack types (chains to construct); the transcripts are the chain recon. From there the engagement branches: credential-exfil → CrabTrap/IronCurtain attacks; action-redirect → Microsoft zero-click chain recon; scope-escape → OWASP ASI03/05. Skipping the benchmark = scoping blind, wasting time on low-priority surfaces.", "InjecAgent replaces the taxonomies entirely, so you only run the benchmark.", "The benchmark only works after the taxonomies identify the chains." ], "answer_index": 1, "rationale": "The benchmark routes traffic to the taxonomies; it does not replace them. Running InjecAgent first is the fastest scoper: minutes to identify which tools are injectable and for which attack types, then route to the matching chain procedures. A B12 engagement that skips the pass scopes blind." }, { "id": "Q12", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why can the indirect-injection-via-tool-output problem NOT be solved at the model level, and where must the defense live?", "options": [ "It can be — just train the model to ignore injected instructions.", "The agent CANNOT structurally distinguish tool output (data) from instruction (policy) because both arrive as TEXT in the context window, and any marker can be forged by the injection. The defense cannot be 'make the model smarter'; it must be ARCHITECTURAL — treat tool outputs as untrusted data (structured layer), enforce schema on outputs, gate egress deterministically. The defense lives in the HARNESS, not the model.", "The defense must live in the model provider's API.", "The problem is unsolvable and should be accepted as a risk." ], "answer_index": 1, "rationale": "This is the structural property InjecAgent measures and every 2B defense targets. There is no reliable in-context marker the model can use to separate data from policy — any marker is forgeable. So the defense is architectural: the harness wraps tool outputs, enforces structure, and gates egress. The model cannot save you; the harness must." }, { "id": "Q13", "bloom": "analysis", "type": "multiple_choice", "prompt": "How should an InjecAgent failure transcript be handled in a B12 engagement, per the B0 evidence rules?", "options": [ "As a public artifact — publish it for transparency.", "As a SENSITIVE artifact — an attack roadmap. Classify at capture time: Restricted if it contains retrieved data (PII); Destroy-on-Report if it contains working injection recipes. The evidence store enforces the class automatically. The transcript is the proof behind the delta and the chain finding.", "Delete it immediately — transcripts have no value.", "Store it permanently in a shared wiki for the whole team." ], "answer_index": 1, "rationale": "The transcripts show exactly how an injection steered the agent — the specific text, the off-task action, any leaked data. They are attack roadmaps. B0's evidence classes apply: Restricted (PII/weights in retrieved context, destroy on report), Destroy-on-Report (working injection recipes, never cross-client). Classification must be automatic at capture time." }, { "id": "Q14", "bloom": "analysis", "type": "multiple_choice", "prompt": "Distinguish SDD-B03 (2B) from SDD-12 (2A) given the benchmark code is identical.", "options": [ "They are the same module — no difference.", "SDD-12 read InjecAgent as a QUALITY GATE (builder's lens: 'is my harness injectable?', unit = per-tool score, the number is the deliverable). SDD-B03 reads it as a MEASUREMENT INSTRUMENT (red-team's lens: 'did this defense hold and by how much?', unit = before/after delta, the triple is the finding). The stance, authorization (B0), transcript handling, and unit of analysis (chain not tool) all change. Same benchmark, opposite side of the bridge.", "SDD-12 is the offensive version; SDD-B03 is the defensive version.", "SDD-B03 uses a different benchmark than SDD-12." ], "answer_index": 1, "rationale": "The code is identical. The difference is role and stance. SDD-12 was the builder hardening their own harness (quality gate). SDD-B03 is the authorized red-teamer measuring a target's defenses (measurement instrument). The question, lens, authorization, handling, and unit of analysis all shift across the bridge." }, { "id": "Q15", "bloom": "analysis", "type": "multiple_choice", "prompt": "Design the InjecAgent procedure to measure whether a deterministic egress gate (IronCurtain-style) is effective. What are the runs, what is pinned, and what is the deliverable?", "options": [ "Run once with the gate on; report the single number.", "Three things. RUNS: (1) baseline with the gate OFF (or against the pre-gate commit), (2) defended with the gate ON, same task set. PINNED: model version, sampling params (eg temp 0.0), task set (same injected outputs), the ONLY variable changed is the gate. DELIVERABLE: a per-tool per-attack-type scorecard for both runs + the delta (eg 'credential-exfil dropped from 48% to 9% on web-fetch') + the failure transcripts for review. The triple (before, after, config) is the effectiveness claim.", "Run the gate against production traffic and count blocks.", "Ask the gate vendor for their benchmark numbers." ], "answer_index": 1, "rationale": "This is the B03.2 procedure made concrete. The before run gives the denominator; the after run gives the numerator; the pinned configuration makes the delta reproducible. The per-attack-type scorecard tells you the gate closed credential-exfil but maybe not action-redirection. The transcripts are the proof. The triple is the claim. No baseline = no claim." } ] }