Issues

• --max-workers > 1 crashes: Pool.starmap passes arguments to benchmark_suite in the wrong order

  ethz-spylab/agentdojo#212 · Filter by author yomna29 opened on Sep 30, 2026

• Detector-level results on AgentDojo tool outputs: 10 open-source injection classifiers, incl. the default transformers_pi_detector model

  ethz-spylab/agentdojo#211 · Filter by author rudras-star opened on Sep 24, 2026
  1

• tool_filter produces zero tool calls with a local model that tool-calls normally without it

  ethz-spylab/agentdojo#209 · Filter by author giselleevita opened on Sep 23, 2026
  1

• Replaying the ground truth as a hijacked agent: a model-free evaluation mode, and one accounting detail

  ethz-spylab/agentdojo#208 · Filter by author cgrtml opened on Sep 22, 2026

• LocalLLM pipeline silently drops fenced tool calls and never caps generation (affects all local-model results)

  ethz-spylab/agentdojo#202 · Filter by author bithabib opened on Sep 20, 2026
  1

• Workspace v1.2.2 + important_instructions: 0/560 attack success on claude-sonnet-4-6 — is the suite still discriminating?

  ethz-spylab/agentdojo#201 · Filter by author 0xShin0221 opened on Sep 20, 2026

• Three tool interfaces (banking, car rental, Slack invite) promise more than their bodies currently deliver

  ethz-spylab/agentdojo#194 · Filter by author rohithreddybc opened on Sep 12, 2026
  2

• transformers_pi_detector defense is hardcoded to single-message mode — no trajectory-aware option in the benchmarked defense list

  ethz-spylab/agentdojo#192 · Filter by author ArksherX opened on Sep 10, 2026

• slack utility check raises KeyError when post_webpage does not execute

  ethz-spylab/agentdojo#188 · Filter by author mohameduk opened on Sep 3, 2026
  1

• 9% of (user task, injection task) pairs stay inside the tools the user has already authorised - and an offer to contribute more

  ethz-spylab/agentdojo#187 · Filter by author luiacuaniello opened on Sep 2, 2026

• Optional EvalPort interop for tasks and TaskResults

  ethz-spylab/agentdojo#185 · Filter by author adhabnr-ux opened on Aug 22, 2026

• Proposal: generic pre-execution authorizer for FunctionsRuntime

  ethz-spylab/agentdojo#184 · Filter by author giselleevita opened on Aug 19, 2026

• Banking injection tasks 0/1/3: operator precedence makes the subject check dead code

  ethz-spylab/agentdojo#181 · Filter by author Prabscodes opened on Aug 17, 2026
  1

• Submission scope for a Banking-specific defense result

  ethz-spylab/agentdojo#178 · Filter by author piyushkr0509 opened on Jul 30, 2026

• Question about ASR results on the Workspace EASY subset

  ethz-spylab/agentdojo#172 · Filter by author Aquarids opened on Jul 18, 2026

• slack injection_task_5: security_from_traces simulates state from attempted calls, scoring blocked (never-executed) sequences as attack success

  ethz-spylab/agentdojo#168 · Filter by author webpro255 opened on Jul 4, 2026
  2

• Workspace UserTask34 can pass after omitting hot water from the Chocolate Chip Cookies ingredients

  ethz-spylab/agentdojo#162 · Filter by author gss10282023 opened on May 17, 2026
  1

• Banking UserTask5 utility checks the old $50 Spotify transaction instead of the required new $5 payment

  ethz-spylab/agentdojo#161 · Filter by author gss10282023 opened on May 17, 2026
  1

• Question: would AgentDojo welcome a detection-evaluator extension? (proposal + offer to draft)

  ethz-spylab/agentdojo#160 · Filter by author eeee2345 opened on May 11, 2026

• tool_filter defense sends "vllm_parsed" as model name instead of real model id

  ethz-spylab/agentdojo#158 · Filter by author francescopirox opened on Apr 30, 2026
  2

• Feature: LLM-as-a-Judge Evaluation Framework

  ethz-spylab/agentdojo#156 · Filter by author MuneezaAzmat opened on Apr 22, 2026

• Some doubts about the security metric

  ethz-spylab/agentdojo#154 · Filter by author lrandirene opened on Apr 9, 2026
  1

• Streaming Evaluations

  ethz-spylab/agentdojo#151 · Filter by author ACMCMC opened on Apr 7, 2026

• LocalLLM _parse_model_output only preserves the first function call

  ethz-spylab/agentdojo#150 · Filter by author DerOeko opened on Apr 3, 2026

• Support arbitrary OpenAI-compatible endpoints without source modifications or port forwarding

  ethz-spylab/agentdojo#148 · Filter by author dmaniloff opened on Mar 23, 2026