Evaluate | SRE-Bench

Model providers

If you train or serve models and want yours on the Leaderboard, contact Zhuo Zhang directly. Every model is evaluated in its own native agent harness, such as Codex for GPT models, on the same tasks, so the numbers stay comparable across the board.

Tell us which model to evaluate and how we can reach it. We handle the rest, and we share the results with you before anything is published.

Custom harnesses

If you build reverse engineering agents and want to test your own harness, tooling, or agent design against the benchmark, contact Zhuo Zhang with two things.

An API key for the model your agent calls. Use a dedicated benchmark credential with a spending limit, and rotate it after the run. The key reaches the container as an environment variable, is never written to any file, and the container can only reach the official endpoint of the provider you name.
A Docker image that contains the complete agent, with a short config file that tells SRE-Bench how to run it. SRE-Bench runs the image against every task, feeds it the challenge prompt, and grades what it leaves in the submission directory.

The rest of this section explains how to build the image and write the config.

The image

The image is a complete Linux agent environment. SRE-Bench installs nothing into it and blocks general Internet access during the run, so every agent CLI, debugger, decompiler, MCP server, language runtime, and library has to be baked in ahead of time. Three things are required.

A shell and sleep. The image is started as an idle container and its normal Docker entrypoint is ignored, so /bin/sh and sleep must exist.
An executable runner at /usr/local/bin/srebench-agent. SRE-Bench invokes this program once per task, as UID 0. It can be written in any language.
Every tool already installed. Package installation during the run will fail, because the network is restricted to the model API.

A minimal Dockerfile has this shape:

FROM ubuntu:24.04

# Install the agent harness and every analysis tool here.
COPY srebench-agent /usr/local/bin/srebench-agent
RUN chmod +x /usr/local/bin/srebench-agent

WORKDIR /workspace

What the runner receives

The full challenge prompt arrives as UTF-8 text on standard input; read until end of file. It contains the task description and the exact submission instructions. A temporary workspace is mounted at /workspace:

/workspace/challenge/     challenge files and supporting material
/workspace/submission/    final files that will be graded

The runner should work inside /workspace and must place its answer under /workspace/submission, following the paths given in the prompt. A few environment variables are set as well: SREBENCH_PROTOCOL (currently 1), SREBENCH_PROVIDERS with the comma-separated providers you selected, and for each of them the usual API key and base URL variables, such as OPENAI_API_KEY and OPENAI_BASE_URL.

What the runner emits

Standard output is a JSON Lines stream, one object per line, flushed promptly so progress is visible. Diagnostic logs go to standard error and are saved with the raw trace. Three event types are recognised:

{"type":"started","model":"example-model-v1"}
{"type":"llm_call"}
{"type":"final","text":"Submission completed."}

started says the harness launched, llm_call is optional and may be repeated whenever a model call begins, and final says the harness is done. The files under /workspace/submission are what gets graded, not the final event. Your harness owns its own step limits, retries, and timeouts; the only limit applied from outside is an optional wall-clock cap per task.

The config file

Alongside the image, send a short YAML file that tells SRE-Bench how to run it:

version: 1
name: example-agent
image: docker.io/example/srebench-agent@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef

providers:
  - openai
  - anthropic

environment:
  EXAMPLE_ENV_NAME: EXAMPLE_ENV_VALUE

version is always 1.
name is a short label used in logs, container names, and result filenames: up to 64 characters of letters, digits, dots, underscores, and hyphens.
image is a Docker or OCI image reference. Pin it by digest so runs are repeatable; the pulled image ID is recorded with the result. Private registries work as long as we can docker login to them.
providers lists which model APIs the container may reach: openai, anthropic, google, or any combination, and we are expanding the list. This list alone decides the network allowlist and which keys are injected. An empty list means no credentials and no external access.
environment holds non-secret variables for your runner, as strings, numbers, or booleans. Names starting with SREBENCH_, API-key variables, and base URL variables are managed by SRE-Bench and cannot be overridden.

API keys never belong in this file. SRE-Bench reads them from its own host environment and injects them into the container at run time.

What to send

A subject line that starts with [SRE-Bench Eval], so the request is not lost among other mail.
For a model: the model name and how we can access it.
For a harness: the image reference pinned by digest, the YAML config, and the API key for each provider it lists.
The name you want to see on the leaderboard, if the results end up public.

Questions about whether your setup fits are welcome too. One email is enough to get started.

Email Zhuo Zhang

What we keep

Once an evaluation finishes, we do not keep any trajectories, and we do not keep the image, the config, or anything else that describes how your agent works. Nothing about your agent design is retained. We may keep the results, meaning the overall and per-task scores but not the submissions themselves, and whether those scores are published on the leaderboard or stay private is decided together with you.