I Built isu-bench, a Benchmark That Makes AI Agents Solve ISUCON

I built isu-bench, which measures how far a coding agent can speed up an ISUCON-style web service it has never seen, given only 15 benchmark runs. This post also covers the results for Claude Opus 5.5, GPT-6.1 Sol, GPT-6 Astra and DeepSeek V4.1 Flash.

Score charts for every run are at isu-bench.trasta.dev.

Why I built it#

I built isu-bench for the ISUCON2026 contest on October 31, to decide which model to hand the contest to. By the way, in practice Opus clearly looked better, yet for some reason I went with Astra on the day and crashed and burned.

ISUCON2026 was a good problem with elements worth taking in, so after the contest I improved the problem before publishing it. There are also rumors of Astra 6.2 and Fable 5.5, and a big motivation was being ready to measure new models as soon as they come out.

Rules#

• The agent gets the docs, the reference implementation and SSH to three servers, and may run bench up to 15 times
• Failed runs count too, and the 15th run's score is the final score. A FAIL scores 0
• After the 15th run the three servers are rebooted; if the application does not respond, the score is 0
• No time limit. If the agent stops early, it is resumed up to 3 times with "Continue. N of 15 benchmark runs remain."
• Web search is disabled. All traffic goes through a proxy that only allows the model API, package registries and SSH to the servers

Supported agents are Claude Code, Codex and DeepSeek Harness. Each run creates its own VPC and EC2 instances and deletes them all at the end.

A new problem: ISUMAIL#

I first measured on ISUCON14 and ISUCON13. But both problems are old, and the models may have learned their solutions, so I had no confidence that the numbers measured anything. To run on a problem no model has learned, I made a new one, ISUMAIL.

ISUMAIL is a newsletter delivery service: at the scheduled time it renders each subscriber's copy and hands it to an external mail provider. The provider requires a proof-of-work stamp per mail and keeps a reputation per sending server. The three servers differ in CPU architecture, speed and memory, and all traffic between them goes through a router that limits bandwidth.

To keep it out of training data, the source (reference implementation, benchmarker, data generator and specification) is not on GitHub and ships only as public AMIs. So this post does not go into the problem any further than the README does.

The problem design, reference implementation and benchmarker were all made by Claude and Codex. I had subagents and Codex review it for how well it works as an ISUCON problem along the way. Honestly, nothing was particularly hard; the AI built it with ease. What times we live in.

The prompt is not optimized#

The only thing the agent gets is the prompt below, which states the rules. Each agent's CLI is started headless with this prompt, with no instructions on how to proceed or how to use subagents. Neither the prompt nor the harness is optimized, so optimizing the harness the way the ISUCON2026 winner did would probably change the results.

This is the prompt used for ISUMAIL. For ISUCON14 and ISUCON13 only the problem name and the description of the benchmark change.

You are competing in ISUMAIL, a web application performance tuning contest.

The contest manual and the application specification are in docs/ (in Japanese).
The reference implementation is in webapp/.
The application runs on three servers: s1, s2 and s3. Connect with `ssh s1` and so on;
you log in as the `isucon` user, who has sudo. The initial Go implementation
(isumail-go.service) is running, with its source in /home/isucon/webapp on each server.
Your workspace and the servers reach the internet only through a proxy that allows
package registries.

Goal: maximize the score of your 15th benchmark run.

- Run the benchmark with `bench`. It sends isumail.test to s1 and t.isumail.test to s3 through the router for 70 seconds, runs the provider, and prints the result.
  You have exactly 15 runs. Every run counts, including failures.
- The 15th run's result is your final score, and a FAIL scores 0. This session ends
  as soon as the 15th run finishes.
- After that, the three servers are rebooted. If the application does not come back
  and respond, the score is 0.
- There is no time limit.
- You may change anything on the three servers, including code, language, database,
  middleware, configuration and how the servers are used, as long as the application
  behaves as the docs describe.
- Parts of the manual about the contest schedule, the portal, CloudFormation, Discord,
  support and the post-contest re-check do not apply; this prompt replaces them.
- Do not look up ISUMAIL solutions, write-ups or the benchmarker source.

No one will answer questions. Work autonomously until you have used all 15 runs.

Results#

The isu-bench score is the final ISUMAIL score divided by its estimated theoretical maximum, 1,020,000. Language is Go, one trial per configuration.

Even Opus, the highest, reached only about half of the theoretical maximum. Below it came Sol, Astra and DeepSeek, each 2 to 3 times apart from the next. Astra, which I used in the contest, scored 13.2%.

Public solutions of ISUCON14 and ISUCON13 may be in the training data, so their results are kept as extras outside the isu-bench score. They are in the lower half of the chart above and in the README. On ISUCON14, Astra scored 592,820, about 10 times the contest winner's 58,153.

How each model plays it#

The limit here is the number of benchmark runs, not time, and it was fun to see each model use its 15 runs differently.

Opus tries to land a big change first. It ran its first benchmark about a minute in, then worked for about 40 minutes without running another, and jumped from 2,046 to 438,146 on the second run. The third run failed, and from the fourth on it crept up between 470,000 and 510,000.

Astra does nothing of the sort and just keeps spending runs. It ran the benchmark about every 5 minutes and, with two FAILs along the way, raised its score little by little (32,749, 45,873, 55,941, ...) to 135,079 on the 15th run. Sol likewise ran about every 5 minutes.

Cost#

One ISUMAIL run costs about $0.64 per hour in EC2. The four trials here came to about $4.2 in total: about $1.7 for Opus and about $0.8 each for the others. Including trial runs while making the problem and the ISUCON14 and 13 runs, November 1 and 2 cost about $17.

Adding models and donations#

I plan to measure Astra 6.2 and Fable 5.5 when they come out. If there is another model or agent you want measured, please reach out to @trasta on X.

Every run costs money, so donations would make me very happy.