SWE-bench/experiments

Open sourced predictions, execution logs, trajectories, and results from model inference + evaluation runs on the SWE-bench task.

3 mentions on Hacker News by 3 people: 0 stories, 3 comments · first 2024-08-12 · latest 2026-04-23

Mentions per year 3 mentions · busiest year 2024 (1)

20082026

All Stories Comments Newest Top Oldest

…number there rings a lot of questions on my end. If you look at the SWEBench official submissions: https://github.com/SWE-bench/experiments/tree/main/evaluatio... , filter all models after Sonnet 4, and aggregate ALL models' submission…
comment by kaonashi-tyc-01 · 2026-04-23 · in GPT-5.5
…submitting you need to check a box that there was only 1 attempt per problem. See here for example: https://github.com/SWE-bench/experiments/pull/219 Building multiple attempts into your agent is stretching the rules, even if technically…
comment by gronky_ · 2025-08-12 · in Qodo CLI agent scores 71.2% on SWE-bench Verified
looks like they trained their model on SWE-bench and tried to submit https://github.com/swe-bench/experiments/pull/45
comment by uzumak · 2024-08-12 · in Genie: Best AI Software Engineer