openai/SWELancer-Benchmark
View on GitHub ↗ ★ 1,428 archived
This repo contains the dataset and code for the paper "SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?"
5 mentions on Hacker News by 4 people: 0 stories, 5 comments · first 2025-02-19 · latest 2025-02-27
20082026
-
…I assumed from paper and repo overview? For further clarification: 1. See the issue example #14268 https://github.com/openai/SWELancer-Benchmark/tree/08b5d3dff... . It has a patch that is supposed to "reintroduce" the bug into the codebase…
comment by izucken · 2025-02-20 · in SWE-Lancer: a benchmark of freelance software engineering tasks from Upwork -
…just two examples of widespread problems with the introduced bugs and the tests. How about this - https://github.com/openai/SWELancer-Benchmark/blob/08b5d3dff... (Intentionally use raw character count instead of HTML-converted length) Or…
comment by Snuggly73 · 2025-02-19 · in SWE-Lancer: a benchmark of freelance software engineering tasks from Upwork -
For future reference, permalink to the original commit with the RADICAL BUG comment: https://github.com/openai/SWELancer-Benchmark/blob/a8fa46d2b... The new version (as of now) still has a comment making it obvious that there's an…
comment by yorwba · 2025-02-19 · in SWE-Lancer: a benchmark of freelance software engineering tasks from Upwork -
…countries, so the solution had to introduce country specific regex etc. The "reintroduced bug" - https://github.com/openai/SWELancer-Benchmark/blob/main/issu... is just taking that new code and adding , to two countries.... 2. Room showing…
comment by Snuggly73 · 2025-02-19 · in SWE-Lancer: a benchmark of freelance software engineering tasks from Upwork