HammingHQ/bug-in-the-code-stack
View on GitHub ↗ ★ 34 Jupyter Notebook
A new benchmark for measuring LLM's capability to detect bugs in large codebase.
3 mentions on Hacker News by 1 person: 1 story, 2 comments · first 2024-05-20 · latest 2024-05-23
20082026
-
…(yay open-source!) - Gemini 1.0 Pro was bad across the board (super surprising) Link to repo: https://github.com/HammingHQ/bug-in-the-code-stack See full results here: https://hamming.ai/blog/bug-in-the-codestack
comment by sumanyusharma · 2024-05-23 · in New code-focused LLM needle in the haystack benchmark -
▲ 6
New code-focused LLM needle in the haystack benchmarkstory by sumanyusharma · 2024-05-23 · 1 comment
-
…this benchmark more useful! Link to results: https://hamming.ai/blog/bug-in-the-codestack Repo: https://github.com/HammingHQ/bug-in-the-code-stack
comment by sumanyusharma · 2024-05-20 · in Can LLMs find bugs in large Python codebases?