openai/evals
View on GitHub ↗ ★ 19,688 Python
Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
24 mentions on Hacker News by 22 people: 7 stories, 17 comments · first 2023-03-14 · latest 2025-09-30
20082026
-
…providers,: https://www.promptfoo.dev/docs/configuration/guide/ openai/evals//docs/build-eval.md: https://github.com/openai/evals/blob/main/docs/build-eval.md From https://news.ycombinator.com/item?id=45267271 ; > API facades like OpenLLM…
comment by westurner · 2025-09-30 · in Claude Sonnet 4.5 -
…building our own data sets and evaluations for many of the subtasks. We're using openai's evals ( https://github.com/openai/evals ) as a framework to guide our own tooling. Apart from that, we write in Typescript, Python, and Golang. We…
comment by arcb · 2025-06-10 · in Launch HN: BitBoard (YC P25) – AI agents for healthcare back-offices -
This is exactly it. Here’s the pull request where chess evals were added: https://github.com/openai/evals/pull/45 .
comment by vimbtw · 2024-11-15 · in Something weird is happening with LLMs and chess -
…submission form, but for a specific topic, they do have a public framework to submit evals to. https://github.com/openai/evals They use Common Crawl, so get your site into to there.
comment by fragmede · 2024-05-30 · in Ask HN: Are there any movements to create the equivalent of a sitemap for LLM's? -
…https://github.com/microsoft/promptbench ) [a]( https://github.com/confident-ai/deepeval ) [few]( https://github.com/openai/evals ) [frameworks]( https://github.com/EleutherAI/lm-evaluation-harness ) working on evaluation, however, all of…
comment by renchuw · 2024-02-13 · in Show HN: Faster LLM evaluation with Bayesian optimization -
OpenAI/evals > Building an eval: https://github.com/openai/evals/blob/main/docs/build-eval.md "Robustness of Model-Graded Evaluations and Automated Interpretability" (2023) https://www.lesswrong.com/posts/ZbjyCuqpwCMMND4fv/robustness... :…
comment by westurner · 2023-09-10 · in Asking 60 LLMs a set of 20 questions -
…framework. You can use it to evaluate different models but also your entire prompt chain setup. https://github.com/openai/evals They also have a registry of evals built in.
comment by curo · 2023-07-19 · in Ask HN: How are you improving your use of LLMs in production? -
…How about some evidence that ChatGPT isn't even close? Just clone and run OpenAI's own evals repo https://github.com/openai/evals on the GPT-4 API. It performs terribly on novel logic puzzles and exercises that a clever child could learn…
comment by gooseus · 2023-07-05 · in Introducing Superalignment -
You can get GPT 4 access by submitting an eval if gets merged ( https://github.com/openai/evals ). Here's the one that got me access[1] Although from the blog post it looks like they're planning to open up to everyone soon, so that may…
comment by paulgb · 2023-06-13 · in Function calling and other API updates -
…results quality for a long time. OpenAI recently released a package for quantifying a LLM’s quality https://github.com/openai/evals I don’t know what to say about your CTO. People are definitely overestimating how useful GitHub Copilot is…
comment by jrpt · 2023-05-26 · in 58% of US adults have heard of ChatGPT; 14% have tried it -
See: https://github.com/openai/evals/tree/main/evals/registry/eva...
comment by tikkun · 2023-05-12 · in Ask HN: Logical problems ChatGPT-4 can't solve? -
…and expected responses that have to match? They are trying to crowdsource this with OpenAI evals. https://github.com/openai/evals I'm sure they have a lot of internal benchmarks too, but of course they won't share them. >If not, there can…
comment by sebzim4500 · 2023-04-07 · in We need to tell people ChatGPT will lie to them, not debate linguistics -
https://github.com/openai/evals ?
comment by singularity2001 · 2023-03-29 · in Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa -
https://github.com/openai/evals
comment by drexlspivey · 2023-03-29 · in Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa -
OpenAI has been collecting a ton of evals here https://github.com/openai/evals with many of them including some comments about how well GPT-4 does vs GPT-3.5. You could clone that repo, adapt the oaieval script to run against different…
comment by gooseus · 2023-03-23 · in RWKV RNN: Better than ChatGPT?