Hacker GitHub

← All repos

EleutherAI/lm-evaluation-harness

View on GitHub ↗ ★ 14,371 Python

A framework for few-shot evaluation of language models.

13 mentions on Hacker News by 10 people: 2 stories, 11 comments · first 2021-03-22 · latest 2026-05-13

Mentions per year 13 mentions · busiest year 2023 (5)
20082026
AllStoriesComments NewestTopOldest
  1. ▲ 1
    Show HN: EleutherAI / Lm-Evaluation-Harness
    story by marvinified · 2026-05-13 · 0 comments
  2. …in Amazon HackOn 2024, UC Berkeley AGENTX-2025. Open Source: EleutherAI's LMEvaluationHarness[ https://github.com/EleutherAI/lm-evaluation-harness ], HuggingFace's nanoVLM( https://github.com/huggingface/nanoVLM )[integrating metrics and…
    comment by Domazine · 2025-09-07 · in Ask HN: Who is hiring? (September 2025)
  3. …) to run it on a Couple of A100's, and you can benchmark this using this library ( https://github.com/EleutherAI/lm-evaluation-harness )
    comment by varunvummadi · 2024-04-10 · in Mistral AI Launches New 8x22B MOE Model
  4. …https://github.com/confident-ai/deepeval ) [few]( https://github.com/openai/evals ) [frameworks]( https://github.com/EleutherAI/lm-evaluation-harness ) working on evaluation, however, all of them are quite slow, because LLM are slow if you…
    comment by renchuw · 2024-02-13 · in Show HN: Faster LLM evaluation with Bayesian optimization
  5. ▲ 1
    Language Model Evaluation Harness
    story by tosh · 2023-11-25 · 0 comments
  6. …on over 200 tasks. HuggingFace has a leaderboard tracking performance on a subset of these tasks. https://github.com/EleutherAI/lm-evaluation-harness https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
    comment by rgovostes · 2023-06-18 · in OpenLLaMA 13B Released
  7. …specs you'd expect it to outperform Pythia 6.9B at least... this is running on a HEAD checkout of https://github.com/EleutherAI/lm-evaluation-harness (releases don't support hf-casual) for those looking to replicate/debug. Note, another…
    comment by lhl · 2023-04-20 · in StableLM: A new open-source language model
  8. Yeah, although looks like it currently has some issues with coqa: https://github.com/EleutherAI/lm-evaluation-harness/issues/2... There's also the bigscience fork, but I ran into even more problems (although I didn't try too hard)…
    comment by lhl · 2023-04-19 · in StableLM: A new open-source language model
  9. Are you using https://github.com/EleutherAI/lm-evaluation-harness ?
    comment by lunixbochs · 2023-04-19 · in StableLM: A new open-source language model
  10. …tasks like that, and Eleuther actually has a project collecting a handful of them together; see https://github.com/EleutherAI/lm-evaluation-harness/
    comment by leogao · 2021-06-09 · in GPT-J-6B: 6B JAX-Based Transformer
  11. …open source framework for evaluating language models. We are currently working on building one ( https://github.com/EleutherAI/lm-evaluation-harness ) and are excited to share results when we have the harness built. If you don’t think the…
    comment by stellaathena · 2021-03-29 · in The makers of Eleuther hope it will be an open source alternative to GPT-3
  12. …that GPT-Neo was written in was too much of a pain in the ass because Mesh TensorFlow is the worst. https://github.com/EleutherAI/lm-evaluation-harness
    comment by stellaathena · 2021-03-22 · in GPT Neo: open-source GPT model, with pretrained 1.3B & 2.7B weight models
  13. …that GPT-Neo was written in was too much of a pain in the ass because Mesh TensorFlow is the worst. https://github.com/EleutherAI/lm-evaluation-harness
    comment by stellaathena · 2021-03-22 · in GPT Neo: open-source GPT model, with pretrained 1.3B & 2.7B weight models