NVIDIA/RULER
View on GitHub ↗ ★ 1,629 Python
This repo contains the source code for RULER: What’s the Real Context Size of Your Long-Context Language Models?
10 mentions on Hacker News by 9 people: 1 story, 9 comments · first 2024-04-13 · latest 2025-07-11
20082026
-
…up being some confusion with the RULER benchmark from NVIDIA given the (somewhat shared) domain: https://github.com/NVIDIA/RULER EDIT: by shared I only mean the adjacency to LLMs/AI/ML, RL is a pretty big differentiator though and project…
comment by spmurrayzzz · 2025-07-11 · in Show HN: RULER – Easily apply RL to any agent -
https://github.com/NVIDIA/RULER results in benchmark other than needle in haystack seem solid all the way to 128k
comment by avereveard · 2024-12-08 · in Show HN: Countless.dev – A website to compare every AI model: LLMs, TTSs, STTs -
It should be benchmarked against something like RULER[1] 1: https://github.com/hsiehjackson/RULER (RULER: What’s the Real Context Size of Your Long-Context Language Models)
comment by smusamashah · 2024-08-29 · in 100M Token Context Windows -
Absolutely. Gemini results tend to drop off after 128k tokens according to RULER: https://github.com/hsiehjackson/RULER
comment by cathalc · 2024-05-15 · in Gemini Flash -
…ability to do anything complex drops off hugely after 32K tokens. RULER is a much better test: https://github.com/hsiehjackson/RULER > Despite achieving nearly perfect performance on the vanilla needle-in-a-haystack (NIAH) test, all models…
comment by thorum · 2024-05-14 · in GPT-4o's Memory Breakthrough – Needle in a Needlestack -
▲ 2
Ruler: What's the Real Context Size of Your Long-Context Language Models?story by thorum · 2024-05-09 · 0 comments
-
Can someone test this with ruler please? https://github.com/hsiehjackson/RULER In practice all of these long contexts show degraded performance (there's a table on the repo). For my NLP work I find that GPT-4-turbo is much worse after…
comment by msp26 · 2024-04-29 · in Gradient AI Releases 1M Context Llama 3 8B