Hacker GitHub

← All repos

lechmazur/nyt-connections

View on GitHub ↗ ★ 250 Python

Benchmark that evaluates LLMs using 759 NYT Connections puzzles extended with extra trick words

24 mentions on Hacker News by 3 people: 2 stories, 22 comments · first 2024-10-22 · latest 2026-05-29

Mentions per year 24 mentions · busiest year 2025 (20)
20082026
AllStoriesComments NewestTopOldest
  1. …https://github.com/lechmazur/buyout_game 10th out 36. https://github.com/lechmazur/pact/ 14th out 25. https://github.com/lechmazur/nyt-connections/ 60th out 81. https://github.com/lechmazur/debate 16th out of 29.
    comment by zone411 · 2026-05-29 · in The mysterious Hy3 LLM is topping OpenRouter Model Rankings by a large margin
  2. They're improved compared to 4.5 on my Extended NYT Connections benchmark ( https://github.com/lechmazur/nyt-connections/ ). Sonnet 4.6 Thinking 16K scores 57.6 on the Extended NYT Connections Benchmark. Sonnet 4.5 Thinking 16K scored…
    comment by zone411 · 2026-02-17 · in Claude Sonnet 4.6
  3. Scores 92.0 on my Extended NYT Connections benchmark ( https://github.com/lechmazur/nyt-connections/ ). Gemini 2.5 Flash scored 25.2, and Gemini 3 Pro scored 96.8.
    comment by zone411 · 2025-12-17 · in Gemini 3 Flash: Frontier intelligence built for speed
  4. I've benchmarked it on the Extended NYT Connections benchmark ( https://github.com/lechmazur/nyt-connections/ ): The high-reasoning version of GPT-5.2 improves on GPT-5.1: 69.9 → 77.9. The medium-reasoning version also improves: 62.7 →…
    comment by zone411 · 2025-12-11 · in GPT-5.2
  5. Sets a new record on the Extended NYT Connections: 96.8. Gemini 2.5 Pro scored only 57.6. https://github.com/lechmazur/nyt-connections/
    comment by zone411 · 2025-11-18 · in Gemini 3
  6. Sets a new record on the Extended NYT Connections benchmark: 96.8 ( https://github.com/lechmazur/nyt-connections/ ). Grok 4 is at 92.1, GPT-5 Pro at 83.9, Claude Opus 4.1 Thinking 16K at 58.8. Gemini 2.5 Pro scored 57.6, so this is a huge…
    comment by zone411 · 2025-11-18 · in Gemini 3
  7. I've benchmarked it on the Extended NYT Connections ( https://github.com/lechmazur/nyt-connections/ ). It scores 20.0 compared to 10.0 for Haiku 3.5, 19.2 for Sonnet 3.7, 26.6 for Sonnet 4.0, and 46.1 for Sonnet 4.5.
    comment by zone411 · 2025-10-15 · in Claude Haiku 4.5
  8. Matches Grok 4 at the top of the Extended NYT Connections leaderboard: https://github.com/lechmazur/nyt-connections/
    comment by zone411 · 2025-09-20 · in Grok 4 Fast
  9. …to o3 Medium Reasoning, and GPT-5 Mini Medium Reasoning scores close to o4-Mini Medium Reasoning: https://github.com/lechmazur/nyt-connections/
    comment by zone411 · 2025-08-07 · in GPT-5
  10. I benchmarked the 120B version on the Extended NYT Connections (759 questions, https://github.com/lechmazur/nyt-connections ) and on 120B and 20B on Thematic Generalization (810 questions, https://github.com/lechmazur/generalization ).…
    comment by zone411 · 2025-08-05 · in Open models by OpenAI
  11. Grok 4 sets a new high score on my Extended NYT Connections benchmark (92.4), beating o3-pro (87.3): https://github.com/lechmazur/nyt-connections/ . Grok 4 Heavy is not in the API.
    comment by zone411 · 2025-07-10 · in Grok 4 Launch [video]
  12. …my benchmarks so far. Got first place in two of them: https://github.com/lechmazur/confabulations https://github.com/lechmazur/nyt-connections https://github.com/lechmazur/generalization https://github.com/lechmazur/writing
    comment by zone411 · 2025-06-12 · in First thoughts on o3 pro
  13. …Pro Preview (05-06), scoring 58.7. The decline observed between 03-25 and 05-06 has been reversed - https://github.com/lechmazur/nyt-connections/ .
    comment by zone411 · 2025-06-05 · in Gemini-2.5-pro-preview-06-05
  14. On the extended version of NYT Connections - https://github.com/lechmazur/nyt-connections/ : Claude Opus 4 Thinking 16K: 52.7. Claude Opus 4 No Reasoning: 34.8. Claude Sonnet 4 Thinking 64K: 39.6. Claude Sonnet 4 Thinking 16K: 41.4 (Sonnet…
    comment by zone411 · 2025-05-22 · in Claude 4
  15. …GPT-4.5 Preview 34.2 8 Claude 3.7 Sonnet Thinking 16K 33.6 9 Qwen QwQ-32B 16K 31.4 10 o1-mini 27.0 https://github.com/lechmazur/nyt-connections/
    comment by zone411 · 2025-03-25 · in Gemini 2.5
  16. It significantly improves upon GPT-4o on my Extended NYT Connections Benchmark. 22.4 -> 33.7 ( https://github.com/lechmazur/nyt-connections ).
    comment by zone411 · 2025-02-27 · in GPT-4.5
  17. …benchmark. Claude 3.7 Sonnet scores 18.9. I'll run my other benchmarks in the upcoming days. https://github.com/lechmazur/nyt-connections/
    comment by zone411 · 2025-02-25 · in Claude 3.7 Sonnet and Claude Code
  18. …Benchmark: https://github.com/lechmazur/writing Extended NYT Connections LLM Benchmark: https://github.com/lechmazur/nyt-connections/ and a couple more that I haven't updated very recently.
    comment by zone411 · 2025-02-10 · in PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models
  19. …over the o1-mini (42.2) and surpassing DeepSeek R1 (54.4), but it falls short of the o1 (90.7). ( https://github.com/lechmazur/nyt-connections/ )
    comment by zone411 · 2025-02-01 · in OpenAI O3-Mini
  20. …benchmark on it: 18.6, up from 14.8 for Qwen 2.5 72B. I'll run my other benchmarks later. https://github.com/lechmazur/nyt-connections/
    comment by zone411 · 2025-01-28 · in Qwen2.5-Max: Exploring the intelligence of large-scale MoE model
  21. In my NYT Connections benchmark, it hasn't performed well: https://github.com/lechmazur/nyt-connections/ (see the table).
    comment by zone411 · 2024-10-30 · in DeepSeek v2.5 – open-source LLM comparable to GPT-4, but 95% less expensive
  22. It improves to 25.9 over the previous version of Claude 3.5 Sonnet (24.4) on NYT Connections: https://github.com/lechmazur/nyt-connections/ .
    comment by zone411 · 2024-10-22 · in Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku