lechmazur/generalization

View on GitHub ↗ ★ 74

Thematic Generalization Benchmark: measures how effectively various LLMs can infer a narrow or specific "theme" (category/rule) from a small set of examples and anti-examples, then detect which item truly fits that theme among a collection of misleading candidates.

5 mentions on Hacker News by 1 person: 1 story, 4 comments · first 2025-01-14 · latest 2025-08-05

Mentions per year 5 mentions · busiest year 2025 (5)

20082026

All Stories Comments Newest Top Oldest

…) and on 120B and 20B on Thematic Generalization (810 questions, https://github.com/lechmazur/generalization ). Opus 4.1 benchmarks are also there.
comment by zone411 · 2025-08-05 · in Open models by OpenAI
…of them: https://github.com/lechmazur/confabulations https://github.com/lechmazur/nyt-connections https://github.com/lechmazur/generalization https://github.com/lechmazur/writing
comment by zone411 · 2025-06-12 · in First thoughts on o3 pro
…Benchmark, however, it doesn't match the scores of Claude 3.7 Sonnet or Gemini 2.0 Pro Exp. ( https://github.com/lechmazur/generalization ) I should have the results from the multi-agent collaboration, strategy, and deception benchmarks…
comment by zone411 · 2025-02-28 · in GPT-4.5
…LLM Step Game: https://github.com/lechmazur/step_game LLM Thematic Generalization Benchmark: https://github.com/lechmazur/generalization LLM Creative Story-Writing Benchmark: https://github.com/lechmazur/writing Extended NYT Connections…
comment by zone411 · 2025-02-10 · in PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models