Hacker GitHub

← All repos

adamkarvonen/chess_gpt_eval

View on GitHub ↗ ★ 92 Python

A repo to evaluate various LLM's chess playing abilities.

20 mentions on Hacker News by 5 people: 1 story, 19 comments · first 2023-09-23 · latest 2026-02-19

Mentions per year 20 mentions · busiest year 2024 (8)
20082026
AllStoriesComments NewestTopOldest
  1. …assumptions than anything else. https://arxiv.org/abs/2403.15498 https://arxiv.org/abs/2501.17186 https://github.com/adamkarvonen/chess_gpt_eval
    comment by famouswaffles · 2026-02-19 · in AI is not a coworker, it's an exoskeleton
  2. …% legal move rate, ~1800 Elo) https://arxiv.org/abs/2403.15498 https://arxiv.org/abs/2501.17186 https://github.com/adamkarvonen/chess_gpt_eval
    comment by famouswaffles · 2026-02-17 · in Claude Sonnet 4.6
  3. …did play at that level and did not lose track of the pieces. There's no cheating or weirdness. https://github.com/adamkarvonen/chess_gpt_eval
    comment by famouswaffles · 2025-08-13 · in LLMs aren't world models
  4. …care about that. But LLMs can play chess just fine. [0] https://arxiv.org/pdf/2403.15498v2 [1] https://github.com/adamkarvonen/chess_gpt_eval
    comment by famouswaffles · 2025-07-31 · in AI is a floor raiser, not a ceiling raiser
  5. LLMs can play chess fine. The best model you can play with is decent for a human - https://github.com/adamkarvonen/chess_gpt_eval SOTA models can't play it because these companies don't really care about it.
    comment by famouswaffles · 2025-04-16 · in OpenAI o3 and o4-mini
  6. …instruct does so quite well (for a human) at 1800 ELO. Does this mean they can truly reason now ? https://github.com/adamkarvonen/chess_gpt_eval
    comment by famouswaffles · 2025-04-06 · in Recent AI model progress feels mostly like bullshit
  7. …about 5 or less illegal moves in 8205. It's not here but turbo instruct has been evaled before. https://github.com/adamkarvonen/chess_gpt_eval
    comment by famouswaffles · 2024-11-22 · in OK, I can partly explain the LLM chess weirdness now
  8. https://github.com/adamkarvonen/chess_gpt_eval
    comment by famouswaffles · 2024-11-22 · in OK, I can partly explain the LLM chess weirdness now
  9. turbo instruct does play chess reasonably well. https://github.com/adamkarvonen/chess_gpt_eval Even the blog above says as much.
    comment by famouswaffles · 2024-11-15 · in Something weird is happening with LLMs and chess
  10. …been noted by other people to do much better at generating legal chess moves than the other models. https://github.com/adamkarvonen/chess_gpt_eval gave models "5 illegal moves before forced resignation of the round" and 3.5 had very few…
    comment by gs17 · 2024-11-15 · in Something weird is happening with LLMs and chess
  11. …blog but turbo-instruct's chess ability has been pretty thoroughly tested and it does play chess. https://github.com/adamkarvonen/chess_gpt_eval
    comment by famouswaffles · 2024-11-15 · in Something weird is happening with LLMs and chess
  12. …4. It does make illegal moves. It's rare (~5 in 8205) but it happens. https://github.com/adamkarvonen/chess_gpt_eval 5. You can or well you used to be able to inspect the logprobs. I think Open AI have stopped doing this but the
    comment by famouswaffles · 2024-11-14 · in Something weird is happening with LLMs and chess
  13. Gpt-3.5-turbo-instruct had something like 5(or less) illegal moves in 8205 https://github.com/adamkarvonen/chess_gpt_eval I expect the rest to be much worse if 4's performance is any indication
    comment by famouswaffles · 2024-11-14 · in Something weird is happening with LLMs and Chess
  14. …plays at about 1800 Elo and makes legal moves consistently well past openings. https://github.com/adamkarvonen/chess_gpt_eval https://adamkarvonen.github.io/machine_learning/2024/01/03/c...
    comment by og_kalu · 2024-03-26 · in Is GPT-4 a good data analyst? (2023)
  15. …level I spoke of, you will lose repeatedly over whatever number or stretch of games you imagine. https://github.com/adamkarvonen/chess_gpt_eval 3.5 Instruct (different model from regular 3.5 that can't play) can play chess. There's no…
    comment by og_kalu · 2023-12-10 · in Three things that LLMs have made us rethink
  16. …wrong about the move rate. Is 1 in ~1650 really underperforming a 1800 Elo Human ? How much so ? https://github.com/adamkarvonen/chess_gpt_eval That said, >if you understand that it's very often not approaching tasks from first principles…
    comment by og_kalu · 2023-11-09 · in Are language models good at making predictions?
  17. …misremembered the move rate. 5 (potentially less) illegal moves in 8205. So 1 in ~1650. Uh..oops. https://github.com/adamkarvonen/chess_gpt_eval
    comment by og_kalu · 2023-11-09 · in Are language models good at making predictions?
  18. …misremembered the move rate. 5 (potentially less) illegal moves in 8205. So 1 in ~1650. Uh..oops. https://github.com/adamkarvonen/chess_gpt_eval A plane is comparable to a bird because they both fly even if they do it very differently.…
    comment by og_kalu · 2023-11-09 · in Are language models good at making predictions?
  19. Correct. An example can be seen here: https://github.com/adamkarvonen/chess_gpt_eval/blob/master/g...
    comment by seraine · 2023-09-23 · in GPT-3.5-instruct is a ~1800 ELO chess player. 150 games of GPT-3.5 vs. stockfish
  20. ▲ 4
    In a sample of >1000 games, GPT-3.5-turbo-instruct plays chess with ~1800 elo
    story by sebzim4500 · 2023-09-23 · 4 comments