Hacker GitHub

← All repos

lechmazur/confabulations

View on GitHub ↗ ★ 251 HTML

Hallucinations (Confabulations) Document-Based Benchmark for RAG. Includes human-verified questions and answers.

14 mentions on Hacker News by 5 people: 2 stories, 12 comments · first 2024-10-10 · latest 2026-02-26

Mentions per year 14 mentions · busiest year 2025 (10)
20082026
AllStoriesComments NewestTopOldest
  1. Well here's some: Confabulation/Hallucination - https://github.com/lechmazur/confabulations Failure to read context - https://georggrab.net/content/opus46retrieval.html Deleting tests to make them pass -…
    comment by grey-area · 2026-02-26 · in I don't know how you get here from “predict the next word”
  2. …https://artificialanalysis.ai/leaderboards/models https://gorilla.cs.berkeley.edu/leaderboard.html https://github.com/lechmazur/confabulations https://dubesor.de/benchtable https://help.kagi.com/kagi/ai/llm-benchmark.html…
    comment by redman25 · 2025-12-26 · in MiniMax M2.1: Built for Real-World Complex Tasks, Multi-Language Programming
  3. GPT-5 set a new record on my Confabulations on Provided Texts benchmark: https://github.com/lechmazur/confabulations/
    comment by zone411 · 2025-08-07 · in GPT-5
  4. I benchmarked it on four of my benchmarks so far. Got first place in two of them: https://github.com/lechmazur/confabulations https://github.com/lechmazur/nyt-connections https://github.com/lechmazur/generalization…
    comment by zone411 · 2025-06-12 · in First thoughts on o3 pro
  5. …it will then spend more test time compute in the quality sources, not the press-releases. [1] https://github.com/lechmazur/confabulations/
    comment by SubiculumCode · 2025-06-05 · in Differences in link hallucination and source comprehension across different LLM
  6. …size comparing how often LLMs confabulate answers based on provided texts, I have a benchmark at https://github.com/lechmazur/confabulations/ . It's always interesting to test new models with it because the results can be unintuitive…
    comment by zone411 · 2025-06-05 · in Differences in link hallucination and source comprehension across different LLM
  7. …model overall, beating all OpenAI models, for example. I wonder what Alibaba did to achieve that. https://github.com/lechmazur/confabulations
    comment by anon373839 · 2025-05-01 · in Running Qwen3 on your macbook, using MLX, to vibe code for free
  8. …Benchmark, nearly matching Gemini 1.5 Pro (Sept) as the best-performing non-reasoning model. ( https://github.com/lechmazur/confabulations ) - Improves upon GPT-4o's score on the Thematic Generalization Benchmark, however, it doesn't match…
    comment by zone411 · 2025-02-28 · in GPT-4.5
  9. …show a difference between reasoning and non-reasoning models: LLM Confabulation (Hallucination): https://github.com/lechmazur/confabulations/ LLM Step Game: https://github.com/lechmazur/step_game LLM Thematic Generalization Benchmark:…
    comment by zone411 · 2025-02-10 · in PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models
  10. The problem is confabulations. In my benchmark ( https://github.com/lechmazur/confabulations/ ), you see models produce non-existent answers in response to misleading questions that are based on provided text documents. This can be…
    comment by zone411 · 2025-01-28 · in How has DeepSeek improved the Transformer architecture?
  11. …the rate as an indicator. They did something similar to what I did on my hallucinations benchmark ( https://github.com/lechmazur/confabulations/ ), only using questions where at least one model made a mistake. I added this note: "The…
    comment by zone411 · 2024-10-30 · in SimpleQA
  12. …with newer models. I tested confabulations based on provided documents (relevant for RAG) here: https://github.com/lechmazur/confabulations/ . Note the significant difference between GPT-4 Turbo and GPT-4o.
    comment by zone411 · 2024-10-20 · in The AI Investment Boom