manyoso/haltt4llm
View on GitHub ↗ ★ 223 Python
This project is an attempt to create a common metric to test LLM's for progress in eliminating hallucinations which is the most serious current problem in widespread adoption of LLM's for many real purposes.
4 mentions on Hacker News by 3 people: 2 stories, 2 comments · first 2023-03-29 · latest 2023-04-04
20082026
-
…based on how well they are at saying "I don't know" (of course knowing is still rewarded higher): https://github.com/manyoso/haltt4llm
comment by est31 · 2023-04-04 · in Introducing Agents in Haystack: Make LLMs resolve complex tasks -
…of the correct answer appears anywhere in the output. Even if other incorrect answers also appear. https://github.com/manyoso/haltt4llm/blob/main/take_test.py#... So the above answer would have been correct were it not for the fact that it…
comment by Imnimo · 2023-03-31 · in GPT4All outscores GPT-3.5 on new hallucination metric -
▲ 36
GPT4All outscores GPT-3.5 on new hallucination metricstory by manyoso · 2023-03-30 · 13 comments
-
▲ 4
New metric for LLM hallucinations with results for GPT-3, GPT3-5 and Alpaca Lorastory by manyoso · 2023-03-29 · 3 comments