ai-robots-txt/ai.robots.txt
View on GitHub ↗ ★ 4,243 Python
A list of AI agents and robots to block.
22 mentions on Hacker News by 17 people: 3 stories, 19 comments · first 2024-03-28 · latest 2026-02-11
20082026
-
https://github.com/ai-robots-txt/ai.robots.txt
comment by dmit · 2026-02-11 · in End of an era for me: no more self-hosted git -
User agents not IPs, but: https://github.com/ai-robots-txt/ai.robots.txt
comment by jarofgreen · 2025-12-14 · in Stop crawling my HTML – use the API -
▲ 2
ai.robots.txt – A list of AI agents and robots to blockstory by smartmic · 2025-09-25 · 0 comments
-
…your links and post them there. I recently ran into this as it turns out the Facebook entries in https://github.com/ai-robots-txt/ai.robots.txt also block the crawler FB uses for link previews. 2) From your first post, > But what about…
comment by jarofgreen · 2025-07-17 · in I was wrong about robots.txt -
…break the rules. But if you wanted to supplement the list, might be worth sourcing the ones from https://github.com/ai-robots-txt/ai.robots.txt at some frequency.
comment by rovr138 · 2025-05-31 · in Using lots of little tools to aggressively reject the bots -
https://github.com/ai-robots-txt/ai.robots.txt
comment by JadoJodo · 2025-03-20 · in Claude can now search the web -
…Don't want to completely kill my site from indexing, but if I did I'd 100% set up a tarpit. [^1]: https://github.com/ai-robots-txt/ai.robots.txt
comment by pfych · 2025-02-25 · in The benefits of learning in public -
I’ve been using https://github.com/ai-robots-txt/ai.robots.txt on my Gitea instance.
comment by captn3m0 · 2025-01-29 · in Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt? -
Linked upthread: https://github.com/ai-robots-txt/ai.robots.txt/blob/main/rob...
comment by jazzyjackson · 2025-01-18 · in Amazon's AI crawler is making my Git server unstable -
https://darkvisitors.com/agents https://github.com/ai-robots-txt/ai.robots.txt
comment by jsheard · 2025-01-18 · in Amazon's AI crawler is making my Git server unstable -
…0 - https://marcusb.org/hacks/quixotic.html 1 - I use the ai.robots.txt user agent list from https://github.com/ai-robots-txt/ai.robots.txt
comment by marcus0x62 · 2025-01-16 · in Nepenthes is a tarpit to catch AI web crawlers -
…least respect robots.txt you can use this list to get ahead of them before they pummel your site. https://github.com/ai-robots-txt/ai.robots.txt There's no easy solution for bad bots which ignore robots.txt and spoof their UA though.
comment by jsheard · 2025-01-16 · in Nepenthes is a tarpit to catch AI web crawlers -
…a comprehensive list of LLM and AI related bots to aid in blocking these abusive strip miners. https://github.com/ai-robots-txt/ai.robots.txt
comment by vunderba · 2024-12-30 · in AI companies cause most of traffic on forums -
…but this list is handy for telling the ones that do respect robots.txt to kindly fuck off: https://github.com/ai-robots-txt/ai.robots.txt
comment by jsheard · 2024-12-30 · in AI companies cause most of traffic on forums -
…To block AI bots, you'll need to add known AI bots[1] to your blog's robots.txt file [2]. [1] https://github.com/ai-robots-txt/ai.robots.txt/blob/main/rob... [2] https://docs.bearblog.dev/neat-bear-features/#edit-robotstxt or…
comment by nreece · 2024-12-05 · in Ask HN: Where Do Thou Blog -
https://github.com/ai-robots-txt/ai.robots.txt/blob/main/rob...
comment by zorked · 2024-11-17 · in Cloudflare.com's Robots.txt -
might as well do the full list: https://github.com/ai-robots-txt/ai.robots.txt/blob/main/rob... cloudflare have a button for this: https://blog.cloudflare.com/declaring-your-aindependence-blo...
comment by nikcub · 2024-09-26 · in OpenAI to become for-profit company -
▲ 2
Ai.robots.txtstory by plurby · 2024-09-23 · 1 comment
-
https://github.com/ai-robots-txt/ai.robots.txt
comment by input_sh · 2024-09-01 · in AI Has Created a Battle over Web Crawling -
> I'm glad that most of them seem to respect robots.txt. https://github.com/ai-robots-txt/ai.robots.txt/blob/main/tab... Some of them identify themselves by user agent but don't respect robots.txt, so you have to set up your server to 403…
comment by jsheard · 2024-07-25 · in OpenAI Announces SearchGPT -
…Exclusion Protocol If anyone would like to join in, there's an actively maintained robots.txt here: https://github.com/ai-robots-txt/ai.robots.txt Yes, I know this isn't legally binding and scrapers can ignore it if they want to.
comment by jsheard · 2024-07-21 · in The data that powers AI is disappearing fast -
▲ 44
AI-Shunning robots.txtstory by glynnormington · 2024-03-28 · 90 comments