unclecode/crawl4ai
View on GitHub ↗ ★ 86,764 Python
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
11 mentions on Hacker News by 10 people: 8 stories, 3 comments · first 2024-09-29 · latest 2026-07-10
20082026
-
> many AI companies engage in web crawling Individuals do too. Tools like https://github.com/unclecode/crawl4ai make it simple to obtain public content but also paywalled content including ebooks, forums and more. I doubt these folks are…
comment by hbcondo714 · 2025-03-25 · in Devs say AI crawlers dominate traffic, forcing blocks on entire countries -
I like the idea but Firecrawl and GPT4o is quite heavy. I use https://github.com/unclecode/crawl4ai in some projects, it works very well without these dependencies and is modular so you can use LLMs but do not have to :)
comment by IndieCoder · 2024-11-22 · in Show HN: Llms.txt Generator – Turn websites into a text file to feed to any LLM -
Just realised that I missed adding the link to the example - https://github.com/unclecode/crawl4ai/issues/126
comment by codelion · 2024-10-20 · in What docs-as-code means