unclecode/crawl4ai
View on GitHub ↗ ★ 86,764 Python
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
11 mentions on Hacker News by 10 people: 8 stories, 3 comments · first 2024-09-29 · latest 2026-07-10
20082026
-
▲ 1
Crawl4AI: Open-source web crawler and scraper for LLMsstory by mpfect · 2026-07-10 · 0 comments
-
▲ 3
Ask HN: What do you use (AI based) web crawlers for?story by dhruv_ahuja · 2026-01-23 · 0 comments
-
▲ 1
Crawl4AI: Open-Source LLM Friendly Web Crawler and Scraperstory by zoooey · 2025-11-18 · 0 comments
-
▲ 4
crawl4ai: The Adaptive Intelligence Updatestory by hbcondo714 · 2025-07-12 · 0 comments
-
▲ 7
Crawl4AI is an open-source, LLM-friendly web crawler and scraperstory by rmason · 2025-04-29 · 2 comments
-
> many AI companies engage in web crawling Individuals do too. Tools like https://github.com/unclecode/crawl4ai make it simple to obtain public content but also paywalled content including ebooks, forums and more. I doubt these folks are…
comment by hbcondo714 · 2025-03-25 · in Devs say AI crawlers dominate traffic, forcing blocks on entire countries -
▲ 1
Crawl4ai – open-source web crawler for your AI agentstory by basic_banana · 2025-02-14 · 0 comments
-
I like the idea but Firecrawl and GPT4o is quite heavy. I use https://github.com/unclecode/crawl4ai in some projects, it works very well without these dependencies and is modular so you can use LLMs but do not have to :)
comment by IndieCoder · 2024-11-22 · in Show HN: Llms.txt Generator – Turn websites into a text file to feed to any LLM -
Just realised that I missed adding the link to the example - https://github.com/unclecode/crawl4ai/issues/126
comment by codelion · 2024-10-20 · in What docs-as-code means -
▲ 2
Crawl4AIstory by synergy20 · 2024-10-05 · 0 comments
-
▲ 6
Crawl4AI: Open-Source Web Crawler for Seamless AI Data Scrapingstory by ProbeCraft · 2024-09-29 · 0 comments