Hacker GitHub

← All repos

apify/crawlee

View on GitHub ↗ ★ 26,316 TypeScript

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

10 mentions on Hacker News by 8 people: 3 stories, 7 comments · first 2018-06-02 · latest 2025-10-03

Mentions per year 10 mentions · busiest year 2018 (3)
20082026
AllStoriesComments NewestTopOldest
  1. …https://github.com/encode/httpx https://github.com/scrapy/scrapy https://github.com/apify/crawlee
    comment by renegat0x0 · 2025-10-03 · in How I block all 26M of your curl requests
  2. …as we go, including anti-blocking, adaptive crawling, etc. To see where this might go, check https://github.com/apify/crawlee
    comment by jancurn · 2024-07-09 · in Show HN: Crawlee for Python – a web scraping and browser automation library
  3. …the most, it says "The web scraping and browser automation library." I opened a PR to change it: https://github.com/apify/crawlee/pull/1480
    comment by httbs · 2022-08-24 · in Show HN: Crawlee – Web scraping and browser automation library for Node.js
  4. …Crawlee does appear to do the basic checks though, like checking navigator.webdriver: https://github.com/apify/crawlee/blob/master/test/browser-po... Last time I checked (over a year ago) I couldn't find any public code to make…
    comment by jonatron · 2022-08-23 · in Show HN: Crawlee – Web scraping and browser automation library for Node.js
  5. …they've got their own proxy service (or you can roll your own), and their open source framework ( https://github.com/apify/crawlee ) is excellent. * I've also experimented with running both their SDK (crawlee) and Playwright directly on…
    comment by nlh · 2022-08-10 · in Ask HN: What are the best tools for web scraping in 2022?
  6. …by autoscaling number of parallel Puppeteer instances based on memory and CPU. Our open source SDK ( http://github.com/apifytech/apify-js ) implements this using class PuppeteerCrawler ( https://www.apify.com/docs/sdk/apify-runtime-js/lates…
    comment by mtrunkat · 2018-06-05 · in Observations running 2M headless sessions
  7. …{ // Chrome headless detected - navigator.webdriver exists or was redefined } [1]: https://github.com/apifytech/apify-js/blob/262a2e604b1adb3d8...
    comment by eboyjr · 2018-06-02 · in Bypassing website anti-scraping protections