webrecorder/browsertrix-crawler
View on GitHub ↗ ★ 1,189 TypeScript
Run a high-fidelity browser-based web archiving crawler in a single Docker container
5 mentions on Hacker News by 4 people: 0 stories, 5 comments · first 2022-04-11 · latest 2023-08-01
20082026
-
(Disclaimer: I work at Webrecorder) Our automated crawler browsertrix-crawler ( https://github.com/webrecorder/browsertrix-crawler ) uses Puppeteer to run browsers that we archive in by loading pages, running behaviors such as auto-scroll,…
comment by tw4l · 2023-08-01 · in Webrecorder: Capture interactive websites and replay them at a later time -
…Even has options to load browser profiles so you can crawl while being authenticated on sites. [0] https://github.com/webrecorder/browsertrix-crawler
comment by _a9 · 2023-05-15 · in Come back, c2.com, we still need you -
https://github.com/webrecorder/browsertrix-crawler works pretty well. Same tech as https://archiveweb.page/ but non-interactive
comment by phiresky · 2023-01-09 · in Dweb: Offline Internet Archive -
…https://github.com/ArchiveBox/ArchiveBox [2] https://webrecorder.net [3] https://replayweb.page [4] https://github.com/webrecorder/browsertrix-crawler
comment by _a9 · 2022-10-07 · in How to download all of Wikipedia onto a USB flash drive -
…Online (SUCHO): https://www.sucho.org/ They seem to be using various tools, like Browsertrix: https://github.com/webrecorder/browsertrix-crawler It sounds promising for interactive sites: > Support for custom browser behaviors, using…
comment by enhdless · 2022-04-11 · in Ask HN: Best approaches to archiving interactive web journalism/writing