Hacker GitHub

← All repos

ArchiveTeam/grab-site

View on GitHub ↗ ★ 1,617 Python

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

22 mentions on Hacker News by 9 people: 0 stories, 22 comments · first 2016-02-08 · latest 2026-07-06

Mentions per year 22 mentions · busiest year 2024 (6)
20082026
AllStoriesComments NewestTopOldest
  1. …[2] https://wiki.archiveteam.org/index.php/Wget_with_WARC_output [3] https://github.com/ArchiveTeam/grab-site
    comment by toomuchtodo · 2026-07-06 · in Speedcube.de forum archive using wget and squashfs
  2. https://github.com/archiveteam/grab-site or browsertrix may be easier to use for some, it's what was used to save a lot of the data.gov stuff before it got taken down.
    comment by nikisweeting · 2026-06-14 · in Show HN: Kage – Shadow any website to a single binary for offline viewing
  3. …to read WARC files from crawling the target site previously. https://github.com/webrecorder/warcio https://github.com/ArchiveTeam/grab-site
    comment by toomuchtodo · 2025-12-24 · in Show HN: Got Tired of LLMs refusing to visit URLs Built open source analyzer CLI
  4. https://github.com/ArchiveTeam/grab-site might be helpful. I'm a fan of the ability to create WARC archives from a target, uploard the WARC files to object storage (whether that is IA, S3, Backblaze B2, etc), and then keep them in cold…
    comment by toomuchtodo · 2024-10-16 · in ArchiveBox is evolving: the future of self-hosted internet archives
  5. I'm aware of this tool, but I'm sure there are caveats in terms of "totally" cloning a website: https://github.com/ArchiveTeam/grab-site
    comment by knowaveragejoe · 2024-10-04 · in HTTrack Website Copier
  6. https://github.com/ArchiveTeam/grab-site
    comment by toomuchtodo · 2024-09-19 · in We're losing our digital history. Can the Internet Archive save it?
  7. https://github.com/ArchiveTeam/grab-site https://wiki.archiveteam.org/index.php/The_WARC_Ecosystem
    comment by toomuchtodo · 2024-06-10 · in WARC-GPT: An open-source tool for exploring web archives using AI
  8. I would recommend grab-site instead if you’re archiving, but wget is easier if mirroring content. https://github.com/ArchiveTeam/grab-site
    comment by toomuchtodo · 2024-06-08 · in How to download a copy of a website using Wget
  9. …Internet Archive recommends. A quick search turned up this from the Archive Team and Jason Scott https://github.com/ArchiveTeam/grab-site ( https://wiki.archiveteam.org/index.php/Who_We_Are ) but I found that in less than 15 seconds of…
    comment by TedDoesntTalk · 2024-01-29 · in Ask HN: How can I back up an old vBulletin forum without admin access?
  10. …be available publicly (until some future date). You can always run your own crawls with grab site: https://github.com/ArchiveTeam/grab-site
    comment by toomuchtodo · 2023-12-10 · in Omg.lol: An Oasis on the Internet
  11. There's github.com/ArchiveTeam/grab-site, but unfortunately it's not maintained very well.
    comment by TheTechRobo · 2022-03-02 · in Help preserve the internet with Archiveteam's warrior
  12. Why not use grab-site? https://github.com/ArchiveTeam/grab-site
    comment by toomuchtodo · 2022-01-10 · in How to archive a website that's shutting down soon
  13. Have you tried grab site [1]? [1] https://github.com/ArchiveTeam/grab-site
    comment by toomuchtodo · 2020-06-16 · in Authenticating Shared Web Caches
  14. …the Wayback machine to provide a permanent archive. This is just apathetic laziness on their part. https://github.com/ArchiveTeam/grab-site
    comment by toomuchtodo · 2020-03-31 · in The Unreal Engine Wiki is now permanently offline
  15. …by Wayback. If you prefer, I can do this for you as part of my existing archival operations. [1] https://github.com/ArchiveTeam/grab-site
    comment by toomuchtodo · 2019-10-18 · in The Wayback Machine: Fighting Digital Extinction in New Ways
  16. On a similar note, tools like [grabsite]( https://github.com/ArchiveTeam/grab-site ) wisely use robots.txt as a method of finding additional paths to archive when crawling sites.
    comment by slang800 · 2019-07-10 · in Robots.txt as a Security Measure?
  17. https://github.com/ArchiveTeam/grab-site if you're super serious. Also archive.org will probably accept those output warcs.
    comment by voltagex_ · 2019-05-14 · in Ask HN: What overlooked class of tools should a self-taught programmer look into
  18. Consider https://github.com/ArchiveTeam/grab-site (ArchiveTeam/grab-site) in the future. There is a difference between fetching/mirroring content and recording the http header request and responses when retrieving all objects for a site. I…
    comment by toomuchtodo · 2019-03-09 · in ArchiveBox: Open-source self-hosted web archive
  19. author here. AMA. since I wrote this, i started experimenting with grab-site: https://github.com/ludios/grab-site it's a wrapper around (and a fork of) wpull but the main advantage over it is it can do on-the-fly reconfiguration of delay,…
    comment by anarcat · 2018-11-23 · in Archiving web sites
  20. …could take this and post process your internet history on a rolling basis to accomplish your goal. https://github.com/ludios/grab-site
    comment by toomuchtodo · 2018-11-22 · in Archiving web sites
  21. Thanks for doing this Aaron! Something to consider for the future: if you use grab-site ( https://github.com/ludios/grab-site ), which is a close cousin of ArchiveTeam's ArchiveBot, it'll retrieve all of the content from HackerNews (making…
    comment by toomuchtodo · 2016-10-31 · in Hacker News Dataset Update October 2016
  22. > Any more good archiving tools? For backing up websites, I released https://github.com/ludios/grab-site . Compared to HTTrack, it makes it easy to add ignores to skip unwanted URLs after a crawl has already started. It also saves to WARC…
    comment by ivank · 2016-02-08 · in GitHub-backup: backs up everything GitHub knows about a repository or a user