Hacker GitHub

← All repos

kordless/grub-2.0

View on GitHub ↗ ★ 18 Python

Grub is an AI powered Web crawler.

8 mentions on Hacker News by 1 person: 0 stories, 8 comments · first 2021-08-04 · latest 2022-09-05

Mentions per year 8 mentions · busiest year 2021 (4)
20082026
AllStoriesComments NewestTopOldest
  1. Thanks for this. Very useful. Any interest in adding a crawler? https://github.com/kordless/grub-2.0
    comment by kordlessagain · 2022-09-05 · in Solr’s Dense Vector Search for indexing and searching dense numerical vectors
  2. Not finished, but the Selenium based crawler works pretty well to combat most blocks: https://github.com/kordless/grub-2.0 . I have a slightly modified version running in production for Mitta. For IP blocks, try this:…
    comment by kordlessagain · 2022-08-25 · in Ask HN: I want to dive into how to make search engines
  3. …but am pivoting to prompt management for GPT-3. I've Open Sourced the code for the crawler here: https://github.com/kordless/grub-2.0 . The entire system uses Google Vision for extracting text. I dislike fiddling with the DOM... If you are…
    comment by kordlessagain · 2022-08-01 · in Ask HN: Decent, open source search engine?
  4. …a new version of Grub to address this, but I've found my motivation for working on it lacking: https://github.com/kordless/grub-2.0 A friend and I assembled a module for that repo called Aperture, which implements a scalable screenshot…
    comment by kordlessagain · 2022-07-29 · in Manticore Search: Elasticsearch Alternative
  5. …before indexing. If you need links extracted, Selenium also supports parsing the assembled DOM: https://github.com/kordless/grub-2.0/tree/main/aperture
    comment by kordlessagain · 2021-11-15 · in How to scrape and extract hyperlink networks with BeautifulSoup and NetworkX
  6. …to share with who ever wants to take a look. The repo for deploying Solr on Google Cloud is here: https://gitHub.com/kordless/grub-2.0
    comment by kordlessagain · 2021-11-12 · in Ask HN: Would you build a new search engine today?
  7. …JavaScript, which means BS may not see the text in the DOM. Here is the code to do some of that: https://github.com/kordless/grub-2.0 Link extraction is ignored, but could be done with BS on the rendering of the DOM.
    comment by kordlessagain · 2021-08-12 · in Mastering Web Scraping in Python: Crawling from Scratch
  8. If anyone needs a screenshot bot: https://github.com/kordless/grub-2.0 Still need to do some work on it, but it does do the job!
    comment by kordlessagain · 2021-08-04 · in Show HN: Till – Unblock and scale your web scrapers, with minimal code changes