rumca-js/crawler-buddy on Hacker News

…often using my crawling server because rss are often protected against bots https://github.com/rumca-js/crawler-buddy

comment by renegat0x0 · 2026-08-01 · in How Google helped destroy adoption of RSS feeds (2023)

…link metadata - h ttps://github.com/rumca-js/RSS-Link-Database-2025 - 2025 year link meta data - h ttps://github.com/rumca-js/crawler-buddy - crawler engine

comment by renegat0x0 · 2026-05-11 · in Ask HN: What are you working on? (May 2026)

…- Internet meta database - https://github.com/rumca-js/Internet-feeds - list of Internet feeds - https://github.com/rumca-js/crawler-buddy - crawling framework - https://github.com/rumca-js/RSS-Link-Database-2026 - link meta from year 2026…

comment by renegat0x0 · 2026-04-13 · in Ask HN: What Are You Working On? (April 2026)

…try to keep in good shape is 'docker' image. I update it once everything seems to be quite stable. https://github.com/rumca-js/crawler-buddy

comment by renegat0x0 · 2026-03-25 · in Why I forked httpx

…- https://github.com/rumca-js/Internet-Places-Database - database of domains and youtube channels - https://github.com/rumca-js/crawler-buddy - web crawling / web scraping tool - https://github.com/rumca-js/webtoolkit - web crawling toolki…

comment by renegat0x0 · 2026-03-09 · in Ask HN: What Are You Working On? (March 2026)

…- database of RSS feeds - https://github.com/rumca-js/yafr - very simple RSS reader - https://github.com/rumca-js/crawler-buddy - crawling project - https://github.com/rumca-js/Django-link-archive - another RSS reader

comment by renegat0x0 · 2026-02-09 · in Ask HN: What are you working on? (February 2026)

…into simple and fast web page using SQLite table [2]. Scraping done right is not a problem. [0] https://github.com/rumca-js/crawler-buddy [1] https://github.com/rumca-js/Internet-Places-Database [2] https://rumca-js.github.io/search

comment by renegat0x0 · 2026-01-21 · in Waiting for dawn in search: Search index, Google rulings and impact on Kagi

…bit different, but also allows to scrape efficiently. Json http communication rather than cli. https://github.com/rumca-js/crawler-buddy More like a framework for other mechanisms

comment by renegat0x0 · 2026-01-14 · in Show HN: Webctl – Browser automation for agents based on CLI instead of MCP

Still working on https://github.com/rumca-js/Internet-Places-Database - database of Internet places https://github.com/rumca-js/crawler-buddy - crawling server https://github.com/rumca-js/Internet-feeds - list of RSS feeds

comment by renegat0x0 · 2026-01-04 · in Ask HN: What are you working on? (Jan 2026)

…a web crawler and I have both scraping byte limit and timeout, so zip bombs dont bother me much. https://github.com/rumca-js/crawler-buddy I think garbage blabber would be more effective.

comment by renegat0x0 · 2025-11-15 · in Messing with scraper bots

…own crawling system to mitigate these problems, and have one consistent mean of running selenium. https://github.com/rumca-js/crawler-buddy Based on library https://github.com/rumca-js/webtoolkit

comment by renegat0x0 · 2025-10-31 · in AI scrapers request commented scripts

…tests different mechanisms and browser headers while exploring the web. My scraping mechanism: https://github.com/rumca-js/crawler-buddy Web crawler / RSS reader https://github.com/rumca-js/Django-link-archive

comment by renegat0x0 · 2025-10-26 · in Feed the bots

…internet links https://github.com/rumca-js/Internet-Places-Database Still crawling framework https://github.com/rumca-js/crawler-buddy Still RSS client https://github.com/rumca-js/Django-link-archive

comment by renegat0x0 · 2025-10-13 · in Ask HN: What are you working on? (October 2025)

…and web crawling system [1]. Links: [0] https://github.com/rumca-js/Django-link-archive [1] https://github.com/rumca-js/crawler-buddy

comment by renegat0x0 · 2025-08-29 · in RSS is awesome

…The downside is that it may not be the fastest solution, and I have not tested it against proxies. https://github.com/rumca-js/crawler-buddy

comment by renegat0x0 · 2025-08-08 · in Turn any website into an API

…Additionally I have a project that is able to read RSS links and provides it in JSON response https://github.com/rumca-js/crawler-buddy

comment by renegat0x0 · 2025-07-02 · in Teaching feed readers about YouTube subscriptions

…domains, links - https://github.com/rumca-js/Django-link-archive - RSS client, web crawler - https://github.com/rumca-js/crawler-buddy - web scraper, web crawler, with JSON interface A project is like a pet. You cannot just "stop" caring…

comment by renegat0x0 · 2025-06-30 · in Ask HN: What Are You Working On? (June 2025)

…and all common properties are read into a simple JSON data. Maybe somebody will find it useful. https://github.com/rumca-js/crawler-buddy

comment by renegat0x0 · 2025-06-26 · in RSS Server Side Reader

…similar. You make GET request to server with any of supported crawlers and obtain result in JSON https://github.com/rumca-js/crawler-buddy/ Supports request, selenium, Httpx, curl cffi, etc

comment by renegat0x0 · 2025-06-20 · in Hurl: Run and test HTTP requests with plain text

…for me not to setup selenium for the x time. I just use one crawling server for my projects. Link: https://github.com/rumca-js/crawler-buddy

comment by renegat0x0 · 2025-06-10 · in Show HN: PyDoll – Async Python scraping engine with native CAPTCHA bypass

…of crawling, selenium, and others. Returns data in JSON (status code, text contents, etc). [1] https://github.com/rumca-js/crawler-buddy

comment by renegat0x0 · 2025-05-11 · in Scraperr – A Self Hosted Webscraper

…like thumbnails, so all nuance from RSS is lost - detects feeds from sites automatically Links [0] https://github.com/rumca-js/crawler-buddy/blob/main/src/webt... [1] https://github.com/rumca-js/crawler-buddy

comment by renegat0x0 · 2025-05-06 · in Show HN: Feedsmith — Fast parser & generator for RSS, Atom, OPML feed namespaces

…new projects like RSS readers easier [1] [0] https://github.com/rumca-js/Django-link-archive [1] https://github.com/rumca-js/crawler-buddy

comment by renegat0x0 · 2025-01-18 · in Ask HN: Which RSS reader do you use?

…status was returned. While tiny tiny RSS is nice, I also wrote interface to read URLs from the net. https://github.com/rumca-js/crawler-buddy This gives me clean data of web page, title, description, etc, which I can further integrate into…

comment by renegat0x0 · 2025-01-17 · in I ditched the algorithm for RSS