google/robotstxt
View on GitHub ↗ ★ 3,479 C++
The repository contains Google's robots.txt parser and matcher as a C++ library (compliant to C++11).
8 mentions on Hacker News by 7 people: 0 stories, 8 comments · first 2019-07-01 · latest 2025-03-18
20082026
-
https://github.com/google/robotstxt
comment by cavisne · 2025-03-18 · in AI crawlers haven't learned to play nice with websites -
…on mobile devices. Many pieces of software have this to some degree, see robots.txt as an example: https://github.com/google/robotstxt/blob/bacd8ba94b6d066ff6c...
comment by SamuelAdams · 2019-08-19 · in Rules for Intuitive UX -
…understand why they did this, but I have to say I was disappointed to see only 7 commits at https://github.com/google/robotstxt/commits/master dating back to June 25th. When I read "This library has been around for 20 years and it contains…
comment by simonw · 2019-07-01 · in Google’s robots.txt parser is now open source -
I guess lots of people misspell ~disalow~ disallow[1] 1. https://github.com/google/robotstxt/blob/master/robots.cc#L6...
comment by orf · 2019-07-01 · in Google’s robots.txt parser is now open source -
Google is also really generous with how they will let you spell "disallow": https://github.com/google/robotstxt/blob/59f3643d3a3ac88f613... :D
comment by jxcl · 2019-07-01 · in Google’s robots.txt parser is now open source -
…[1] https://www.stonetemple.com/does-google-respect-robots-txt-n... [2] https://github.com/google/robotstxt/blob/59f3643d3a3ac88f613...
comment by jxcl · 2019-07-01 · in Google’s robots.txt parser is now open source -
…characters may be contained in a bots user agent. This is due to strictness in the REP standard. https://github.com/google/robotstxt/blob/master/robots_test.... // A user-agent line is expected to contain only [a-zA-Z_-] characters and…
comment by jchw · 2019-07-01 · in Google’s robots.txt parser is now open source -
Code here: https://github.com/google/robotstxt
comment by rasmi · 2019-07-01 · in Google’s robots.txt parser is now open source