minimaxir/get-all-hacker-news-submissions-comments
View on GitHub ↗ ★ 127 Python
Simple Python scripts to download all Hacker News submissions and comments and store them in a PostgreSQL database.
13 mentions on Hacker News by 2 people: 2 stories, 11 comments · first 2015-05-11 · latest 2018-02-19
20082026
-
You could do that with the Algolia API: https://github.com/minimaxir/get-all-hacker-news-submissions... But as noted, BigQuery is more pragmatic.
comment by minimaxir · 2018-02-19 · in Ask HN: What has HN given you? -
Not quite what you asked: https://github.com/minimaxir/get-all-hacker-news-submissions... source: https://news.ycombinator.com/item?id=14337275 search related: https://hn.algolia.com/?query=bigquery-public-data:hacker_ne... found:…
comment by j_s · 2017-11-30 · in Ask HN: Is there a script to download past 3 months of HN or top n posts? -
Direct URL to project details: https://devpost.com/software/tagger-news A few comments: 1. To other commenters, as with the HN Vue demo a week ago ( https://news.ycombinator.com/item?id=14284877 ), the project is a technical…
comment by minimaxir · 2017-05-14 · in Tagger News takes a subset of HN articles, analyzes using ML, and applies tags -
A couple years ago, I did my own analysis of all Hacker News submissions ( http://minimaxir.com/2014/02/hacking-hacker-news/ ) and also wrote a script around that time to get all data ( https://github.com/minimaxir/get-all-hacker-news-submi…
comment by minimaxir · 2016-10-11 · in Data Mining Hacker News: Front vs. Back -
> At 4 GB, I'd just as soon query this locally, but this looks like a fun exercise. This requires scraping all the Hacker News data manually , for which I have a tool to do so ( https://github.com/minimaxir/get-all-hacker-news-submissions..…
comment by minimaxir · 2016-01-18 · in Top Books on Amazon Based on Links in Hacker News Comments -
Clickables: https://bigquery.cloud.google.com/table/fh-bigquery:hackerne... https://github.com/fhoffa/notebooks/blob/master/analyzing%20... https://github.com/minimaxir/get-all-hacker-news-submissions...…
comment by minimaxir · 2015-10-23 · in Tell HN: Full Hacker News dataset now available on BigQuery -
▲ 238
Tell HN: Full Hacker News dataset now available on BigQuerystory by minimaxir · 2015-10-23 · 43 comments
-
Relevant: if you're looking for a way to scrape all Hacker News submissions and comments, I have a GitHub repository containing Python scripts showing how to do just that: https://github.com/minimaxir/get-all-hacker-news-submissions...…
comment by minimaxir · 2015-10-19 · in Hacker News as a case study to test the wisdom of the crowd theory -
> the difficulty is that investigating a complex dataset is by definition too complex to really fit into a blog post It's not complex , but it requires answering scaleability questions. How do you fit the data into memory? How do you even…
comment by minimaxir · 2015-09-19 · in Statistics for Hackers -
I have a repository of scripts for downloading all Hacker News data using Python and storing in PostgreSQL: https://github.com/minimaxir/get-all-hacker-news-submissions...
comment by minimaxir · 2015-08-12 · in Ask HN: Any HN stats analysis been done? -
I have a GitHub repository for getting all the HN data the hard way using Python and the Algolia API, and storing them in PostgreSQL: https://github.com/minimaxir/get-all-hacker-news-submissions...
comment by minimaxir · 2015-07-11 · in A dataset of every Reddit comment -
It's worth nothing that according to the source code, this uses the Algolia HN API and not the official HN API, which is a smart move because the official API still doesn't have bulk requests. As a slightly off-topic aside, I have a GitHub…
comment by minimaxir · 2015-06-19 · in Show HN: Wayback HN, a cross between the Wayback Machine and Hacker News -
▲ 3
Show HN: Simple Python scripts to download all Hacker News submissions/commentsstory by minimaxir · 2015-05-11 · 0 comments